# Automated Mapping of Adaptive App GUIs from Phones to TVs

Han Hu<EMAIL_ADDRESS>Monash UniversityWellington
RoadClaytonVictoriaAustralia3800 ,  Ruiqi Dong<EMAIL_ADDRESS>Swinburne
University of TechnologyJohn StreetHawthornVictoriaAustralia3122 ,  John
Grundy<EMAIL_ADDRESS>Monash UniversityWellington
RoadClaytonVictoriaAustralia3800 ,  Thai Minh Nguyen
<EMAIL_ADDRESS>Monash UniversityWellington
RoadClaytonVictoriaAustralia3800 ,  Huaxiao Liu<EMAIL_ADDRESS>Jilin
UniversityNo.2699, Qianjin RoadChangchunJilinPR China130015 and  Chunyang Chen
<EMAIL_ADDRESS>Monash UniversityWellington
RoadClaytonVictoriaAustralia3800

(2023)

###### Abstract.

With the increasing interconnection of smart devices, users often desire to
adopt the same app on quite different devices for identical tasks, such as
watching the same movies on both their smartphones and TV. However, the
significant differences in screen size, aspect ratio, and interaction styles
make it challenging to adapt Graphical User Interfaces (GUIs) across these
devices. Although there are millions of apps available on Google Play, only a
few thousand are designed to support smart TV displays. Existing techniques to
map a mobile app GUI to a TV either adopt a responsive design, which struggles
to bridge the substantial gap between phone and TV or use mirror apps for
improved video display, which requires hardware support and extra engineering
efforts. Instead of developing another app for supporting TVs, we propose a
semi-automated approach to generate corresponding adaptive TV GUIs, given the
phone GUIs as the input. Based on our empirical study of GUI pairs for TV and
phone in existing apps, we synthesize a list of rules for grouping and
classifying phone GUIs, converting them to TV GUIs, and generating dynamic TV
layouts and source code for the TV display. Our tool is not only beneficial to
developers but also to GUI designers, who can further customize the generated
GUIs for their TV app development. An evaluation and user study demonstrate
the accuracy of our generated GUIs and the usefulness of our tool.

graphic user interface, cross-screen, UI detection, GUI pattern

††copyright: acmcopyright††journalyear: 2023††doi:
10.1145/1122445.1122456††journal: TOSEM††journalvolume: X††journalnumber:
Y††article: Z††publicationmonth: 2††copyright: none††ccs: Software and its
engineering††ccs: Human-centered computing HCI design and evaluation methods

##  1\. Introduction

Graphical User Interfaces (GUI) are ubiquitous in almost all modern desktop
software, mobile applications (apps), and online websites. They provide a
visual bridge between a software application and end-users through which they
can interact with each other. A good visual design makes an application
attractive and easy to use, which significantly affects the success of the
application and the loyalty of its users (Goldschmidt, 1994). With the
pervasiveness and diversity of devices, especially mobile ones, users tend to
use the same app across different platforms in different scenarios (e.g.,
watching YouTube on a phone on the bus while using it on a tablet or TV at
home) (Joorabchi et al., 2013). When utilizing apps on TV and tablets,
especially those within the media, education, and tools categories, the
adaptation to a large-screen TV/tablet display substantially elevates the user
experience. Therefore, to ensure usability, software GUIs need to be adaptive
to various screen sizes and densities, ranging from small 4.7-inch
smartwatches to 75-inch smart TVs.

Although one app may run well on different platforms with the same operating
system (e.g., Android), the content may not display well without customization
for different devices or screens. For example, one app designed for the
Android phone will only be displayed in the middle of the TV screen with a
large dark space on the two sides due to the different aspect ratios as seen
in Figure 1 (_Direct mapping_). Additionally, smart TVs and mobile phones
interact differently. Mobile phones employ fingers to touch and swipe, while
smart TVs require remote controls. TVs need redesigned GUIs to accommodate
different interactions as well as differences in size and layout.

Currently, there are three ways commonly used to deal with this issue. First,
many GUI development frameworks provide support of responsive and adaptive
design (adaptiveUI, 2022) for developers to adapt the GUI to any possible
screen size, such as Android’s Material Design (Laine et al., 2021) and
iOS(Bhuttoo et al., 2017). The _Desktop mode_ in Figure 1 shows an example of
the responsive layout. Since responsive and adaptive layout techniques simply
adapt the GUI to the current device’s screen size based on the size
constraints and proportions of the original phone screen, the GUI has to
clumsily enlarge some UI components, increase the distance between UI
components, or alter the layout of UI components. Although the GUI occupies
the current screen, its layout is quite clumsy and significantly diminishes
the user experience. Responsive design is primarily focused on optimizing for
small screens like smartphones and tablets. Therefore, it may not be optimized
for larger screens like TVs, leading to issues when viewing mobile apps on
TVs. Additionally, TVs often have different interaction methods than mobile
devices, such as a remote control or voice commands, which may not be
supported by the app’s responsive design.

Figure 1. Examples of current large screen GUI conversions

Second, there are some screen mirroring apps (e.g., Chromecast (chromecast,
2021)), which are used to cast music, video, play games, and display photos on
a big screen. However, these are effective primarily for video projection
rather than general app screens. Phone manufacturers like Samsung, Apple,
Huawei, and OnePlus have introduced the concept of Desktop Mode (huaweiDesk,
2021; Dex, 2021) for cross-screen GUI conversion via their phones. Desktop
Mode maps the current phone GUIs to larger external screens via an HDMI
adaptor or WiFi by redesigning GUI layouts individually to improve the user
experience with a larger screen.

Third, as revealed in our empirical study in Section 2.1, developers often
create a brand-new GUI for the corresponding TV application due to their
significant differences. While there are only a small number of thousands of
apps supporting Android TV (AndroidTVApp, 2021)111Some apps are specifically
developed for TV and cannot run on the Phone., this is a minuscule fraction
compared with the millions of Android apps, which greatly limits users’ app
usage on the TV. Developing both a mobile app and a standalone TV app from
scratch is both time-consuming and labor-intensive. Considering the user side,
a comparable design could involve providing data or research on user
preferences for consistency across different devices. This method reduces the
need for them to learn new navigation and interaction patterns (Li et al.,
2017; Lowdermilk, 2013). Popular apps always share a similar design between
phone apps and TV apps, for example, YouTube and Spotify. From the developer
side, a consistent GUI can reduce development costs and time, as it allows
developers to reuse code and design elements across different devices
(Lowdermilk, 2013). Both the phone app and the corresponding TV app have
similar functions and can share current designs and resources. Reusing
existing materials can result in significant savings on engineering expenses,
and a similar user interface makes it easier for mobile app users to adapt to
the TV app (Mendoza, 2013; Lowdermilk, 2013).

To overcome these limitations, we propose an approach for generating the
appropriate GUI layout for Android TV based on the existing GUI design for
phones. Unlike responsive design, where a screen ’flows’ from a phone design
into a larger device, adaptive projection in our approach offers tailor-made
solutions. Given the app GUI of a page on a mobile phone, our approach
automatically generates the corresponding GUI code to make it adaptive to the
TV screen. The development team, including the visual designers and
developers, will benefit from our approach. Once they finish the phone app
development, our approach can automatically generate the GUI design and TV
implementation for any phone page. Developers and designers can easily
customize it to enhance their productivity without starting from scratch.

To build the tool, we first investigate how many apps currently support smart
TV in 5,580 popular apps from Google Play(googleplay, 2021). We find only
5.34% of apps support running on smart TV. Second, we perform a formative
study on the characteristics of current TV-phone apps and GUIs. We collect
1405 TV-phone app pairs from Google Play and Dangbei(dangbei, 2021) but only
collect 589 TV-phone GUI pairs with clear GUI correspondence in these app
pairs. We then summarize 12 and 9 categories of GUI components groups common
on mobile and TV, respectively, and analyzed their corresponding conversion.
Based on the empirical study results, we build our tool in the following
steps: group isolated GUI components parsed from GUI metadata captured by UI
Automator (uiautomator, 2021), convert phone GUI groups into corresponding TV
groups, optimize their layouts by OR constraints formulas (Jiang et al.,
2019), and translate current TV GUI to our language - and platform free GUI
domain specific language (DSL) for further rendering.

We choose two of the best-known and most commonly used technologies: desktop
mode and direct mapping as the baselines for the evaluation. We select mIoU,
overall satisfaction, and structure rationality as the metrics to evaluate the
converted TV GUIs by our approach and baselines. We also recruited 20
participants with extensive experience in Android GUI development, design, and
use to evaluate the converted TV GUIs. The experimental results show that TV
GUIs converted by our approach achieve 21.05%, 10.42%, and 21.31% improvement
in mIoU, overall satisfaction, and structure rationality than the best current
conversion techniques. Besides, a pilot user study also provides the initial
evidence of the usefulness of our tool for bootstrapping adaptive GUI from
phone to TV.

Our contributions to this work are summarized as follows:

  * •

As far as we know, this is the first study on automated GUI conversion from
smartphone to TV GUI;

  * •

We propose a new approach to generate TV-hosted GUIs from Android mobile phone
GUIs;

  * •

We carry out an empirical study to understand the current status of GUI
support in the phone app to TV display and how is the GUI mapping patterns
between two platforms; and

  * •

We demonstrate the effectiveness of our approach with extensive automated
evaluation and manual checking. We also provide initial evidence of our tool’s
usefulness via a pilot user study.

##  2\. Empirical study of GUIs between phones and TVs

To better understand the current status and characteristics of GUI adaptation
to smart TV, we conduct a large-scale empirical study to answer two questions:
1) How many phone apps support TV display? 2) How do app GUIs change between
TVs and smartphones?

###  2.1. RQ1 How many phone apps support TV displays?

To answer RQ1, we analyze a large number of industrial Android apps from all
33 categories on Google Play (googleplay, 2021). We crawl the top 200 most
popular apps in each category (at the time of Jun 2021). Since some apps are
not free, we obtain 5,580 apps that support Android phones by default.

According to the official guidelines of developing Android TV (AndroidTVDe,
2021), it is compulsory for apps running on TV to declare a TV activity with
an intent filter CATEGORY_LEANBACK_LAUNCHER in the Android manifest file of
Android projects. In addition to the declaration, TV-enabled apps often have
separate TV layouts called layout-television and layout-tv. Therefore, we
decompile Android APKs to check whether the specified intent filters or layout
XML files exist. We find only 298 of these apps supporting TV display,
accounting for only 5.34% of the total. These 298 apps belong to 29 categories
including Weather (16.78%), Education (15.10%), Tool (11.07%).

Even if an app advertises that it supports a TV display, this does not imply
that it is appropriately optimized for the TV. We manually check 298 apps that
claim to support TV displays. We physically run the apps on a smart TV and
critically evaluate how well various GUI pages adapt to the TV display. GUI
pages exhibiting discrepancies such as substantial black margins at the
screen’s bilateral extents, discordant aspect ratio components, disordered
layouts, and unoptimized navigation, among other factors – which all may
diminish the user experience on TV displays – are regarded as inadequate
instances of the TV adaptation. We discover only 11 of 298 apps allow TV
display on all GUI pages. 287 out of 298 apps modify a few representative GUI
pages, such as the home or landing page, to accommodate TV displays. These 287
apps have an average of 22.3 Android activities, but we only find support for
TV display in an average of 4 Android activities. Other GUIs of these apps
also look poor on TV, with large black margins on both screen sides, and
mismatched aspect ratio components, as the _Direct mapping_ in Figure 1. Due
to the significant difference between phone and TV displays, some development
teams tend to develop separate apps for different platforms.

Figure 2. Examples of Phone GUI Groups

###  2.2. RQ2 How do app GUIs change between TVs and smartphones?

We collected custom-designed TV apps and their matching phone apps for a
comparative study since there are relatively few apps that support both TV and
phone.

####  2.2.1. TV-phone App Pairs Collection

We collected 249 TV apps from Google Play’s TV category (googleTV, 2021) and
2,556 from Dangbei (dangbei, 2021), which is one of the largest TV app stores.
We eliminated any apps that have not been updated for over two years.

We then matched the TV apps’ corresponding smartphone apps on Google Play. To
begin, we search for phone apps with the same app name, developers, and
category. Second, if the corresponding phone version app cannot be found, we
broaden our search to include apps with similar app names developed by the
same developers. Finally, for the TV apps that do not find the phone-version
apps, four volunteers manually collect their matching phone apps.

We match 1,405 TV-phone app pairs, with the three most common categories being
_Video (42%)_ , _Education (23%)_ and _Tool (21%)_. Video apps, such as
Youtube, IQIYI, are most suitable for smart TVs due to the characteristics of
TV itself, so video apps have become the most popular apps on TV. Because
smart TVs are mostly used at home, there are a number of educational apps for
children. Tool apps, like TV app store, remote control and projection control
are also ubiquitous on TVs.

####  2.2.2. TV-phone GUI Pair Collection

We use the DroitBot (Li et al., 2017), Fastbot (Cai et al., 2020) and
Uiautomator2 (uiautomator2, 2021) to automatically explore apps and collect
rendered screenshots and metadata of GUIs in apps. The metadata is a
documentary object model (DOM) tree of current GUIs, which includes the
hierarchy and properties (e.g., class, bounding box, layout) of UI components.
We can infer the GUI hierarchy from the DOM Tree hierarchy in metadata. After
removing duplicates, we obtain 6,697 Android GUI data and 4,112 TV GUI data.
We notice that most TV apps simplify or restructure their GUIs to accommodate
various usage scenarios and requirements of smart TVs during this process.

At TV-phone GUI pairing, we exploit the semantic similarity of Android
activity names to pair the TV and phone GUIs automatically. Then, we
automatically compare UI components on GUI pages in order to match state-level
GUI pairings in a lesser granularity. Finally, we manually check the
automatically discovered pairs and select the final valid TV-phone pairs. We
extract activity names from each GUI and encode them into numerical semantic
vectors using a pre-trained BERT (Devlin et al., 2019) model. Then, we match
the TV-phone GUI pairs by comparing their semantically close activity vectors.
For example, the GUI in activity homeActivity and mainActivity are matched by
close semantic vectors. However, one Android activity may have multiple
Android fragments (fragment, 2022) and GUI states (Machiry et al., 2013; GUI
state, 2022) with different UI components and layouts in current industrial
apps. We further compare GUI components between phones and TVs to pair TV-
phone GUIs at lower granularity. In pairing, UI components are identified by
their types and properties. UI components between phones and TVs with the same
types and properties are considered paired GUI components. For example, two
_TextViews_ with the same texts, two _ImageViews_ with the same images, two
_Buttons_ with the same texts are considered the paired components. If more
than half of the UI components in two GUIs are paired, they are considered a
state-level TV-phone GUI pair. Finally, we manually checked all discovered
pairs and identified 589 TV-phone state-level GUI pairs with clear GUI
correspondence between phone and TV components.

####  2.2.3. GUI component grouping

A series of UI components that are near in position and hierarchy of the GUI
and tend to have the same functionality are referred to as GUI
groups222Sometimes called GUI patterns (guiBuild, 2022; Neil, 2014). Exploring
these group changes based on UI groups rather than each individual component
is more beneficial (Neil, 2014). So, we must first identify and categorize GUI
groups that are common to both TV and phone before summarising the guidelines
for GUI changes from phone to TV.

We perform an iterative-open coding process, which is widely used to generate
categories in Software Engineering (Seaman, 1999; Liu et al., 2018; Sgier,
2012). We do this on 120 randomly selected phone GUIs and 80 TV GUIs
(approximately 2% of collected phone and TV GUIs) to categorize their GUI
groups. Four volunteers with Android design experience undertook three steps
in our open coding procedure. At first, we are inspired by Google design
(GoogleMaterial, 2021) and development guidelines (androidGuidelines, 2022),
every volunteer categorizes GUI groups in selected GUIs individually. After
the initial coding, let four volunteers have a discussion and merge conflicts.
They clarify scope boundaries among categories and misunderstandings in this
step. In the third step, they iterate to revise classifications and discuss
with each other until a consensus is reached. Finally, we determined 12 phone
group types: Icon + Info, Tool Bar, Bottom Tab Layout, Search, Top Tab Layout,
Pic Side Info, Pic + Info, Side Nav, Short Video Player, Video/Music Player,
Big Pic and List View and 9 TV group types: Icon + Info, Tool Bar, Search, Tab
Layout, Channel, Grid Layout, Pic + Info, Video/Music Player and List View.
Figure 2 and 3 show examples of summarized phone and TV groups. These phone
and TV GUI group categories can be divided into two subcategories. GUI groups
in the first subcategories are widely used in both phone and TV, e.g. Tool
Bar, Search, and Video/Music Player. GUI groups in the second subcategory only
exist in phones or TVs respectively, e.g. such as Bottom Tab Layout and
Channel.

To verify the accuracy of GUI grouping and classification, we first randomly
sample 20 TV and smartphone apps in collected 1405 TV-phone app pairs in
Section 2.2.1, respectively. The selected apps have covered all 6 TV app
categories (Tool, Video, Music & Audio, Education, Entertainment, and Weather)
listed on Google Play (Google Play TV, 2023). Then, we manually calculate the
distribution of each GUI group in selected TV and smartphone apps. Finally, we
find that these summarized GUI components groups have covered 93.69% and
93.47% of GUI groups on phones and TVs, respectively. In our analysis, we
observe the presence of uniquely shaped GUI components in both phone and TV
interfaces. These components resist typical categorization due to their
complex functional requirements in production and everyday use. They
constitute 6.31% and 6.53% of phone and TV GUIs, respectively. These specific
GUI components primarily appear within individual apps or among apps from the
same developer. We categorize these components under an $Others$ group
category for the purpose of our study. To address these edge cases in the GUI
conversion process, we propose the development of default templates. Given the
relatively minor percentage of GUI components in the $Others$ category and
their limited appearance in specific applications, we assert that our
categorization effectively represents the vast majority of commonly used phone
and TV GUIs.

Table 1 shows the details of GUI group distributions. Subcolumns _Group_ and
_Distribution_ of columns _Phone_ and _TV_ denote the GUI group categories and
distribution in the experiment. TV’s subcolumn _Group_ does not contain the
_Others_ GUI group, which comprises 6.53% of all TV GUIs in the experiment. On
phones, the most popular categories of components groups are Icon + Info
(13.31%), Tool Bar (11.41%) and List View (11.14%), but on TV, categories Pic
+ Info (19.13%), Grid Layout (13.37%) and List View (13.18%) are most common
categories. The official guideline of TV GUI design (AndroidTVDe, 2021)
suggests two principles for TV GUI design _All TV GUIs should display in
landscape mode_ and _The core TV GUIs use card-like views instead of ListView
or ViewPager to make better use of horizontal screen space and accommodate TV
interaction_. Standard current TV GUIs obey these two principles to use more
grid layouts and card-like widgets. Thus card-like categories Pic + Info and
Grid Layout are more popular on TVs than on phones.

Figure 3. Examples of Key TV GUI Groups

####  2.2.4. Component group alignment

After conducting our GUI components grouping study, we notice that the design
principles of TV and phone GUIs are vastly different, resulting in no obvious
one-to-one alignments between most TV-phone GUI groups. Furthermore, the
contents of one phone GUI group may be dispersed throughout numerous groups in
the TV GUI, and vice versa. Therefore, we summarize heuristic rules from our
collected TV-phone GUI pairs for automatic GUI mapping from phone to TV.

Firstly, we randomly divide the 589 TV-phone GUI pairs into experimental and
validation sets in an 8:2 ratio. In the experimental set, four volunteers
follow the same three steps to perform open coding to analyze and extract
conversion rules. Table 1 shows the extracted GUI group match rules from phone
to TV. To accommodate how the TV and remote interact, each TV GUI group uses
card-like views. The converted TV GUI group recalculates the new size
depending on the quantity and types of components in the existing GUI to fit
the TV screen size. According to the GUI group study in Section 2.2.3, we use
Grid Layout as our default template for mapping. Component groups with the
same meaning Icon + Info, Tool Bar, Search, Top Tab Layout, Video/Music Player
and List View in phone and TV are transferred directly. According to the
characteristics and the official design guideline (AndroidTVDe, 2021) of TV,
Pic Side Info, Pic + Info and Big Pic are all converted to Pic+Info in TV.
After exploration, current TV GUIs tend to replace components in the phone’s
Side Navigation and Bottom Tab Layout to Channel in TV, so we follow this
trend. Short Video Player should use customized templates in TV, but there’s
no such TV app with this GUI feature at the moment. As a result, we don’t
provide the corresponding TV group individually at the moment, instead relying
on Video Player to convert.

We use the validation set to verify these mapping rules. Our first step
involves manually identifying and extracting GUI components from respective
groups within the phone GUI. Subsequently, we locate corresponding components
in the matching TV GUI. Our final process entails verifying if these TV GUI
components align with the anticipated TV groups and comply with the matching
rules established in our experimental set. Given _m_ instances of _Side Nav_
groups in our validation set, and $n$ corresponding TV GUI groups classified
as $Channel$ groups, we compute the mapping rule accuracy as the ratio $n/m$.
Note that if the phone GUI group is eliminated in the corresponding TV GUI,
the case is considered invalid and will not be counted. The _Mapping Accuracy_
in Table 1 demonstrates the correctness rate of each mapping rule. Finally, we
find that the correctness of rules 1, 2, 3, 4, 5 and 7 are 96%, 99%, 91%, 99%,
99%, 99% and 100%, respectively, indicating that these direct mapping rules
are accurate and universal. Rules 8, 9, 10, 11, and 12 have an accuracy of
83%, 95%, and 87%, 90%, and 99%, respectively, suggesting these change rules
are also accurate and common.

Table 1. Component group matching between phone and TV. TV’s subcolumn _Group_
does not contain the _Others_ GUI group, which comprises 6.53% of all TV GUIs
in the experiment.

Index | Phone | TV | Mapping Accuracy
---|---|---|---
Group | Distribution | Group | Distribution
1 | Icon + Info | 13.31% | Icon + Info | 8.32% | 96%
2 | Tool Bar | 11.41% | Tool Bar | 7.83% | 99%
3 | List View | 11.14% | List View | 13.18% | 91%
4 | Top Tab Layout | 8.88% | Top Tab Layout | 7.68% | 99%
5 | Search | 7.98% | Search | 7.12% | 99%
6 | Others | 6.31% | Grid Layout (Default) | 13.37% | 90%
7 | Video/Music Player | 3.50% | Video/Music Player | 7.56% | 100%
8 | Pic Side Info | 8.90% | Pic + Info | 19.13% | 83%
9 | Pic + Info | 8.67% | 95%
10 | Big Pic | 3.52% | 87%
11 | Bottom Tab Layout | 10.31% | Channel | 9.28% | 90%
12 | Side Nav | 6.07% | 99%

###  2.3. Summary and Implications

Our empirical study shows that: (1) Only 5.34% of popular phone apps support
TV displays. (2) In TV-phone GUI pairs, there is not much explicit one-to-one
correspondence between phone and TV component groups. (3) We summarize 12 and
9 categories of GUI components groups on phone and TV, covering 93.69% and
93.47% popular phone and TV GUIs, respectively. (4) We extract and evaluate 12
existing GUI group-mapping rules from phone to TV based on summarised GUI
component groups.

The lack of TV-display support for phone apps confirms the necessity of tool
development for semi-automated GUI mapping between phone and TV. That
motivates our study and the empirical findings of component group mapping are
the backbone of our proposed approach.

##  3\. Semi-automated TV GUI Generation

Motivated by our empirical findings and the official layout criteria for
Android TV (Android TV, 2021), we here propose our semi-automated Android-
based TV GUI generation approach. Our approach develops a lightweight
migration system that converts run-time GUIs in a series of phases, including
component recognition and grouping, template matching, layout optimization,
and GUI domain-specific languages (DSL) generation. The overall pipeline of
our approach is illustrated in Figure 4.

We first gather the run-time GUI metadata generated by UI
Automator(uiautomator, 2021), including screenshots and GUI metadata, and
analyze hierarchical metadata to identify GUI components. Secondly, to
construct a suitable GUI mapping, we design a component grouping algorithm
that leverages the GUI information extracted from metadata to group isolated
elements properly. Thirdly, we compare elements’ attributes and hierarchical
similarities to classify grouped elements into appropriate types and match
corresponding TV templates. Then, we optimize the whole GUI layouts to adapt
TV screens by OR-constraints (Jiang et al., 2019; Jiang et al., 2020).
Finally, given the incompatibility of phone and TV systems, a cross-platform
TV GUI DSL is designed to describe generated TV GUI for further compilation
and rendering.

Figure 4. Overview of Automated GUI Conversion from Phone to TV

###  3.1. Component Grouping

As mentioned in Section 2.2.3, we first group isolated GUI components to GUI
groups as the basic unit of follow-up work. We parse metadata captured by UI
Automator to accurately infer the pix-based coordinate of GUI components’
bounding box and classify these components to proper types like TextView,
ImageView, and Button. Once each GUI component’s bounding boxes and type in a
rendered screen are confirmed, the next phase is to group atomic components
with similar domain-specific functions to one component group. Figure 5
illustrates a running example for our component grouping algorithm. Overall,
the algorithm consists of three-level grouping with different assembly
granularities. To begin, we gather all atomic components that have significant
relationships, as well as all images and their text descriptions. Second, we
then group components in the same row by their hierarchy, type, and total
width. Third, we then merge adjacent rows with the same hierarchy and pix-
based areas.

Figure 5. A running example of components grouping. The block dotted box 1
represents atomic grouping, the blue 2 represents row grouping, and the red 3
represents multi-row grouping.

####  3.1.1. Atomic Components Grouping

Regarding text pairs are very close on the Y axis, they tend to be a pair of
descriptions, with the caption at the top and the additional explanation of
the caption at the bottom (Neil, 2014; andriodUI, 2022; Zhang et al., 2021).
Texts below or on the right of images always have strong semantics with them,
such as previewed movie names, image descriptions, etc (Neil, 2014; andriodUI,
2022). Thus, heuristics are designed to aggregate relevant components based on
component type, position, size, and structural relationship.

For text pair grouping, given upper text $T_{u}$ and below texts set $T_{s}$,
if texts in $T_{s}$ meet the following requirements, they will be grouped with
$T_{u}$: (1) On X-axis, $T_{u}$ and $T_{s}$ overlap, or the gap between them
must be less than $0.025\times screen\ horizontal\ resolution$. (2) On Y-axis,
$T_{s}$ is the closet element around the $T_{u}$. The gap between them must be
lower than $0.025\times screen\ vertical\ resolution$. We employ the
subsequent three phases to confirm that our heuristics contains sufficient
relevant component possibilities and to acquire the best applicable empirical
parameters. Following the relevant work of Xiaoyi etc. (Zhang et al., 2021)
and the official Android GUI design guidelines (Android TV, 2021; adaptiveUI,
2022), we first determine all potential positions and distance range of the
images and their accompanying isolated UIs. Second, we randomly selected 10
GUI pages in each category of our collection of phone apps (90 GUI pages in
total). We experiment on the selected data to verify whether our heuristic
rule can cover all the relevance possibilities. The experimental results
indicate that our heuristic principles are capable of covering all
possibilities in existing GUIs. Finally, we assessed the impact of all
empirical coefficients within the range provided by Android official
guidelines on the selected pages in increments of 0.001 $\times$ the current
screen’s horizontal and vertical resolution. We ultimately determined 0.025 to
be the optimal empirical coefficient. Potential compatibility issues with
phones and apps may lead to distorted or misaligned UI components in the GUI,
making our heuristic principles inapplicable. However, these circumstances can
be rectified by developers through subsequent development.

For image relevant texts grouping, given texts set $T$ and image $I$, if texts
in $T$ meet the following requirements, they will be grouped with $I$: (1) On
Y-axis, $T$ is the closet element below the $I$ and the gap between $T$ and
$I$ must be less than $0.025\times screen\ vertical\ resolution$. (3) On
X-axis, the midpoints of $T$ and $I$ are the same. If not, their overlap on
X-axis must be more than 50% of the smaller one in $T$ and $I$. (4) If $T$ is
in a grouped text pair, pack both texts as the relevant texts of $I$.

####  3.1.2. Row Components Grouping

According to Google’s Android design guide (androidGuidelines, 2022;
andriodUI, 2022), the standard UI design typically treats each row as the
fundamental unit. Therefore, the UI components in the phone are arranged in
rows. The metadata of the phone’s GUI, as illustrated in Figure 4, describes
its GUI hierarchy in the form of an XML DOM tree. UI components in the same
row are in the same subtree in the DOM tree. The purpose of our row components
grouping is thus to trim the subtrees from the DOM tree that contains all UI
components in one row. When the algorithm finds that components inside the
same DOM subtree have occupied an entire row, it will put all of these
components into one group.

Algorithm 1 demonstrates our row components grouping process. It utilizes a
DOM tree ($tree$) of metadata and a pre-defined group width parameter
($W_{group}$) as inputs. The latter is a critical determinant dictating when
to partition a subtree representing a row on the screen. Specifically, when
the subtree’s width extends to match the phone screen’s width, it is deemed to
occupy an entire row. Given that GUI elements are always displayed with
margins on both the left and right sides of the screen, we empirically define
the requisite width as 0.85 times the total screen width in this work. The
output of Algorithm 1 is the grouped components _G_. The leaf nodes in the DOM
tree are the specific UI components, so Alg. 1 first walks from the root node
of the DOM tree to the terminal leaves, collecting all leaves $L$ (lines 1-6).
Following the identification of leaf nodes, the algorithm backtracks up the
DOM tree from each leaf in $Leaves$ to establish respective subtrees
representative of a screen row. For each leaf $l$, if the width of the current
node ($W_{cur}$) exceeds the predefined group width $W_{group}$, prune at node
$l$. If not, backtrack to the current node’s parent node $p$. If the width of
$p$ still does not exceed the predefined group width $W_{group}$, the
traceback procedure continues until the current node’s width meets the
criterion that is larger than the predefined width (lines 7-12). We then trim
the DOM tree from the node where the backtracking stops to get the cropped
subtree $S$ (line 13). To group all UI components within a single row, the
algorithm subsequently collects all leaves in the subtree $S$ as a group
$g_{l}$ (lines 14-19).

Algorithm 1 Row Components Grouping

1:DOM tree $Tree$, required group width $W_{group}$

2:Row components group set $G$

3:$Leaves\leftarrow[]$

4:for each component $c\in C$ do

5: if length(getChildren ($c$)) == 0 then

6: $Leaves$.add($c$) //collect all leaves in $tree$

7: end if

8:end for

9:for each Leaf $l\in Leaves$ do

10: $node_{cur}\leftarrow l$, $W_{cur}\leftarrow$ $l$.width

11: while $W_{cur}<=W_{group}$ do

12: $p\leftarrow$ getParent($l$) //Backtrack to the parent node $p$ of $l$ in
$Tree$

13: $node_{cur}\leftarrow p$, $W_{cur}\leftarrow$ $p$.width

14: end while

15: $S\leftarrow$ trimTree($node_{cur}$) //Trim $tree$ at node $node_{cur}$ to
get a subtree $S$

16: $g_{l}\leftarrow[]$ // the set to collect all leaves in $S$

17: for each $l^{\prime}\in S$ do

18: if $l^{\prime}\in L$ then

19: $g_{l}$.add($l^{\prime}$)

20: end if

21: end for

22: $G$.add($g_{l}$ ) //Add $g_{l}$ to $G$

23:end for

24:return $G$

Figure 6 shows a running example of the Alg. 1. The backtracking starts at
leaf TextView _Call Me By Fire_ and reaches the leaf’s parent subtree 1.
However, the width of subtree 1 is less than the required group width
$W_{group}$. We then keep going back to subtree 2, but the width of subtree 2
is still not satisfied. All the way back to subtree 3, its width is more than
the required group width. The algorithm thus stops at subtree 3 and trims
subtree 3 from the GUI DOM tree as a GUI component row.

####  3.1.3. Multi-row Components Grouping

Components in one component group may be spread across multiple rows, such as
ListView and GridLayout. In these component groups, the components of
different rows tend to have the same structure and component types. Thus,
given two adjacent rows $r_{i}$ and $r_{j}$, based on their bounding boxes, we
compute the relative positions of the upper left and lower left corners of
components in each row. If components in $r_{i}$ and $r_{j}$ have the same
types and relative upper left and lower left corners, two rows will be
grouped. As shown in Figure 6, subtree 3 has the same structure as the row
below, so merge these two rows into one GUI group in step 4. Note that we
allow two rows in one group in the implementation with one different
component.

Figure 6. A running example of Algorithm 1 and multi-row component grouping.

###  3.2. TV Template Matching

We use GUI built-in and visual features to categorize groups and match related
templates after component grouping. We first summarize the unique attributes
of each type of group. The built-in attributes of the group Top Tab Layout,
Bottom Tab Layout, _Tool Bar_ , _Search_ , _Video/Music Player_ , _List View_
, and _Side Nav_ are their component types, position and relationship with
their sibling components. The unique built-in attributes of the groups _Top
Tab Layout_ , _Bottom Tab Layout_ are the component types and their positions
in the GUI pages. Both groups will use the GUI components of the tab layout
class and will be situated in the upper or lower half of the GUI pages,
accordingly. Similarly, the unique built-in attributes of the groups _Tool
Bar_ , _Search_ , and _Side Nav_ are also their specific component types and
positions. The group _Tool Bar_ is located at the top of the GUI page. The
_Search_ group is located in the upper half of the GUI pages with a search
box. The _Side Nav_ group is located on the leftmost side of the GUI page and
has a unique side navigation property in the DOM tree. The group _Video/Music
Player_ and _List View_ also have player and list view attributes unique to
the Dom tree, respectively. The image size and related information position
features are used to classify groups Icon + Info, Pic Side Info, Pic + Info,
and Big Pic.

When matching templates, we count the number of built-in attributes that each
group has in common with each template. After matching all templates, if the
maximum number of matching attributes exceeds the threshold, the most
comparable template is assigned to the group. If it is below the threshold,
the group is deemed unrecognized. In the case of unrecognized GUI groups, we
provide a general Grid Layout-based template for their conversion. The
threshold is set at 2 after multiple iterations of error correction based on
experimental feedback.

Figure 7 demonstrates an example of classifying GUI groups. We notice when
parsing the metadata of this GUI page that there are some GUI components at
the top with the fields _searchText_ , _searchBtn_ , and _search_container_ in
their attributes. These are similar to the attributes of the _Search_
template. As a result, this GUI group is categorized under the category
_Search_. The top of the page has a GUI group with the class type _ActionBar-
Tab_. Its features of position and class type meet the template _Top Tab
Layout_ , and hence it is classified to category _Top Tab Layout_. To proceed,
we can now continue to identify GUI groups of categories _Big Pic_ ,
_Pic+Info_ , and _Bottom Tab Layout_ from the page.

Figure 7. An example of template match for the phone GUI.

Following the identification of GUI groups in the phone page, we convert these
categorized phone GUI groups into matching TV GUI groups using the mapping
rule in Table 1.

###  3.3. Layout optimization

Due to various screen sizes and design principles, direct layout mapping may
lead to issues like large white space on the right or too much content
squeezed into a small area. The Android TV design guidelines (TVDesign, 2022;
AndroidTVDe, 2021) state that TV layouts should be landscape and have more
card-like components since this is more suited to TV interaction and enables
the display of only the most essential image and text contents. So, if we just
map each GUI group into the TV screen one by one, or simply change the GUI
orientation from portrait to landscape without optimizing the layout, the
final produced TV layout would be quite inflexible and violate the design
standards of TV GUIs.

On phone GUIs, huge images frequently take up a whole row, as in Figures 2 and
7. However, the design guidelines for Android TV (Android TV, 2021; TVDesign,
2022; AndroidTVDe, 2021) state that images taking up an entire row would seem
to be quite odd and significantly worsen the user experience. Additionally, if
one image is too big, the remaining components will be too small for viewers
watching the TV from far away to see. Currently, as _Desktop mode_ in Figure
1, Android’s adaptive technology immediately projects onto the TV based on the
proportion of its components. The slider picture occupied the whole TV screen
and entirely obscured the location of other components after being adjusted by
the adaptive layout, resulting in an odd overall effect.

To overcome these issues, we propose a TV-based GUI layout OR-constraints
(Jiang et al., 2019; Jiang et al., 2020) to optimize converted TV layouts. The
OR-constraint is a combination of soft constraints, with the whole thing being
a hard constraint. The hard restrictions must be met, but the soft constraints
are not required. We set soft constraints for each component in one row, and
all components in one row must meet the hard constraints for the Android TV
layout. Unlike template-based approaches, which necessitate pre-designing
templates and manually specifying rules for when each alternative should be
invoked, constraints-based layout optimization can be more flexible and
adaptive for a variety of screens without fixed templates and rules. Different
from the Android adaptive GUI layout, the constraint-based layout optimization
approach, in conjunction with the TV group template and guidelines, can
arrange the layout of UI components on the whole screen. This prevents the
scenario when some UI components, due to the original size of the phone, are
too huge to be transferred to the TV.

We summarize the layout requirements from the TV design guidelines (andriodUI,
2022; Android TV, 2021; TVDesign, 2022) and our empirical study. We convert
these requirements into constraints, and force converted GUI groups to
optimize their layouts to satisfy these constraints. We lift three predefined
heuristics as basic constraints for TV GUI layout: (1) Arrange GUI widgets
from left to right. (2) If there is not enough space in the current row for
the following widget, it will begin on the leftmost side of the next row. (3)
Each component should be put within a predefined size range. Every TV GUI
design must adhere to these three hard constraints. At the same time, TV GUIs
also have one soft constraint: (4) There should be no black gaps on either
side of the TV screen, which means the UI Component should fill the whole
width of the screen. If there isn’t enough space in the converted TV GUI,
components will be removed in order of their size in the phone GUI, from small
to large. The Z3 SMT solver (De Moura and Bjørner, 2008) is used to solve OR-
constrains.

The following part shows how we formulate these constraints. Given a row with
$n$ components $W$. For each component $w_{i}\in W$, assume its left
$w^{left}_{i}$, width $w^{width}_{i}$, top $w^{top}_{i}$, and current
available width in the row $r_{a}$. Let $w^{height}_{j}\in W$ denotes the max
components height in the current row. The maximum/minimum widths and heights
of components $w_{i}$ are represented as $width^{max}_{i}$, $width^{min}_{i}$,
$height^{max}_{i}$, and $height^{min}_{i}$. For the constraint (1), we convert
it to the following formula:

(1) $\displaystyle
C_{1}:=(w^{left}_{i}=w^{left}_{i-1}+w^{width}_{i-1})\land(w^{top}_{i}=w^{top}_{i-1})$
$\displaystyle\land(r_{a}>=width^{min}_{i})$

For the constraint (2), it is formulated as

(2) $\displaystyle
C_{2}:=(w^{left}_{i}=0)\land(w^{top}_{i}>=w^{top}_{j}+w^{height}_{j})$
$\displaystyle\land(r_{a}<width^{min}_{i})$

For constraint (3), we assign preferred widths, and heights for common GUI
component types. So,

(3) $\displaystyle
C_{3}:=(width^{min}_{i}<=w^{width}_{i}<=width^{max}_{i})\land$
$\displaystyle(height^{min}_{i}<=w^{height}_{i}<=height^{max}_{i})$

For constraint (4), let $r_{tv}$ represent the maximum width of the TV screen
in one row, and its logical expression is

(4) $C_{soft}^{1}:=\sum_{i=1}^{n}(w_{i}^{width})=r_{tv}$

Each row represents a single formula unit. If a GUI group covers multiple
rows, the GUI group is referred to as a formula unit. For each formula unit,
$C_{1}$ and $C_{2}$ are OR constraints, and $C_{3}$ is a hard constraint. In
addition to three basic constraints, some weighted soft constraints $C_{soft}$
are followed.

The final formula is thus:

(5) $C_{unit}:=((C_{1}\ \lor\ C_{2})\ \land\ C_{3})\lor C_{soft}^{k}$

where $k$ represents the number of soft constraints. We can dynamically apply
new soft constraints to Equation 5 dependent on the demands of the TV GUI.

###  3.4. DSL for GUI Code Synthesis

Current apps for phones and TVs may be developed in different development
environments and programming languages. In order to extend our approach to as
many platforms as possible, we design a novel Language- and platform-free GUI
Domain Specific Language (DSL) for code synthesis to assist developers with
secondary development and cross-platform compilation once we’ve optimized the
layout.

To properly describe the GUI features, we provide a lightweight card-style DSL
with a pre-written GUI block library based on the properties of TV GUI. When
we get converted TV groups, we build a set of DSL statements by traversing the
type, size, position, and relationship of all converted rows from top to
bottom and left to right. Every component group is turned to a DSL statement
in this project. To limit the size of the DSL keyword vocabulary, the DSL
merely concentrates on the types and layout of GUI components in each row.

Current GUI groups are distinguished by their position, kind, size, and
hierarchical relationships. To reflect these features, we let $L_{i}$
represent the layout type of the $i_{th}$ GUI row, for example, row and
multirow. $C_{j}$ represents the TV GUI group to which the $j_{th}$ component
belongs, for example, _Tool Bar_ and _List View_. $P_{m}$ represents the
additional textual parameters for $C_{j}$, for example, the image title in GUI
group _Pic+Info_. Each DSL statement follows the following syntax rule:

(6)
$\begin{split}Statement_{i}:=&L_{i}(C_{1}(P_{1},P_{2},...P_{m}),C_{2}(P_{1},P_{2},...P_{m}),...,C_{j}(P_{1},P_{2},...,P_{m}))\end{split}$

The notation $L$ signifies the layout phrase, dictating the arrangement of
input components. Its possible values, $Row$ and $Col$, indicate horizontal
and vertical placements of UI contents, respectively. The symbol $C$
designates the category of the GUI group, encapsulating the nine identified TV
GUI groups in Section 2.2.3. Meanwhile, $P$ denotes the properties of this GUI
group, which may include text information for the $Icon+Info$ and $Pic+Info$
groups, the selected state of $TabLayout$, and the image source of $Pic+Info$,
among others.

Figure 8 exhibits an instance of our lightweight DSL. The second line of DSL
encapsulates the layout of the first row in the TV GUI. The term Row is our
layout phrase ($L$), stipulating the horizontal positioning of all input
components. The $Tab$ in $Tab(VARIETY)$ symbolizes the _Top Tab Layout_
category in TV GUI groups ($C$), and $VARIETY$ in $Tab(VARIETY)$ signifies the
text property ($P$) within the group. Both the category phases $Slider$ and
$Img$ are subcategories of the TV GUI group $Pic+info$ (lines 3, 4, 5). The
sub-category that $Pic+info$ invokes depends on the size of the input image.

Figure 8. An Example of Our TV GUI DSL

The DSLs will be translated into real-world code according to the platform’s
requirements and renders the TV interface. We pre-write the TV GUI style
library and install a client app with the TV GUI style library on TV.
According to the DSL keywords, the style library calls the associated GUI
code, converts it to real-world code, and invokes the system to render it.

###  3.5. Implementation

Uiautomator2(uiautomator2, 2021), Fastbot (Cai et al., 2020), and Droidbot (Li
et al., 2017) are used to capture GUI metadata. All algorithms in the pipeline
are implemented by Python3. Our pipeline generates DSL for TV layout, then
sends the intermediates to TV. We pre-install a client app on TV to receive
generated DSL and translate DSL to real-world code to render TV screens. The
TV client app is built on the Leanback (Leanback, 2021) library. Leanback is a
library for TV user interface and is provided by Google. To produce a GUI,
Leanback must be developed based on the interface. With our DSL, Leanback
input is automatically generated without the need for the developer to again
spend time creating the GUI. According to the optimized size of each GUI
component, each GUI group type is split into three subclasses during
implementation: large, medium, and small. Based on Leanback, we developed
templates for all the subclasses of the 9 TV GUI groups we summarized.
According to the GUI DSL, the client app calls the corresponding template,
generates the corresponding TV GUI code, and renders it.

##  4\. Accuracy Evaluation

We carry out experiments to evaluate the accuracy of the GUI grouping and the
converted TV GUI effects in our pipeline.

###  4.1. Accuracy of GUI Grouping

The accuracy of GUI component grouping serves as the foundation for succeeding
techniques. Hence, we first evaluate the accuracy of our proposed grouping
algorithm.

####  4.1.1. Procedure

To confirm the generalizability of our grouping algorithm, we randomly select
another 10 apps that are not in our dataset. To ensure the quality of selected
apps, we only select apps with at least one million Google Play installations.
A total of 100 GUI pages are selected as the test dataset, with 10 GUI pages
from each app being randomly selected. Next, we perform the grouping algorithm
on the selected GUI pages and collect grouping results. The grouping results
are then manually checked to ensure their reasonableness.

####  4.1.2. Metric

A fair result for a GUI group must contain only UI elements with strong
logical connections. When manually verifying the rationality of grouping, a
group fails if it contains UI components that are logically unrelated to other
UI components inside the group. When we refer to ‘logically related UI
components’, we mean an intuitive association of GUI elements based on
interactivity and role. Interactivity pertains to the direct communicative
relationship between components, such as an input field and a ’submit’ button.
The ’role’ encapsulates the components’ collective function within the broader
UI context. Hence, ‘logically related UI groups’ signify clusters of UI
elements intuitively assembled based on their interaction synergy and shared
role in the user interface.

We use the exact match rate to illustrate the percentage of reasonable GUI
groups. The exact match is a binary metric, with a value of 1 if correct and 0
otherwise. If there are $m$ GUI groups on this GUI page, and $n$ of them are
reasonable, then the grouping accuracy of this GUI page is $n/m$. Figure 10
demonstrates an example of correct and wrong GUI grouping on one GUI page.
Groups 1, 3, and 4 in Figure 10 are considered to be correctly grouped since
all UI components within the group are logically related. But group 2 is
considered to be grouped incorrectly. This is because, in addition to the
search-related UI components in group 2 (the UI components boxed by the blue
dashed line in Figure 10), there are two logically unrelated icons, which
should not be classified into this search-related UI group. The grouping
accuracy of Figure 10 is 0.75. Considering the subjective nature of
determining ’logically related UI components’, we employ three individuals
with a minimum of one year’s experience in GUI development to independently
conduct the exact match evaluation. Thereafter, the Fleiss Kappa value
(Fleiss, 1971) is used to measure the level of agreement among these three
evaluators. Fleiss Kappa values are interpreted as follows: [0.01, 0.20]
signifies slight agreement, (0.20, 0.40] indicates fair agreement, (0.40,
0.60] represents a moderate agreement, (0.60, 0.80] suggests substantial
agreement, and (0.8, 1] signifies near perfect agreement. Instances where the
Fleiss Kappa value falls below 0.8 prompts a discussion, analysis, and re-
evaluation by the three evaluators until a Fleiss Kappa value exceeding 0.8 is
achieved. To evaluate our method’s ability to minimize the isolated UI
components, we follow related works (Zhang et al., 2021; Chen et al., 2019) to
select the proportion of reduced UI components as the second metric. Suppose
there are $J$ UI components on the original GUI pages, and $K$ UI components
and UI groups are left after grouping, then the ratio of the reduced
components is $(j-k)/j$.

####  4.1.3. Baseline

Xiaoyi et al. (Zhang et al., 2021) propose a similar approach to group UI
components for efficient navigation. They develop multiple heuristics that
group UI components based on their UI types, sizes, and spatial relationships
on the rendered phone screen. Considering similar application scenarios, we
choose their method as the baseline for our experiments.

Table 2. Evaluation Results of Accuracy of GUI Grouping ID | App | Category | #Installation (Million) | ExactMatch | ReducedUI
---|---|---|---|---|---
Ours | BL (Zhang et al., 2021) | Ours | BL (Zhang et al., 2021)
1 | iQIYI Video | Entertainment | 5 | 0.86 | 0.71 | 0.62 | 0.49
2 | Coursera | Educational | 10 | 0.72 | 0.61 | 0.55 | 0.37
3 | Evernote | Productivity | 100 | 0.80 | 0.77 | 0.61 | 0.43
4 | Kodi | Tool | 50 | 0.85 | 0.81 | 0.62 | 0.52
5 | Pinterest | Lifestyle | 500 | 0.74 | 0.62 | 0.56 | 0.40
6 | Wonder | Art & Design | 5 | 0.79 | 0.72 | 0.53 | 0.47
7 | Fiverr | Bussiness | 10 | 0.83 | 0.69 | 0.55 | 0.51
8 | ABC listen | Music & Audio | 1 | 0.91 | 0.81 | 0.64 | 0.56
9 | Fitbit | Health & Fitness | 50 | 0.84 | 0.78 | 0.56 | 0.59
10 | Kik | Communication | 100 | 0.76 | 0.73 | 0.52 | 0.45
Average | 0.81 | 0.73 | 0.58 | 0.48

####  4.1.4. Results

Table 2 summarizes the information of the selected 10 apps and the accuracy
results of the GUI grouping. The column _#Installation_ demonstrates the
number of app installations. The Fleiss Kappa value of the first grouping
results for three evaluators of our 10 apps are all between 0.91 and 1. Three
evaluators discuss the different parts and finally agree on a unified final
result. The subcolumn _Ours_ and _BL_ show our and the baseline’ experimental
results in the exact match rate and reduced UI components rates, respectively.
Our approach achieves 0.81 in average exact match, which is 10.96% higher than
the baseline _BL_ (Zhang et al., 2021) (0.73). Our approach reduces isolated
GUI components in GUI pages by an average of 58%, which is 20.83% higher than
the baseline (48%). Both results demonstrate the effectiveness of our GUI
grouping algorithms.

Figure 10 depicts the results of a pair of our method and baseline’s method
after grouping, with the first subplot representing our method and the second
representing the baseline’s. Our approach splits the GUI page into 5 GUI
groups and other GUI components. Correspondingly, the baseline separates the
page into 8 distinct groupings and components. Clearly, our grouping results
are more precise and overlook fewer individual components. For example, in our
group 2 and the corresponding groups 2 and 3 of the baseline, our method
considers the possible related information surrounding the text, successfully
groups the related images on the right side together, and merges the adjacent
groups with the same structure, whereas the grouping result of the baseline
method omits the related image data. In more complex scenarios, such as group
3 and group 4 in subfigure (a), corresponding to groups 4, 5, 6, and 7 in
subfigure (b), the results of groups 4 and 5 in subfigure (b) omit the
majority of the information since baseline’s approach cannot handle the case
of numerous lines of text. Our approach is built with more general atomic,
row, and multi-row grouping methods, so that the group results contain as much
important data as feasible, and reduces the number of groups on a GUI page to
expedite the subsequent conversion operation.

Figure 9. Examples of Correct/Wrong GUI Grouping

Figure 10. Examples of Our and Baseline’s Grouping Results. The first one is
our grouping results and the second one is baseline’s grouping results.

###  4.2. Accuracy of GUI Conversion

Once the TV GUI DSLs have been generated, we convert them into source code and
then run them to get the rendered run-time GUI pages. On the one hand, the
impact of the same run-time GUI can be accomplished through a variety of code.
On the other hand, there is a gap between the GUI source code and the rendered
GUI effect, and the GUI source code does not reflect the rendered GUI effect
in its entirety (Rountev and Yan, 2014). To demonstrate the efficacy of the
approach, we choose to evaluate the rendered effect of apps running through
the translated DSLs. We perform both an automatic evaluation and a user study
to evaluate the performance of the whole automated GUI conversion approach.
All 589 pairs in Section 2.2 are used in the automated evaluation to
objectively evaluate the overall effect of the method. The quality of visual
transformation is strongly dependent on human perception, so we also include a
user study to assess our approach’s performance. Due to the great efforts of
the user study, we randomly sample 42 (7%) GUI pairs among 589 pairs from 8
apps in 6 categories.

####  4.2.1. Evaluation Metrics

According to recent studies (Chen et al., 2018; Moran et al., 2018b), when
using phones and TVs, users largely rely on the layout of images and texts to
understand GUIs. Uniform GUIs facilitate user adaptability from mobile to TV
app. Additionally, a consistent GUI promotes a reduction in developmental cost
and time by enabling the reuse of code and design elements (Mendoza, 2013;
Lowdermilk, 2013). Therefore, we substantiate the accuracy of GUI conversion
by quantifying the similarity between TV GUIs that are automatically generated
and those manually designed, serving as the ground truth in this study. The
mIOU (Jaccard, 1912), which could effectively evaluate the layout gap between
each type of the component in one GUI page, has been widely used in GUI
evaluation (Kumar et al., 2011; Laine et al., 2021; Zheng et al., 2019; Chen
et al., 2018; Xie et al., 2020; Zhao et al., 2021; Moran et al., 2018c). Based
on its characteristics and suitability for our specific study, we select mIoU
as the metric to evaluate the layout similarity of generated TV GUIs and
ground truths. The TV versions in phone-TV GUI pairs have been redesigned and
optimized with a more logical layout of text and images in the GUI to
accommodate the features of the TV. So we select these redesigned TV GUIs as
the ground truth of the corresponding phone GUI in the automated evaluation.

The mIoU (Mean Intersection Over Union), also known as the Jaccard Index, is a
prominent image segmentation assessment measure that computes the IOU for each
class before calculating the average over classes. The IoU is calculated by
dividing the overlap area between predicted class positions and ground truth
by the area of union between predicted position and ground truth. This is
computed by:

(7) $mIoU=\frac{1}{k}\sum_{i=0}^{k}\frac{TP(i)}{TP(i)+FP(i)+FN(i)}$

where $k$ means $k$ classes in both images, $TP(i)$, $FP(i)$ and $FN(i)$
represent the distribution of true positive, false positive, and false
negative of $i_{th}$ class between two compared images.

Our empirical study in section 2.2 shows that the current GUIs of paired
phones and TVs often do not have strict correspondence. Besides, whether a GUI
design makes sense depends significantly on the user’s subjective perception.
A GUI with a low mIOU in the automatic evaluation may be deemed acceptable by
some users. For example, some GUI may have a more reasonable GUI design than
the ground truth. These reasonable GUI designs got low scores because their
layouts are not in line with the ground truth. To eliminate the bias, we
adopted two metrics (Zhao et al., 2021), structure rationality and overall
satisfaction for participants in the user study to rate the quality of the
generated TV GUI by considering the characteristics of the TV apps. These
metrics were inspired by the web GUI evaluation (Kumar et al., 2011; Laine et
al., 2021; Zheng et al., 2019) and image evaluation (Hore and Ziou, 2010;
Ponomarenko et al., 2008). First, structure rationality is used to evaluate
component layout rationality, which refers to the placement of components in
the GUI as well as the reasoning behind their combination and sorting. Second,
overall satisfaction is to evaluate the overall design’s pleasing qualities.
For each metric, the participants will give a score ranging from 1 to 5, with
1 being the least satisfactoriness and 5 representing the highest
satisfactoriness.

####  4.2.2. Baselines

Desktop mode (huaweiDesk, 2021; Dex, 2021) is widely used in various
smartphones, such as Samsung, OnePlus, Huawei, and Oppo. It allows users to
connect an external display to an Android smartphone or tablet to make content
easier to view, just like on a TV or computer. The desktop mode is optimized
for larger displays with resizable windows and a different layout for GUIs.
HDMI adapter or WiFi is required to use the desktop mode. In this experiment,
considering the current use range and maturity of computer mode, we first
choose Huawei EMUI desktop mode as the baseline, which is one of the earliest
phone models to support desktop mode.

Currently, Google provides the big-screen responsive layout component for
Android system (Android TV, 2021; AndroidTVApp, 2021; AndroidTVDe, 2021;
adaptiveUI, 2022; googleTV, 2021). When an Android app runs on different-sized
screens, adaptive and responsive Android GUI components will adjust their
positions and sizes to fit the screen size of each device. In our empirical
study, we found that part of the apps using these technologies automatically
adapts to different screens. Even though these technologies often offer a
worse user experience than a fully hand-optimized GUI, it is nevertheless
worthwhile to compare them to semi-automatic methods. Thus, our second
baseline is the result of directly mapping adaptive Android GUI from phone to
TV. Unlike the desktop mode, no external equipment is required for direct
mapping.

In the user study, we also compare our method to the redesigned TV GUI, which
serves as the ground truth for the automated evaluation. The comparison with
the ground truth will provide a clear image of the efficacy gap between our
method and the redesigned GUI, which requires extra work.

####  4.2.3. Procedures

In the automatic evaluation, we select redesigned TV GUIs in 589 pairs as our
ground truths. We use our proposed approach to convert every phone GUI to DSLs
of TV GUI. Then we use the client app on TV to generate code and render GUIs
on the Android TV emulators (androidStudio, 2022). The emulator is configured
with 4 CPU processors, 4 GiB of RAM, and 1 GiB of SD card. The API level
version of the system image is 26. When GUIs are rendered, we get the metadata
of GUIs to generate their corresponding wireframes. The content of pictures
and text in the GUI is not taken into consideration since we are comparing the
structure of the rendered GUI, not the pixels of the produced GUI, therefore
we convert images to red blocks and texts to green blocks in wireframes. In
the same way, we generate wireframes of baselines and our approach’s generated
TV GUI. Finally, we evaluate the mIoU between wireframes of our generated TV
GUIs and ground truths.

In the user study, we recruit 20 participants who are professional designers
and developers with more than 3-year Android development experiment for this
user study. We first introduce them with a detailed explanation of tasks and
the two GUI evaluation metrics structure rationality and overall satisfaction.
Meanwhile, we provide participants with all generated GUI designs from
different methods and ask them to give the score of each GUI design in two
metrics of the user study. Note that they do not know which TV GUI is from
which method and all of them will evaluate the TV GUI design individually
without any discussion. For each test case, participants are asked to choose
one GUI they think works best.

####  4.2.4. Results

Table 3 demonstrates average results on the testing set of our approach and
three baselines. The ground truth GUI, which requires extra engineering effort
to redesign one-to-one, is undoubtedly more favorable than the other three
automatically generated approaches. In 19 cases out of 42, _Ground Truth_ is
deemed the most efficient, followed by our method (15), _Desktop mode (7)_ ,
and Direct mapping (1). It reaches the highest structure rationality (4.27)
and overall satisfaction (4.20). However, this approach requires customization
for each GUI page, adding significantly to the engineering expenses and making
it non-generalizable. Compared to ground truth’s approach, ours performs
marginally worse (0.17 lower on _Structure Rationality_ and 1.2 lower on
Overall Satisfaction). Our approach produces the best results across 15 cases,
which is quite similar to the ground truth (19), and we do not require the
additional engineering costs associated with tailoring each GUI page. Our
approach outperforms the other two baselines in mIoU, overall satisfaction,
and structure rationality by significant margins in both metrics. Compared
with the baseline _desktop mode_ , our approach achieves 21.05%, 10.42%, and
21.31% improvement in mIoU, overall satisfaction, and structure rationality.
According to the experimental findings, our method outperforms the other two
automated methods (_Direct mapping_ and _Desktop mode_). Our approach can also
produce comparable outcomes when compared to the ground truth without
incurring additional expenses.

Table 3. Average score of evaluation. * denote the statistical significance
$p-value<0.01$ with other two metrics.

Approach | mIoU |  | Structure
---
Rationality
| Overall
---
Satisfaction
| Vote
---
Best
Direct mapping | 0.07 | 2.44 | 2.67 | 1
Desktop mode | 0.19 | 3.38 | 3.55 | 7
Ground truth | 1 | 4.27 | 4.20 | 19
Ours | 0.23 | 4.10* | 3.92* | 15

To further analyze our method and the other two automatic methods (_Direct
mapping_ and _Desktop mode_), we plot the boxplots of the scores of these
three methods over _Structure Rationality_ and _Overall Satisfaction_ in user
study in Figure 11. In both box plots, the gap between our first and third
quartiles was lower than in _Desktop mode_ , indicating that our ratings are
more concentrated and stable. The maximums of _Desktop mode_ in both box plots
are higher than ours. There are a few cases that have been individually
optimized for _Desktop mode_ display, so the scores are particularly high,
which increases the maximum score of the whole. However, if the page is not
individually optimized, the effect of _Desktop mode_ is significantly lower
than that of our method. The scores of _Direct mapping_ are all significantly
lower than the other two, indicating that users generally do not accept this
conversion.

To understand the significance of the differences between the user study
results of baselines and our approach, we carry out the Mann-Whitney U test
(Fay and Proschan, 2010) on the overall satisfaction and structure rationality
with _Direct mapping_ and _Desktop mode_ , respectively. Since there are two
baselines, we carry out the test between our approach with each baseline
separately. The test results in Table 3 show that our tool can significantly
create better GUI conversion in overall satisfaction and structural
rationality from phone to TV than _Desktop mode_ (both $p-value<0.01$) and
_Direct mapping_ (both $p-value<0.01$).

Figure 11. Score distribution of the structure rationality and overall
satisfaction of three approaches

To better illustrate our results, Figure 12 lists examples of generated TV GUI
by different approaches. For the case above in Figure 12, the average
structure rationality for _Direct mapping_ , our tool and _Desktop mode_ are
1.8, 4.4, and 3.2. The average overall satisfaction for _Direct mapping_ , our
tool and _Desktop mode_ are 1.7, 4.0, and 3.6. For the case below in Figure
12, the average structure rationality for _Direct mapping_ , our tool and
_Desktop mode_ are 1.4, 4.6, and 1.5. The average overall satisfaction for
_Direct mapping_ , our tool and _Desktop mode_ are 2.0, 4.3, and 1.9.
Generally, for _Desktop model_ , only a few specific apps and pages are
manually optimized, so when testing with various types of pages, uncustomized
pages are original phone pages without optimization, resulting in low scores.
Faced with large TV screens, most cases are very blunt without considering the
design criteria of the current platform’s widgets and users’ usage habits.
_Direct mapping_ places all components as where they are on the phone,
resulting in a very poor user experience for TV users. Besides, it can only
convert part of GUI components, which has poor universality. This it is
generally not accepted by users.

Our approach sometimes may also generate inappropriate TV mapping, especially
for those non-standard GUI inputs. For the phone GUI in Figure 13, promotional
images of films in the second row do not show completely due to the size of
the mobile screen. In this case, UI Automator incorrectly provides us with the
name of these films with details in Bottom Tab Layout. Thus, our approach
fails to get the correct name of films due to the limitation of UI Automator,
just like the example shown in red boxes. As our approach gains confusing
hierarchies from UI Automator, it does not correctly identify the Bottom Tab
Layout. Therefore, the approach fails Bottom Tab Layout to convert it into a
TV Channel.

Figure 12. Conversion Examples by Direct Mapping, our tool, and Desktop Mode
Figure 13. An example of metadata data error

##  5\. Usefulness Evaluation

According to the findings of our empirical study in Section 2.1, more
developers prefer to redevelop the GUI pages of new TV apps. Even while
experienced developers can produce the matching TV GUI pages quickly, it still
affects the development efficiency and wastes some valuable resources in the
phone GUI. We carry out a user study to evaluate the usefulness of our
generated TV GUI for bootstrapping corresponding TV GUI design and
implementation by real-world developers.

###  5.1. Procedures

We recruit 6 participants who are all working in software companies and have
at least one-year Android GUI developing experience. Participants are required
to design and implement the corresponding conversion TV GUI by referring to
the given Phone GUI. We provide 6 phone GUIs which have covered main 12 phone
GUI groups. The official corresponding designed TV GUIs of 6 phone GUIs are
also collected for the subsequent satisfaction evaluation. Participants are
required to design and implement the layout skeleton and set component
properties, including the view type, size, order, and padding. Note that
participants are allowed to replace some component types with placeholders
without affecting the rendering and overall design.

The study consists of two groups of three participants: the experimental
group, consisting of three participants, is asked to proceed on the basis of
our tool, and the control group, which starts work from scratch on a TV app
design. The experiment group is allowed to use our tool to automatically
generate a draft TV GUI and update the generated source code to re-design TV
GUI directly. Each member in the experimental group learns in advance how to
use our tool to generate source code and render it on TV. Participants in both
groups have comparable development experience by pre-study their developing
background to ensure both groups have similar expertise in total. All
participants are only allowed to use Android Studio and Java to avoid bias and
have up to 20 minutes for each implementation. Three academic Ph.D. students
who are not involved in the study are asked to evaluate 6 participants’
results and rate their satisfaction on a five-point scale, with 5 being the
highest and 1 the lowest. The satisfaction metric(Kumar et al., 2011; Laine et
al., 2021; Zheng et al., 2019) is to evaluate the overall pleasing qualities
of this GUI page. The evaluators must determine if the layout, content, and UI
type of all UI components on this GUI page are appropriate. When rating,
raters do not know which TV GUI is developed by which group and the manual-
designed TV GUI by companies are used for the reference. Similar to the
accuracy evaluation, we use the Fleiss Kappa value (Fleiss, 1971) to measure
the agreement among the three raters. Fleiss Kappa values in the range of
[0.01, 0.20], (0.20, 0.40], (0.40, 0.60], (0.60, 0.80], and (0.8, 1]
correspond to the slight, fair, moderate, substantial, and almost perfect
agreement, respectively. If the Fleiss Kappa value is less than 0.8, the
divergent cases will be discussed, analyzed, and re-scored by the three raters
until the Fleiss Kappa value is greater than 0.8. We provide 6 participants
with the same development environment and resources and record the time it
takes participants to complete each TV GUI.

Figure 14. Distribution of cost time and satisfaction scores of control and
experimental groups

Table 4. Average cost time and satisfaction scores of control and experimental
groups. ** denote the statistical significance $p-value<0.01$.

Indicator | Control | Experiment
---|---|---
Cost Time (min) | 16.28 | 11.56**
Average Score | 1.66 | 3.35**

###  5.2. Results

The box plot in Figure 11 and the average score in Table 14 show that the
experimental group implements TV GUI conversion faster (average of 11.56
minutes) with a higher satisfaction score (average of 3.35) than the control
group (average of 16.28 minutes and 1.66 satisfaction score). Members of the
experimental group use the DSL to design the GUI, but members of the control
group write actual code to design the GUI. Therefore, their working
implementations are different, and the outcomes of the control group members
are closer to the actual app implementation. Theoretically, different working
implementations could lead to bias in the results, but all invited
professionals have at least one year of Android development experience. They
master the Android GUI development procedure. Some developers use placeholders
to replace some UI components, allowing them to focus on developing the GUI
rather than completing the code logic and syntax. While the working
implementations of the two groups may not be equal, the efforts of redesigning
and constructing an adequate TV GUI are the decisive factor in determining the
outcomes of the two groups experiments. Our tool successfully and effectively
assists developers in developing more suitable TV GUIs faster, taking into
account the apparent time difference between 11.56 minutes and 15.6 minutes on
average. In experiments, two participants in the control group fail to finish
at least one GUI conversion within 20 minutes. However, everyone in the
experimental group completed the conversion within 20 minutes and had time to
personalize the GUIs. This results in significantly higher satisfaction scores
than the control group (an average score increase of 1.64).

Figure 15 shows an example of two TV GUI conversions from the experimental and
control group. Note that we only evaluate the structure and overall soundness
of the GUI, so we do not evaluate the content in the participant’s GUI,
allowing placeholders. When designing GUI, members in the experimental group
can refer to the output layout of our tool, and can customize the TV GUI on
this basis. Our tool is an excellent inspiration for the experimental group,
saving them time in designing and developing the TV GUI and giving them a
bottom line for their work. We also carry the Mann-Whitney U test on the cost
time and satisfactory scores. The test results suggest that our tool can
significantly help the experimental group convert phone GUI
faster($p-value<0.01$) and create better TV GUI ($p-value<0.01$).

A participant in the experimental group commented on our tool that “The
automatic conversion results provided by the tool can give me good development
design guidance and hints, greatly improving my development efficiency”.
Overall, the user study results provide preliminary evidence of the usefulness
of our tool. The majority of participants think that the GUI DSL we built is
simple to comprehend and use. They said that our GUI DSL is a useful solution
to address the present issue of GUI code reuse caused by the version
differences between mobile phone systems and television systems. Our GUI DSL
also makes it simple for designers without any programming background to
collaborate on projects. Participants also pointed out some flaws in our
existing approach. It is not appropriate for converting real-time demanding
apps like games since the generated GUI needs to be re-rendered by the client
app on the TV. Because the image on the mobile GUI isn’t designed with a
large-screen TV, sometimes the image will seem stretched on the TV GUI after
conversion, which can negatively impact the user experience. To address these
issues, we will design more efficient GUI rendering technology solutions and
design the display of image resources after screen size conversion in future
work.

Figure 15. An example of experimental and control group

##  6\. Threats to validity

Potential threats to validity in our user study for performance evaluation
primarily stem from subjectivity or bias inherent in participants’ abilities
and backgrounds. Similarly, the experiment on GUI grouping accuracy could be
influenced by individual interpretations of ‘logically related UI components.’
We have taken measures to mitigate these threats: participants are all GUI
development professionals with diverse skills and experiences. We’ve provided
comprehensive examples of ’logically related’ components pre-experiment to
ensure a shared understanding. We refrain from revealing which results are
ours or the baselines’ during the user study. Moreover, results are
objectively assessed by comparing the mIoU to the redesigned native TV GUI,
with tool performance evaluated via a blend of user study and automated
evaluation results. In the usefulness evaluation, we primarily relied on
‘satisfaction score’ and implementation time as the evaluative metrics for our
approach and the baselines. This may not fully encapsulate all elements
pertinent to GUI usability and aesthetics. However, while the ‘satisfaction
score’ is central to our usefulness evaluation, we employ other metrics in
separate experiments designed to measure different aspects of our tool.
Collectively, these various metrics corroborate the effectiveness of our
proposed tool, contributing to a comprehensive assessment of its performance.
To broaden the scope of the tool’s efficacy evaluation, we intend to introduce
additional valid metrics in future work.

The existence of a small number of online service apps that leverage web
technologies for cross-platform adaption poses an additional threat to the
validity of user studies. Some cross-platform frameworks, such as Flutter
(Flutter, 2021) and React JS (React, 2021), claim support for Android TV
adaptation. However, we discovered that there are no comparable apps for
Android TV. Therefore, we did not collect any valid online service-based
phone-TV app pairs. There may be new GUI features in online service apps. Thus
we will continue to collect online service apps in our future research and
study their features to improve our algorithm.

The major internal validity of the conversion pipeline is that some GUI
components are difficult to classify and hence challenging to convert
appropriately according to the GUI category. In the GUI grouping phase, there
may be GUI pages that do not follow the Android GUI design principles,
resulting in mistakes in GUI groups, which may cause errors in subsequent
group classification and mapping. Therefore, pipeline errors may be introduced
throughout these steps of our semi-automated TV GUI generation pipeline due to
the possibility of an error during GUI recognition and grouping. Errors in the
GUI recognition and GUI grouping phrases may ultimately result in
unsatisfactory overall performance. Current data collection tools (UI
Automator) cannot accurately obtain metadata of some third-party GUI and
$WebView$ views, which further makes it difficult to classify. To mitigate the
validity, we use Grid Layout as the default template to map these unrecognized
GUI groups, which can be seen in Table 1. Grid Layout templates can ensure
that the information on mobile phones is not lost after mapping and basically
meet the user experience.

The major threat to the external validity of the conversion pipeline is the
fragmentation of Android devices. The screen, OS version, and UI styles of
Android devices are markedly different and difficult to unify. To mitigate
this issue, we design language- and platform-free GUI DSL for code synthesis.
The DSL translator pre-installed on the relevant TV converts the DSL into the
GUI source code based on the TV’s properties. Improper code translation may
potentially result in a poor final GUI conversion. Consequently, we define the
TV GUI libraries in advance and employ quality-checked GUI libraries to
alleviate this potential threat.

The potential generalization vulnerability of the UI grouping and conversion
procedures based on heuristic rules is also a threat to validity. GUI
developers may not follow the prevalent UI layout guidelines when designing
GUI pages, which may cause our grouping heuristics to become ineffective. To
mitigate the threat and ensure that our approaches are applicable to all
existing phone-TV GUI pair types, we evaluate our heuristics in external real-
world apps in Section 4.1. Google advises that the Android TV app’s UI be
designed with a card-like layout, thus we implement a default grid layout-
based template for dealing with the unexpected rare UI types. Additionally, we
provide OR constraints for Android TV. These constraints are more flexible
than rules and automatically calculate TV GUI layouts that correspond to the
present GUI. The use of OR constraints broadens the generalizability of our
method. Our approach also proposes to convert GUI to a cross-platform GUI DSL
that allows programmers to further customize the source code for various
platforms and versions. In exceptional circumstances, the developer can
rewrite a portion of the source code to conform to the current GUI layout
specifications based on the generated DSL. To improve the efficiency of UI
grouping, we plan to collect the grouped UI components on mobile devices and
TVs, train a deep learning model, and allow the deep learning model to
automatically match GUI components of the same domain.

##  7\. Related Work

###  7.1. GUI implementation

Implementing a GUI focuses on making the GUI work with proper layouts and
widgets of a GUI framework. Nguyen and Csallner (Nguyen and Csallner, 2015)
reverse engineer the UI screenshots by rule-based image processing method and
generate GUI code. They support only a small set of most commonly used GUI
components. More powerful deep learning-based methods (Beltramelli, 2018; Chen
et al., 2018; Moran et al., 2018a) have been recently proposed to leverage the
big data of automatically collected UI screenshots and corresponding code.
Some recent works explore issues between UI designs and their implementations.
Moran et al. (Moran et al., 2018b) check if the implemented GUI violates the
original UI design by comparing the image’s similarity with computer vision
techniques. The follow-up work (Moran et al., 2018c) further detects and
summarizes GUI changes in evolving mobile apps. Similarly, the semantic vector
for the UI design from our work can help detect the inconsistency among UI
designs within the same app.

Different from those works on GUI implementation which are highly related to
conventional GUI development, we are targeting at specifically GUI projecting
from small-screen mobile phones to the corresponding one on large-screen TVs.

###  7.2. GUI component grouping

There are some similar components of clustering and page segmentation
algorithms in web page analysis. Yandrapally et al. (Yandrapally et al., 2020)
propose a near-duplication detection algorithm to study near-duplication
components in web pages. They characterize and merge functional near-
duplicates by summarizing categories in existing web pages. Crescenzi et al.
(Crescenzi et al., 2005) propose a structural abstraction clustering algorithm
for web pages that groups web pages based on this abstraction. To assess end-
to-end web tests, Yandrapally et al. (Yandrapally and Mesbah, 2021) utilize
VIPS (Cai et al., 2003) for web page segmentation and XPaths of web elements
inside these fragments for establishing their equivalence. VIPS (Cai et al.,
2003) is a popular page segmentation, it proposes an automatic top-down, tag-
tree independent approach to detect web content structure. Mahajan et al.
(Mahajan et al., 2018, 2021) design a clustering technique for web elements
that are based on a combination of visual aspects and DOM-based XPath
similarity.

Although our study focuses on the grouping and segmentation of mobile GUIs,
their work on the clustering of web components enlightens us, and we
incorporate their insights into mobile GUI grouping.

###  7.3. GUI migration across platforms

Due to the difficulty of GUI migration across different platforms, very few
related works are carried out for solving this problem. Pihlajam (Pihlajamäki,
2016) observed multiple games across web and mobile platforms for summarizing
several UI patterns for user interface adaptation. Wong et al. (Wong et al.,
2008) developed a scalable GUI system that dynamically transforms platform-
specific GUI widgets migrated within an application between any of a plurality
of heterogeneous device platforms. Verhaeghe et al. (Verhaeghe et al., 2019)
developed a set of rules for GUI Migration using MDE from GWT to Angular 6.

Although these studies explore the mapping (or partial mapping) between
different platforms and programming languages, none of these works are
investigating app GUI adaptation between phones and TV. Instead of the white-
box migration or layout migration, our study focuses more on black-box GUI
adaptation without the source code of the original GUI on the phone. That
black-box characteristics and significant differences in screen sizes motivate
us to develop a brand new approach to bridge the gap between phone and TV GUI.
Besides, time and resource consumption must be considered because of phones’
current hardware limitations.

###  7.4. Practical tools in industry

Finally, it is worth mentioning some related non-academic projects. There are
many third-party libraries or frameworks which support cross-platform
adaptation such as React.js (React, 2021), Flutter (Flutter, 2021), and also
default Android development (Android TV, 2021). Although these frameworks can
cover multiple platforms such as smartphones, desktops, and tablets, TV is
rarely covered due to its ultra-large screen. Developers have to commit much
additional effort to design new layouts that can be easily understood from 10
feet away and provide navigation that works with just a directional pad and a
select button to make their app successful on TV devices. That is one reason
why few apps support GUI adaptation, and the small number of smart TV users
further discourages app developers.

Samsung supports screen mirroring from the Samsung device to the TV including
photos, videos, presentations, and games (Dex, 2021). There are also other
similar connections between smartphones and TV such as Chromecast(chromecast,
2021), Xiaomi TV stick (miTV, 2021) or wired connection via HDMI, etc. Most of
them are working well in only video projection or some customized apps, and
they require additional support from the TV side. Directly showing phone GUI
on TV will bring many rendering issues such as small components, large black
margins, unreasonable interaction, etc. To overcome those issues, we propose
an automated approach to project the phone GUI to the TV on the run time.

##  8\. Conclusion

At the moment, adaptive technologies between phone and TV GUI are unable to
fulfill the demands of app developers, and the cost of developing and
maintaining new applications is prohibitive, therefore building an automated
GUI conversion tool from phone to TV is challenging but worthwhile work for
developers. An automated approach for converting phone GUI to TV GUI is
presented in this study. Before proposing our approach, we carry out empirical
studies to explore how many current apps support TV displays and how current
apps convert phone GUI to TV GUI. Our tool consists of four integral stages:
GUI components grouping, template matching, layout optimization, and DSL for
code synthesis. Finally, we compile generated GUI DSL to source code for
rendering the final TV GUIs. Our approach offers obvious benefits over
existing mainstream technologies, according to an automated evaluation in 589
valid phone-TV GUI pairs and a user study from 20 Android professionals.
Besides, a pilot user study also illustrates the usefulness of our tool for
app developers.

In the future, we will keep improving our algorithm for generating mapping
GUIs from phone to TV. With more and more apps developed for supporting TV, we
will construct a large parallel corpus of TV-phone GUI pairs. Based on that
data, we will develop an end-to-end machine learning algorithm that will be
more generalized than the current approach. On the other hand, we will extend
our tool to other platforms such as smartwatches, tablets, and vehicle
screens.

## References

  * (1)
  * adaptiveUI (2022) adaptiveUI 2022.  Adaptive GUI in Android.  https://developer.android.com/guide/topics/large-screens/support-different-screen-sizes.
  * andriodUI (2022) andriodUI 2022.  andriodUI.  https://developer.android.com/guide/topics/ui.
  * Android TV (2021) Android TV 2021.  Android TV.  https://developer.android.com/tv.
  * androidGuidelines (2022) androidGuidelines 2022.  androidGuidelines.  https://developer.android.com/guide.
  * androidStudio (2022) androidStudio 2022.  androidStudio.  https://developer.android.com/studio.
  * AndroidTVApp (2021) AndroidTVApp 2021.  AndroidTVApp.  https://9to5google.com/2020/08/10/android-tv-growth-apps-users-operators/.
  * AndroidTVDe (2021) AndroidTVDe 2021.  AndroidTVDe homepage.  https://developer.android.com/training/tv/start/start.
  * Beltramelli (2018) Tony Beltramelli. 2018\.  pix2code: Generating code from a graphical user interface screenshot. In _Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems_. 1–6.
  * Bhuttoo et al. (2017) Vivek Bhuttoo, Kamlesh Soman, and Roopesh Kevin Sungkur. 2017\.  Responsive design and content adaptation for e-learning on mobile devices. In _2017 1st International Conference on Next Generation Computing Applications (NextComp)_. IEEE, 163–168.
  * Cai et al. (2003) Deng Cai, Shipeng Yu, Ji-Rong Wen, and Wei-Ying Ma. 2003\.  Vips: a vision-based page segmentation algorithm.  (2003).
  * Cai et al. (2020) Tianqin Cai, Zhao Zhang, and Ping Yang. 2020.  Fastbot: A Multi-Agent Model-Based Test Generation System Beijing Bytedance Network Technology Co., Ltd.. In _Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test_. 93–96.
  * Chen et al. (2018) Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018.  From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In _Proceedings of the 40th International Conference on Software Engineering_. 665–676.
  * Chen et al. (2019) Qiuyuan Chen, Han Hu, and Zhaoyi Liu. 2019.  Code summarization with abstract syntax tree. In _International Conference on Neural Information Processing_. Springer, 652–660.
  * chromecast (2021) chromecast 2021.  chromecast homepage.  https://store.google.com/product/chromecast?hl=en-GB.
  * Crescenzi et al. (2005) Valter Crescenzi, Paolo Merialdo, and Paolo Missier. 2005\.  Clustering web pages based on their structure.  _Data & Knowledge Engineering_ 54, 3 (2005), 279–299.
  * dangbei (2021) dangbei 2021.  dangbei homepage.  http://us.dangbei.com/.
  * De Moura and Bjørner (2008) Leonardo De Moura and Nikolaj Bjørner. 2008.  Z3: An efficient SMT solver. In _International conference on Tools and Algorithms for the Construction and Analysis of Systems_. Springer, 337–340.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv:1810.04805 [cs.CL]
  * Dex (2021) Dex 2021.  Screen Mirroring from your Samsung device to your TV.  https://www.samsung.com/us/support/answer/ANS00062585/.
  * Fay and Proschan (2010) Michael P Fay and Michael A Proschan. 2010.  Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules.  _Statistics surveys_ 4 (2010), 1.
  * Fleiss (1971) Joseph L Fleiss. 1971\.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ 76, 5 (1971), 378\.
  * Flutter (2021) Flutter 2021.  Flutter - Beautiful native apps in record time.  https://flutter.dev/.
  * fragment (2022) fragment 2022.  fragment.  https://developer.android.com/guide/fragments.
  * Goldschmidt (1994) Gabriela Goldschmidt. 1994\.  On visual design thinking: the vis kids of architecture.  _Design studies_ 15, 2 (1994), 158–174.
  * Google Play TV (2023) Google Play TV 2023.  Google Play TV.  https://play.google.com/store/apps?device=tv.
  * GoogleMaterial (2021) GoogleMaterial 2021.  GoogleMaterial.  https://material.io/design.
  * googleplay (2021) googleplay 2021.  google play homepage.  https://play.google.com/store.
  * googleTV (2021) googleTV 2021.  googleTV.  hhttps://play.google.com/store.
  * GUI state (2022) GUI state 2022.  GUI state.  https://developer.android.com/topic/libraries/architecture/saving-states.
  * guiBuild (2022) guiBuild 2022.  guiBuild.  https://developer.android.com/training/basics/firstapp/building-ui.
  * Hore and Ziou (2010) Alain Hore and Djemel Ziou. 2010.  Image quality metrics: PSNR vs. SSIM. In _2010 20th international conference on pattern recognition_. IEEE, 2366–2369.
  * huaweiDesk (2021) huaweiDesk 2021.  huaweiDesk homepage.  https://consumer.huawei.com/en/support/content/en-us15788618/.
  * Jaccard (1912) Paul Jaccard. 1912\.  The distribution of the flora in the alpine zone. 1\.  _New phytologist_ 11, 2 (1912), 37–50.
  * Jiang et al. (2019) Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger. 2019.  ORC layout: Adaptive GUI layout with OR-constraints. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Jiang et al. (2020) Yue Jiang, Wolfgang Stuerzlinger, Matthias Zwicker, and Christof Lutteroth. 2020.  ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Joorabchi et al. (2013) Mona Erfani Joorabchi, Ali Mesbah, and Philippe Kruchten. 2013\.  Real challenges in mobile app development. In _2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_. IEEE, 15–24.
  * Kumar et al. (2011) Ranjitha Kumar, Jerry O Talton, Salman Ahmad, and Scott R Klemmer. 2011.  Bricolage: example-based retargeting for web design. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 2197–2206.
  * Laine et al. (2021) Markku Laine, Yu Zhang, Simo Santala, Jussi PP Jokinen, and Antti Oulasvirta. 2021.  Responsive and Personalized Web Layouts with Integer Programming.  _Proceedings of the ACM on Human-Computer Interaction_ 5, EICS (2021), 1–23.
  * Leanback (2021) Leanback 2021.  https://developer.android.com/jetpack/androidx/releases/leanback.
  * Li et al. (2017) Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017\.  Droidbot: a lightweight ui-guided test input generator for android. In _2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)_. IEEE, 23–26.
  * Liu et al. (2018) Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018\.  Learning design semantics for mobile apps. In _Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology_. 569–579.
  * Lowdermilk (2013) Travis Lowdermilk. 2013\.  _User-centered design: a developer’s guide to building user-friendly applications_.  ” O’Reilly Media, Inc.”.
  * Machiry et al. (2013) Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013\.  Dynodroid: An input generation system for android apps. In _Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering_. 224–234.
  * Mahajan et al. (2018) Sonal Mahajan, Abdulmajeed Alameer, Phil McMinn, and William GJ Halfond. 2018.  Automated repair of internationalization presentation failures in web pages using style similarity clustering and search-based techniques. In _2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 215–226.
  * Mahajan et al. (2021) Sonal Mahajan, Abdulmajeed Alameer, Phil McMinn, and William GJ Halfond. 2021.  Effective automated repair of internationalization presentation failures in web applications using style similarity clustering and search-based techniques.  _Software Testing, Verification and Reliability_ 31, 1-2 (2021), e1746.
  * Mendoza (2013) Adrian Mendoza. 2013\.  _Mobile user experience: patterns to make sense of it all_.  Newnes.
  * miTV (2021) miTV 2021.  Mi TV Stick homepage.  https://www.mi.com/global/mi-tv-stick/.
  * Moran et al. (2018a) Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018a.  Machine learning-based prototyping of graphical user interfaces for mobile apps.  _IEEE Transactions on Software Engineering_ 46, 2 (2018), 196–221.
  * Moran et al. (2018b) Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018b.  Automated reporting of GUI design violations for mobile apps. In _Proceedings of the 40th International Conference on Software Engineering_. 165–175.
  * Moran et al. (2018c) Kevin Moran, Cody Watson, John Hoskins, George Purnell, and Denys Poshyvanyk. 2018c.  Detecting and summarizing GUI changes in evolving mobile apps. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. 543–553.
  * Neil (2014) Theresa Neil. 2014\.  _Mobile design pattern gallery: UI patterns for smartphone apps_.  ” O’Reilly Media, Inc.”.
  * Nguyen and Csallner (2015) Tuan Anh Nguyen and Christoph Csallner. 2015.  Reverse engineering mobile application user interfaces with remaui (t). In _2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 248–259.
  * Pihlajamäki (2016) Eveliina Pihlajamäki. 2016\.  _From Desktop to Mobile: UI Patterns for User Interface Adaptation in Games_.  Master’s thesis.
  * Ponomarenko et al. (2008) N Ponomarenko, V Lukin, K Egiazarian, Jaakko Astola, Marco Carli, and Federica Battisti. 2008\.  Color image database for evaluation of image quality metrics. In _2008 IEEE 10th workshop on multimedia signal processing_. IEEE, 403–408.
  * React (2021) React 2021.  React - A JavaScript library for building user interfaces.  https://reactjs.org/.
  * Rountev and Yan (2014) Atanas Rountev and Dacong Yan. 2014.  Static reference analysis for GUI objects in Android software. In _Proceedings of Annual IEEE/ACM International Symposium on Code Generation and Optimization_. 143–153.
  * Seaman (1999) Carolyn B. Seaman. 1999\.  Qualitative methods in empirical studies of software engineering.  _IEEE Transactions on software engineering_ 25, 4 (1999), 557–572.
  * Sgier (2012) Lea Sgier. 2012\.  Qualitative data analysis.  _An Initiat. Gebert Ruf Stift_ 19 (2012), 19–21.
  * TVDesign (2022) TVDesign 2022.  TVDesign.  https://tv.withgoogle.com/design-principles/designing-for-tv.html.
  * uiautomator (2021) uiautomator 2021.  uiautomator homepage.  https://developer.android.com/training/testing/ui-automator.
  * uiautomator2 (2021) uiautomator2 2021.  uiautomator2.  https://play.google.com/store.
  * Verhaeghe et al. (2019) Benoît Verhaeghe, Anne Etien, Nicolas Anquetil, Abderrahmane Seriai, Laurent Deruelle, Stéphane Ducasse, and Mustapha Derras. 2019.  GUI migration using MDE from GWT to Angular 6: An industrial case. In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 579–583.
  * Wong et al. (2008) Hoi Lee Candy Wong, Hao-hua Chu, Masaji Katagiri, Yu Song, and Shoji Kurakake. 2008\.  Transformation of platform specific graphical user interface widgets migrated between heterogeneous device platforms.  US Patent 7,392,483.
  * Xie et al. (2020) Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020.  UIED: a hybrid tool for GUI element detection. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1655–1659.
  * Yandrapally and Mesbah (2021) Rahulkrishna Yandrapally and Ali Mesbah. 2021.  Mutation analysis for assessing end-to-end web tests. In _2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 183–194.
  * Yandrapally et al. (2020) Rahulkrishna Yandrapally, Andrea Stocco, and Ali Mesbah. 2020\.  Near-duplicate detection in web app model inference. In _Proceedings of the ACM/IEEE 42nd international conference on software engineering_. 186–197.
  * Zhang et al. (2021) Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. 2021\.  Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Zhao et al. (2021) Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021\.  GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 748–760.
  * Zheng et al. (2019) Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. 2019\.  Content-aware generative modeling of graphic design layouts.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019), 1–15.