# ScreenQA: Large-Scale Question-Answer Pairs
Over Mobile App Screenshots

Yu-Chung Hsiao, Fedir Zubach, Maria Wang, Jindong (JD) Chen
Google Research
{yuchsiao, fedir, mariawang<EMAIL_ADDRESS>
###### Abstract

We present a new task and dataset, ScreenQA, for screen content understanding
via question answering. The existing screen datasets are focused either on
structure and component-level understanding, or on a much higher-level
composite task such as navigation and task completion. We attempt to bridge
the gap between these two by annotating 80,000+ question-answer pairs over the
RICO dataset in hope to benchmark the screen reading comprehension capacity.

##  1 Introduction

Mobile app screenshots have been analyzed using machine learning from multiple
aspects. These analyses range from pixel level understanding, e.g., layout
structural analyses, UI issue detection and correction Li et al. (2022), to UI
element semantics, e.g., icon recognition, button action prediction Sunkara et
al. (2022), to even higher-level functional analyses such as accessibility
support Li et al. (2020c), screen description Wang et al. (2021), and screen
type classification Deka et al. (2017). Comparatively, the content
understanding aspect is relatively understudied. By content, we mean the
information displayed on the screen to convey and satisfy the purpose of using
the app. Examples include star ratings from restaurant reviews, messages from
chat apps, cuisine ingredients from recipe apps, flight status and in-flight
amenities from travel planner apps, etc. Having this capacity of understanding
is important for two reasons: First, the sole reason for many apps and app
categories to exist is to satisfy users’ information need, e.g., weather, map
navigation, and news apps. Second, for task completion111Also referred to as
automation or app control., which requires the eyes-free agent capacity, the
two types of screen understandings — content and action understanding — are
inseparable in order to carry out a task successfully. Without knowing a
screen state properly, a machine learning agent is unable to self-assess if
the action is performed as expected, or unable to provide sufficient feedback
to the user to achieve true eyes-free user experience. More intrinsically,
from a pure research perspective, we are interested in knowing the limit of
machine screen content understanding222This term is analogous to machine
reading comprehension from natural language processing. and what constitutes
the challenges, given that app screenshots are entirely human artifacts made
for convenient comprehension.

Accordingly, we annotated the RICO dataset Deka et al. (2017) with more than
80,000 question-answer pairs, referred to as Screen Question Answering, or, in
short, ScreenQA annotations later in this work, and we will release the
dataset in the public domain.333URL will be released in a subsequent update of
this manuscript. The ScreenQA task requires an agent to answer a user’s
question by selecting one or multiple UI elements from the given screenshot,
as will be formulated in Section 2. Question answering is employed as a
touchstone to sparsely444Because questions are not exhaustively asked against
a given screenshot. verify the quality of screen content understanding. To the
best of our knowledge, this is the first large-scale questions answering
dataset over mobile app screenshots, and the first one to be publicly
available. Much inspired by the SQuAD dataset Rajpurkar et al. (2016), we
hope, by releasing this set of annotations, to encourage the community to
advance technologies toward better screen content understanding. We anticipate
that the advance of such technologies will benefit beyond just the screen UI
and the human computer interaction (HCI) domains. As we will discuss in
Section 3, other vision-language related multimodal domains share similar
challenges with different emphases on respective modalities and contexts.
Comparatively, ScreenQA is language and layout heavy, but it also includes
visual ingredients such as icons and symbols as concise representations in
place of texts, to declutter the UI. It may also include images or art designs
that pose challenges to language centric machine learning agents.

The remaining paper is organized in the following way: Section 2 formulates
the problem, including the problem description and the evaluation metrics. We
discuss relevant prior datasets and annotations in Section 3 to put this work
into perspective. Section 4 describes our annotation method. The annotations
are then analyzed in Section 5 to provide readers both the qualitative and
quantitative views. The paper is concluded in Section 6 with a summary and a
remark on future works.

##  2 Problem Setting

(a) Ambiguous UI element boundaries. Three possibilities are annotated.

(b) Answers to “What’s the temperature on Saturday?”

(c) Semantic groups are the basic unit for ordering, not element coordinates.

Figure 1:  ScreenQA examples. (a) Determining UI elements can be sometimes
ambiguous. Yet, a human can still answer screen questions without the aid of
its view hierarchy. Hence, we do not require the view hierarchy as the input
to the task, instead, employing a more flexible evaluation metric to mitigate
the impact of this ambiguity, introduced in Section 2.3. (b) When asking a
question, a user is assumed not knowing how many answers are relevant on the
screen. All relevant information should be included and ordered by relevance
and the common reading order. (c) Sometimes answers come as groups. In this
case, “Deep Squat, 3 sets, 15x” should be grouped together and, by the reading
order, followed by “Lunge, 3 sets, 10x” when being asked about the first two
movement of deep squat. Simply following UI element coordinates would render
the answer incomprehensible, e.g., “Deep Squat, Lunge, 3 sets, 15x, 3 sets,
10x”. Use semantic groups as the ordering unit instead.

We state the problem and define the evaluation metrics in this section.

###  2.1 Problem statement

The ScreenQA task requires an agent to answer a user’s question by selecting
relevant UI elements from a given single screenshot. When it comes with
multiple relevant UI elements, a list of such UI elements whose contents
minimally satisfy the question should be selected and ranked in descending
order of relevance to the question, if applicable, or following the common
reading order by semantic groups, as will be described in Section 2.2. This
assumes that answers are directly selectable from the screen and logical
reasoning and calculation are not needed. If the screenshot does not contain
the answers to the question, the agent should respond with “no answer.” This
is summarized in Task 1.

Task 1 ScreenQA

Input: a question $Q$ and a screenshot $S$
Output: an answer list $A$ of UI elements selected from $S$ such that their
contents minimally satisfy $Q$. The order of $A$ is further required to be

  * •

Ranked in descending order of relevance to $Q$, if applicable.

  * •

Otherwise, following the common reading order by semantic groups.

If no contents in $S$ can satisfies $Q$, then returns an empty list $A$.

###  2.2 Properties and terminologies

The mobile app UI comes with some nuances. It is worth mentioning a few
properties below.

  * •

View hierarchy, or the structural representation used to render the screen, is
not required in Task 1, to be consistent with the human annotation process in
Section 4. View hierarchy usually provides useful UI element candidates, but
it may not always be reliable, for example, when using WebView or screen
overlays. In such cases, a human annotator can still answer screen questions
entirely from pixels without an issue, so we want to benchmark similarly. We
leave the choice of dependency on view hierarchies to the modelers and, hence,
do not require it. However, this comes with an ambiguity for UI element
boundaries. See an example in Figure 1(a). We devise a more flexible answer
matching process to mitigate such an impact, as will be discussed in Section
2.3.3.

  * •

Avoid question answering over long paragraphs. Although it is permissive by
Task 1, we discourage annotators from asking such questions during the
annotation process. For ScreenQA, we want to focus on learning the
relationships between text segments arranged two-dimensionally on the screen,
and leave the long paragraph question answering, which investigates the
relationships between words, to the traditional NLP domain.

  * •

Avoid logical reasoning. This task assumes answers can directly be extracted
from the screenshot without reasoning, entailment, counting and comparing
numbers. This further exclude yes/no and why questions if not explicitly
displayed on the screen. The reason is that we want to separate “able to read”
and “able to reason” and focus on the former first without generating an over
challenging dataset. A few such excluded examples are: counting items, asking
about the weather a few days from now, what are the items cheaper than X
dollars, etc.

  * •

Ordered by relevance. The task is designed to enable the eyes-free user
experience. That is, a user may not be fully aware of how many relevant
answers are displayed on the screen. For example, in Figure 1(b), when a user
asks “What’s the temperature on Saturday?”, there are actually two
temperatures, high and low, for each day and two Saturdays on the screen. In
this case, the two temperatures should just follow the reading order, and the
two Saturdays follow the relevance order as a user usually refers to the
upcoming Saturday. For a well-designed mobile app, these two usually overlap
well and we do not expect a large ambiguity here.

  * •

Reading order by semantic groups. Sometimes some UI elements are designed as
semantic groups and should be referred to together to keep their semantic
meaning. For example, in Figure 1(c), when a user asks “What are the first two
movements of deep squat?”, then the answer should be “Deep Squat, 3 sets,
15x”, followed by “Lunge, 3 sets, 10x”. In other words, the common reading
order should be based on semantic groups as the unit, rather than simply
sorted by the coordinates of UI elements.

Note that we set up the problem this way strategically in order to prioritize
its solvability considering the progress of current technologies. However,
practically, long vs. short texts and retrieval vs. reasoning are naturally
mixed together in the daily usage of mobile apps. We will leave this type of
composite problems to the future work.

###  2.3 Evaluation metrics

We consider two types of metrics: 1) Average normalized Discounted Cumulative
Gain (Average nDCG) Järvelin and Kekäläinen (2002), which is commonly used in
information retrieval and ranking systems, and 2) Average F1 score, which has
been employed in closed-domain question answering problems, such as the SQuAD
dataset Rajpurkar et al. (2016).

One major difference between our metrics described below and the commonly used
definitions is the unit of predictions. We use the element in the answer list
$A$, described in Task 1, as the unit to determine a hit or a miss for both
metrics. Besides, as UI elements can be ambiguous as mentioned in Section 2.2,
we will describe an answer matching algorithm that mitigate such an impact in
Section 2.3.3.

####  2.3.1 Average nDCG

We use a variant of nDCG that allows varying positions (number of returns) as
opposed to a typical fixed position. This is because, unlike the search
problem, which is fair to evaluate, say, top-10 retrieved documents across
queries, ScreenQA can have different needs of answer lengths across different
questions. For example, a question like “what is the address of the store”
expects a single returned result. A question like “what are the login
options?” expects an enumeration of options on the screen that easily go
beyond five. Accordingly, we allow $v$arying positions as follows: Given a
1-indexed list $A$, which is the predicted answer for the screen-question pair
$(S,Q)$, and a ground truth answer list $A_{g}$ for $(S,Q)$, the Discounted
Cumulative Gain at $v$arying positions (DCGv) is computed by:

$\mbox{DCG}_{v}=\sum^{\|A\|}_{i=1}\frac{r_{i}}{\log_{2}{(i+1)}},$ (1)

where $\|\cdot\|$ is the size of the list argument, $r_{i}$ is the relevance
score for the $i$-th item of $A$. We assign the relevance score 1 for a hit
and 0 for a miss compared with the ground truth $A^{g}$. The corresponding
Ideal Discounted Cumulative Gain (IDCGv) is computed by:

$\mbox{IDCG}_{v}=\sum^{\|A^{g}\|}_{i=1}\frac{1}{\log_{2}{(i+1)}}.$ (2)

The nDCGv is then

$\mbox{nDCG}_{v}=\frac{\mbox{DCG}_{v}}{\mbox{IDCG}_{v}}.$ (3)

Note that nDCGv is still between 0 and 1, hence, convenient for comparing
scores and computing the average.

For a dataset of $N$ examples, each of which is indexed by $i$ and has a
predicted answer $A_{i}$ and $K$ ground truth annotations $A^{g}_{i,j=1\dots
K}$, the average nDCGv can be computed by

$\mbox{avg}(\mbox{nDCG}_{v})=\frac{1}{N}\sum_{i=1}^{N}\mbox{max}_{j}[\mbox{nDCG}_{v}(A_{i},A^{g}_{i,j})].$
(4)

We choose a variant nDCG as the metric because 1) we want to measure the
quality of the ranking. For example, if one incorrectly predicts the result
from the first to the third position, the discount factor brings down the
score from 1.0 to only 0.5. 2) nDCG has an orthogonal design, which is easier
to tweak toward a specific need than the mean average precision (mAP) metric.
For example, one can choose to discount faster or slower by changing the base
of the denominator $\log_{2}(i+1)$ and can choose to penalize irrelevant
predictions by assigning negative scores. Mean reciprocal rank (MRR) and mAP
are much less controllable in these two aspects.

One known drawback of nDCG is that it does not naturally penalize excessive
predictions after the last relevant item. We therefore use the average F1
score as a complementary view of the agent performance.

####  2.3.2 Average F1

Similar to the definition in SQuAD, the average F1 score is computed as below,
following the same notation as in (4):

$\mbox{avg}(\mbox{F}_{1})=\frac{1}{N}\sum_{i=1}^{N}\mbox{max}_{j}[\mbox{F}_{1}(A_{i},A^{g}_{i,j})].$
(5)

Note that F1 does not concern ranking. For some cases, such as enumeration
questions, this is desirable, as the ranking order is merely the reading
order, even if the item order is permuted, the answer quality is in general
not compromised, hence, reasonable to be assigned the same evaluation score.
On the contrary, if relevance ranking is important, such as in Figure 1(b),
then nDCG provides a better view. Since both types of questions exist in the
ScreenQA annotations, it is more complete to evaluate against both metrics.
Also note that the unit of precision and recall computation is based on items
in $A$, unlike SQuAD, which uses words as the unit instead. We describe how to
compare items in an answer $A$ with the ground truth $A^{g}$ in the next
section.

####  2.3.3 Answer matching

As mentioned in Section 2.2, the segmentation of UI elements provided in the
predicted answer list $A$ may not coincide with the UI elements in the ground
truth list $A^{g}$. Yet, if the overall answers are the same, the segmentation
difference should not affect the evaluation score. Therefore, we use the
following empirical procedure to mitigate such an impact, using an illustrated
example (each capitalized character is a word token):

$\displaystyle A$ $\displaystyle=["AB","B","BB","CBA"]$ (6) $\displaystyle
A^{g}$ $\displaystyle=["AB","BC","AB"],$ (7)

  1. 1.

Concatenate items in $A$ into a single item list $A^{c}=[``ABBBBCBA"]$.

  2. 2.

Iterate through each $g\in A^{g}$ and check if $g$ is contained in any item in
$A^{c}$. If so, mark the $g$ as HIT (✓) and mark the corresponding matched
word token in the original $A$ and remove the matched part and split the
remaining parts in $A^{c}$. Otherwise, mark the $g$ as MISS (✗). In this
example, when $g="AB"$, it is a HIT:

$\displaystyle A$ $\displaystyle=[``A_{\text{\char 51}}B_{\text{\char
51}}",``B",``BB",``CBA"]$ $\displaystyle A^{c}$ $\displaystyle=[``BBBCBA"]$
$\displaystyle A^{g}$ $\displaystyle=[``AB"_{\text{\char 51}},``BC",``AB"].$

Then when $g=``BC"$, it is a HIT. Note that the item in $A^{c}$ is split into
two because of matching in the middle:

$\displaystyle A$ $\displaystyle=[``A_{\text{\char 51}}B_{\text{\char
51}}",``B",``BB_{\text{\char 51}}",``C_{\text{\char 51}}BA"]$ $\displaystyle
A^{c}$ $\displaystyle=[``BB",``BA"]$ $\displaystyle A^{g}$
$\displaystyle=[``AB"_{\text{\char 51}},``BC"_{\text{\char 51}},``AB"].$

Last, when $g=``AB"$ again, it is a MISS, $A$ and $A^{c}$ unchanged, hence,
omitted:

$\displaystyle A^{g}$ $\displaystyle=[``AB"_{\text{\char
51}},``BC"_{\text{\char 51}},``AB"_{\text{\char 55}}].$

  3. 3.

Finally, iterate through each $a\in A$. If any $a$ has at least one word token
marked as HIT, then the whole $a$ is a HIT, otherwise, a MISS.

$\displaystyle A$ $\displaystyle=[``AB"_{\text{\char 51}},``B"_{\text{\char
55}},``BB"_{\text{\char 51}},``CBA"_{\text{\char 51}}].$

This procedure converts $A$ and $A^{g}$ into lists of HITs and MISSes. Then
the evaluation metrics in (4) and (5) can be applied. Note that this procedure
is not order invariant. This in turn makes the F1 score not entirely order
independent if any UI element ambiguity happens. This choice is to avoid the
permutation complexity in evaluation. In practice, this is rarely an issue
because when the ambiguity happens, the UI elements involved are almost always
tightly close to each other, making their order practically fixed. See Case 3
in Figure 1(a) as an example.

##  3 Related Datasets and Annotations

ScreenQA has two aspects: multimodality and question answering. We discuss
related problems and datasets from these two aspects and focus our survey on
datasets that are 1) human annotated and 2) released to the public domain.

###  3.1 Multimodality

Mobile app screenshots contain nearly all possible representation of
information through pixels. Most commonly, the information is majorly by text,
blended with icons, symbols, and images.555Also videos, if we consider
consecutive screenshots. We leave out the video modality here in the context
of annotating the underlying RICO screenshots. We discuss three related
multimodal domains.

####  3.1.1 Screen UI for mobile apps

For data released in the public domain, the RICO dataset Deka et al. (2017)
is, to the best of our knowledge, still the largest collection of mobile app
screenshots Deka et al. (2021). It contains 66k unique screenshots and their
corresponding view hierarchies from 9.7k Android apps spanning 27 app
categories. Its overall approach extended ERICA Deka et al. (2016), which is
an interactive trace recording tool and also released 3k traces for 18k unique
screenshots from 1k Android apps for the search intent. LabelDroid Chen et al.
(2020b) and Chen et al. (2020a) by the same authors released a dataset of 55k
UI screenshots from 25 categories of 7.7k top-downloaded Android apps.

Annotations and the corresponding problems can be roughly categorized by the
scope of the contexts. At the UI element level, Sunkara et al. (2022)
annotated 77 icon types by shape, 15 out of which are additionally annotated
with 38 semantic types, reaching about total 500k unique annotations. This
work is further concerned with how UI elements are associated with companion
labels such that the screen understanding between UI elements can be
established. CLAY Li et al. (2022) attempted to resolve the layout and view
hierarchy denoising problem, annotating 60k RICO screenshots, a total of 1.4M
UI elements with bounding boxes and types. Li et al. (2020c) annotated 163k
free-from descriptions for 61k UI elements from 22k RICO screenshots. At the
single-screen level, Wang et al. (2021) collected text summarizations for
screens, consisting of 112k screen descriptions across 22k RICO screenshots.

At the multi-screen level, one challenging direction is screen navigation,
which requires the understanding of screen states, feasible action spaces of
the current screen, and overall task goals. Since multiple types of
understandings are involved, this problem is not strictly focused on screen
content understanding. PixelHelp Li et al. (2020a) contains 187 multi-step
instructions over 780 screenshots for four task categories. MoTIF Burns et al.
(2022) contains 6k fine-grained instructions mixed with infeasible ones, over
for 125 apps spanning 15 app categories. From the data perspective, annotating
this type of problem is labor intensive and usually does not scale well.

In comparison, the ScreenQA dataset is single-screen, focused on screen
contents, and based on the RICO screenshots.

####  3.1.2 Document image understanding

Document image understanding666Also referred to as document analysis and
recognition (DAR) or simply document understanding. concerns understanding
documents represented in pixels or scanned, photographed formats. This domain
is similar to mobile app screens for its text-heavy and non-sequential nature.
The most noticeable dataset is RVL-CDIP Harley et al. (2015), a 400k-image
subset from IIT-CDIP Lewis et al. (2006), a collection of low-resolution noisy
documents, with balanced 16 document-level classes. FUNSD Jaume et al. (2019)
extracted a 199 scanned form images from RVL-CDIP and annotated them with
bounding boxes and 4 text-segment-level classes. SROIE Huang et al. (2019) has
1k scanned receipt images for text localization, OCR, and key information
extraction of 4 entity types. CORD Park et al. (2019) contains 11k scanned
receipt images, annotated with 9 classes and 54 subclasses for text segments
in OCR boxes. These earlier works are more about classification for text
segments or for the whole document image.

A more recent work, DocVQA Mathew et al. (2021), uses a question answering
format for span/segment extraction, with an annotation of 50k questions over
12k rectified, higher resolution document images. DocVQA is highly related to
ScreenQA for its 2D arrangement of texts and for its extractive question
answering format. We believe that the techniques developed for screens and
document images are cross applicable.

####  3.1.3 Visual question answering

Visual question answering (VQA) Antol et al. (2015) and screen UI are
oftentimes mentioned together, especially in the latter community, because of
their vision-language multimodal nature. However, VQA is distinctively
different from screen understanding for two reasons: 1) The visual context for
VQA is usually light in, or even free from, any text, while screen UI is the
opposite, and 2) The images for VQA are typically photos of natural or daily
scenes with objects, while screen UIs are information oriented and arranged in
a certain visual structure. There are some VQA variants comparatively closer
to screen UI, to mention a few: VQA for texts on objects in photos, e.g.,
VizWiz Gurari et al. (2018) and TextVQA Singh et al. (2019), and VQA for
figures and charts, e.g., DVQA Kafle et al. (2018), FigureQA Kahou et al.
(2018), and LEAF-QA Chaudhry et al. (2020). These VQA tasks may appear as part
of screens but in general are different problems.

###  3.2 Question answering

Question answering tasks can be categorized by 1) open- or closed-domain, 2)
answer formats and 3) main capacities to evaluate.777Here we only include one
or two examples per format and per capacity. This is by no means to be
comprehensive. The common answer formats include span Rajpurkar et al. (2016),
entity Talmor and Berant (2018), multiple choice Mihaylov et al. (2018), and
generation Xiong et al. (2019). The capacities to evaluate range from reading
comprehension Yang et al. (2015), multi-hop reasoning Yang et al. (2018), Chen
et al. (2021), logic reasoning Yu et al. (2020), and commonsense reasoning
Talmor et al. (2019).

From this question answering perspective, ScreenQA is a closed-domain question
answering task that expects answers by span (or UI element phrase) selection
for screen reading comprehension. As described in Section 2, we instructed the
data annotators to avoid multi-hop, mathematical counting, and logic
reasoning, in order to focus on the fundamental screen comprehension capacity.

##  4 Annotation Method

Figure 2:  ScreenQA annotation process.

We perform several steps to collect the ScreenQA annotations, as depicted in
Figure 2. Each step is described below.

(a) In-sync VH with occluded UI elements.

(b) In-sync VH for main content, with ghosting VH from hamburger menu.

(c) Out-of-sync VH for main content, though top bar VH is in-sync.

Figure 3:  View hierarchies (VHs) are overlaid on the screenshots with class
names and the first few characters printed to assist annotators to determine
whether the VHs for the main contents are in sync. a) The VH of some UI
elements may be occluded. b) The VH of the hamburger menu is ghosting. Since
the VH for the main content stays intact, data annotators are instructed to
mark this as in-sync. c) Although the app bar and the orange plus button are
in sync, the bounding boxes in the main content area are all out of sync. This
is instructed to mark as out-of-sync and are excluded from further annotations
for questions and answers.

(a) Question annotation interface

(b) Answer annotation interface.

Figure 4:  Data annotation interfaces for question and answer collection. a)
Question annotation was performed in a sequential manner, the later and non-
overlapping annotators can see all previous questions to diversify question
framing and avoid duplication. We also used the sequential process to provide
more feedback and training to the annotators for quality improvement. b) The
answer annotators were tasked to determine if the question is valid and if the
question is answerable from the screen context. If both are positive, the
annotators need to answer the questions by 1) selecting or drawing the
bounding boxes of UI elements, 2) Fill the text for each selected/drawn
bounding box on right right, and 3) ranking them appropriately. The annotators
were also tasked to review and make necessary corrections if the question has
grammatical errors or typos.

###  4.1 Pre-filtering

The pre-filtering stage filters out 1) screenshots from non-English
apps888This is different from “non-English screenshots”, as translation and
dictionary apps could pose confusion., and 2) screenshots whose view
hierarchies (VHs) are out of sync with the main contents. It is a known issue
that in the RICO dataset, some screenshots and their corresponding view
hierarchies are not perfectly in sync: there exists certain time difference
between view hierarchy extraction and screenshot capturing. We want to remove
those screenshots to ensure that all ScreenQA annotations are not subject to
such data noises.

Classifying the sync quality is tricky, even for human readers. One may not be
able to differentiate between occlusion, ghosting, and the actual out-of-sync.
See Figure 3 for examples. Accordingly, we instructed the annotators to focus
on the main content area of the screen and make sure the bounding boxes in
that area are not corrupted, as this is where most contents of interest and
questions come from.

We use 27 annotators to perform this step. Among RICO’s 66k unique
screenshots, about 11k screenshots are from non-English apps, and about 13k
screenshots have out-of-sync view hierarchies.999This out-of-sync number is
different from Li et al. (2020b) because we focus on the main content area.
With the union of these two filtered out, there remains about 51k screenshots
from English apps with in-sync VHs.

###  4.2 Question annotations

For question annotation, we asked the annotators to frame questions given a
screenshot as the context. The annotators were expected to compose 1) natural,
daily-life questions as if using the app. 2) The composed questions should
inquire information that can directly read off from the screen and 3) should
not require logical reasoning, counting and calculation, mathematical
comparison, etc. We further required the annotators 4) not to ask questions
about any advertisement on the screen.

The annotation UI is depicted in Figure 4(a). We asked the annotators to
compose up to five questions given a screenshot in the first pass. In the
second pass, we asked for up to three questions given a screenshot and the
questions previously composed. Each pass involved one annotator for each
screenshot and whoever annotated the screenshot before is excluded from being
assigned to the same screenshot. This ensures that every screenshot is
assigned precisely two annotators to compose questions. We chose this
sequential process 1) to avoid tricky deduplication of similar questions, and
2) to encourage annotators to diversify their questions. Note that the same
set of annotators were involved in the both passes such that each annotator
had an opportunity to develop its own question style in the first pass before
seeing others’ in the second pass. This makes sure that we still have certain
numbers of question styles in the dataset before they converge to each other
in repeated passes.

We again involved the 27 annotators. The first pass of question annotation
generated 46k questions. The second pass added additional 36k questions. These
amount to a total of 82k questions, leaving about 15k screenshots with no
questions annotated, due to lack of interesting contents.

###  4.3 Answer annotations

We used the total 82k questions of 35k screenshots from the previous two-pass
question annotation step to further annotate the corresponding answers. The
annotator who composed the question is excluded from annotating its own answer
to avoid potential biases. The answer annotation UI is shown in Figure 4(b).

Given an example, which contains a screenshot and a question, the annotators
are tasked to

  1. 1.

Fix any grammatical errors or typos of the given question without altering its
intention.

  2. 2.

Answer the question, based on the context of the given screenshot, by 1)
selecting bounding boxes from the underlying view hierarchy leaf nodes that
contain the relevant answers, or drawing bounding boxes if no suitable leaf
nodes can be used, and 2) ranking the answers in descending order of relevance
if applicable, or by the common reading order.

  3. 3.

Consider two exceptions: 1) The question may be incomprehensible or 2) the
screenshot does not contain the answer to the question, due to the
questioner’s lack of understanding of the app. Then the example should be
marked as “invalid questions” and “not answerable from the screenshot”,
respectively.

As this step of annotation is more subjective and requires better
understanding of the presented apps, we used a selected pool of annotators to
perform this step of annotations. We then filtered out the questions marked as
“invalid question” to produce the overall ScreenQA annotations.

##  5 Annotation Analysis

We analyze the annotations of questions and answers in this section.

Category | % | Examples |
---|---|---|---
UI selection & config | 18.1 | Which option is selected? | What is the selected ringtone? |
Quantity number | 11.2 | How many unread messages? | How many pictures are there in Western Europe? |
App name | 11.0 | What is the name of the application? | What is the app name? |
Date time | 10.4 | When was “Heal the Living” released? | When is happy hour? |
Name of item | 3.4 | What is the name of the drug? | What is the name of chef? |
Price | 3.3 | How much is the gift bonus in 3rd place? | What is the price? |
User name | 2.8 | What is the name of the user? | What is the username on telegram? |
Address and direction | 2.5 | What is the current location? | What is the service zip code? |
Enum. of avail. options | 2.3 | Which social media options are given there? | What are the options available for logging in? |
Email address | 2.3 | What is an email address? | What is customer service email? |
Signup/login | 1.7 | Which application can be used to sign up / login? |  |
Person’s name | 1.7 | Who sang the song? | What is the last name? |
Weather | 1.6 | What is the range of temperature shown on Sunday? | What is the weather forecast for Sunday? |
Duration | 1.5 | What is the duration of video? | How long is the song? |
Version information | 1.5 | What is the version number? | What is the new feature in version v3.1.3? |
Score & value | 1.3 | What is height/weight of the person? | What is the score? |
Quantity amount | 1.1 | How much fat is there? | What is the amount? |
Yes/No | 1.0 | Is there any travel plans? | Is there any favorite? |
Phone number | 1.0 | What is the phone number? | What is the prefix for the international mobile number? |
Share/sharing | 0.9 | Which application can be used to share? | Where can I share this application? |
# of Stars | 0.8 | What is the star rating? | How many stars are given to the product? |
Percentage | 0.8 | What is the percentage of … ? | What is the brightness percentage for foreground? |
Age | 0.7 | How old is …? | What is the age? |
Settings | 0.7 | What is the setting of … ? | Which settings are switched on? |
Country | 0.5 | What is the name of the country? | Which country has the +54 code? |
Distance | 0.5 | What is the visibility distance? | How far is it from … ? |
# of Likes | 0.4 | How many likes for … ? | How many likes does … get? |
Permission | 0.4 | Which application is asking for permissions? |  |
# of Reviews | 0.4 | What is the number of comments on … ? | How many comments? |
Gender | 0.3 | What is the gender? | Which gender is displayed on the screen? |
Website | 0.3 | What is the url? | What’s the website address? |
Currency | 0.3 | What is the currency? | What is the currency for the price? |
How to | 0.3 | How to start on boot? | How to pronounce his name? |
Unit of measurement | 0.1 | What is the unit of temperature? | What is the unit of weight and length? |
Color | 0.1 | What is the UI color? | What is the amount of green color? |
Language | 0.1 | Which language is used in the setting? | Which language is being translated into which language? |
Others | 12.7 | What’s the average speed? | What is the user’s middle initial |
|  | What is the spending limit? | Which team has 41 points? |
Total | 100.0 |  |  |
Table 1: Question category distribution and examples.

(a)  Number of composed questions per screenshot.

(b)  Number of bounding boxes used to answer the question.

Figure 5:  Histograms for number of composed questions and number of bounding
boxes in answers. a) The two question annotation passes were capped at five
and three questions, respectively, resulting in the maximum eight questions in
total. b) The cases when a single bounding box forms a sufficient answer
amount to 92% of the questions, hence removed from the chart for the clarity
of the long tail. Anything beyond 11 bounding boxes is less than 0.05%,
accumulatively less than 0.1%.

###  5.1 Question analysis

We collected overall 82k questions over 36k unique screenshots from RICO.
Among the 82k questions, there are 46k unique questions.101010Note that it is
natural and valid to ask the same common questions over various screenshots,
for example, “Which option is selected on the screen?” and “What is the email
address?” Some screenshots receive more questions because they usually contain
more information to be asked about. Yet, the histogram still exhibits a
reasonable exponential decay with a mild slope, as depicted in Figure 5(a).

To further understand what questions have been asked, we categorize the
questions using regular expressions based on a list of empirically determined
question categories. The categories are meant to provide a rough overview of
the question annotations and by no means to provide a precise categorization.
The distribution and examples by these categories are tabulated in Table 1.
Note that the questions were not composed at the annotators’ full discretion:
They are conditioned on the given screenshots. That is to say, the
distribution is implicitly influenced by the RICO crawling process. For
example, as RICO crawled screen traces from freshly installed apps and did not
login an account, a noticeable number of the screen traces end at a login
page. This in turn translates to a higher percentage of questions asked about
app names, email addresses, permissions to login, etc.

###  5.2 Answer analysis

We analyze the answer annotations in two aspects: 1) How often do we need more
than one bounding box and its text to answer the question, and 2) How often do
human annotators find the view hierarchy useful to provide a minimal answer to
the question.

Figure 5(b) illustrates the histogram of number of bounding boxes used in each
answer. About 92% of questions can be answered fully using a single bounding
box. Among these single-bounding-box answers, 52% uses a VH leaf node
directly, while 48% uses a manually drawn bounding box. If we consider all
questions together, still 52% uses VH leaf nodes entirely, while 47% uses
manually drawn bounding boxes. That is, for about half of the total number of
screenshots, human annotators preferred to manually draw the bounding boxes in
order to provide answers that minimally satisfy the question. This observation
reflects the necessity not to require the view hierarchy input for ScreenQA as
described in Task 1.

Interestingly, there exist some cases, about 0.5% of the questions, that the
human annotators used a mixture of VH leaf nodes and manually drawn bounding
boxes as their full answer. By inspecting those cases, we found that these
usually happen 1) when the answer is an enumeration of “inhomogeneous” options
that are organized differently on the screen, such as using email vs. other
APIs to login, and 2) when an answer needs multiple parts to be complete, such
as a date consisting of year, month, and day scattered on the calendar UI, and
a temperature or a measurement requiring a number followed by the
corresponding unit. These parts may not be displayed in the same way,
resulting in lack of useful VH leaf nodes for some of the parts.

##  6 Conclusion

In this work, we proposed the ScreenQA task. We annotated a large-scale
ScreenQA dataset, which contains more than 80,000 question-answer pairs.
Compared to other vision-language multimodal problems, such as document image
understanding and visual question answering, ScreenQA poses its unique
challenges: rich in text, diverse in apps, and blended with icons and symbols.
We hope to use the ScreenQA task and the dataset to encourage the community to
look into this screen content understanding problem, as it enables new
technologies and new user experiences.

##  7 Acknowledgements

The authors would like to thank Srinivas Sunkara for his valuable discussions
and comments on this manuscript.

## References

  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual Question Answering.  In _2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 2425–2433, Santiago, Chile. IEEE.
  * Burns et al. (2022) Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022.  A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility.
  * Chaudhry et al. (2020) Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020.  LEAF-QA: Locate, Encode & Attend for Figure Question Answering.  In _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 3501–3510, Snowmass Village, CO, USA. IEEE.
  * Chen et al. (2020a) Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. 2020a.  Wireframe-based UI Design Search through Image Autoencoder.  _ACM Transactions on Software Engineering and Methodology_ , 29(3):19:1–19:31.
  * Chen et al. (2020b) Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020b.  Unblind your apps: Predicting natural-language labels for mobile GUI components by deep learning.  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , ICSE ’20, pages 322–334, New York, NY, USA. Association for Computing Machinery.
  * Chen et al. (2021) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021.  FinQA: A Dataset of Numerical Reasoning over Financial Data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Deka et al. (2021) Biplab Deka, Bardia Doosti, Forrest Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Ranjitha Kumar, Tao Dong, and Jeffrey Nichols. 2021.  An Early Rico Retrospective: Three Years of Uses for a Mobile App Dataset.  In Yang Li and Otmar Hilliges, editors, _Artificial Intelligence for Human Computer Interaction: A Modern Approach_ , pages 229–256. Springer International Publishing, Cham.
  * Deka et al. (2017) Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications.  In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_ , UIST ’17, pages 845–854, New York, NY, USA. Association for Computing Machinery.
  * Deka et al. (2016) Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016.  ERICA: Interaction Mining Mobile Apps.  In _Proceedings of the 29th Annual Symposium on User Interface Software and Technology_ , UIST ’16, pages 767–776, New York, NY, USA. Association for Computing Machinery.
  * Gurari et al. (2018) Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018.  VizWiz Grand Challenge: Answering Visual Questions from Blind People.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3608–3617, Salt Lake City, UT, USA. IEEE.
  * Harley et al. (2015) Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. 2015.  Evaluation of deep convolutional nets for document image classification and retrieval.  In _2015 13th International Conference on Document Analysis and Recognition (ICDAR)_ , pages 991–995. IEEE Computer Society.
  * Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shjian Lu, and C. V. Jawahar. 2019.  ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1516–1520.
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002.  Cumulated gain-based evaluation of IR techniques.  _ACM Transactions on Information Systems_ , 20(4):422–446.
  * Jaume et al. (2019) Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019.  FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents.  _arXiv:1905.13538 [cs, stat]_.
  * Kafle et al. (2018) Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018.  DVQA: Understanding Data Visualizations via Question Answering.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5648–5656, Salt Lake City, UT. IEEE.
  * Kahou et al. (2018) Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018.  FigureQA: An Annotated Figure Dataset For Visual Reasoning.  In _International Conference on Learning Representations (ICLR)_ , page 20.
  * Lewis et al. (2006) D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard. 2006.  Building a test collection for complex document information processing.  In _Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’06, pages 665–666, New York, NY, USA. Association for Computing Machinery.
  * Li et al. (2022) Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. 2022.  Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale.  In _CHI Conference on Human Factors in Computing Systems_ , pages 1–13, New Orleans LA USA. ACM.
  * Li et al. (2020a) Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020a.  Mapping Natural Language Instructions to Mobile UI Action Sequences.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8198–8210, Online. Association for Computational Linguistics.
  * Li et al. (2020b) Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020b.  Mapping Natural Language Instructions to Mobile UI Action Sequences.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8198–8210, Online. Association for Computational Linguistics.
  * Li et al. (2020c) Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020c.  Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5495–5510, Online. Association for Computational Linguistics.
  * Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021.  DocVQA: A Dataset for VQA on Document Images.  In _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 2199–2208, Waikoloa, HI, USA. IEEE.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.
  * Park et al. (2019) Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019.  CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.  In _Workshop on Document Intelligence at NeurIPS 2019_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Singh et al. (2019) Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019.  Towards VQA Models That Can Read.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8309–8318, Long Beach, CA, USA. IEEE.
  * Sunkara et al. (2022) Srinivas K. Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James W. Stout. 2022.  Towards Better Semantic Understanding of Mobile Interfaces.  In _Proceedings of the 30th International Conference on Computational Linguistics_ , Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018.  The Web as a Knowledge-Base for Answering Complex Questions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 641–651, New Orleans, Louisiana. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wang et al. (2021) Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021.  Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning.  In _The 34th Annual ACM Symposium on User Interface Software and Technology_ , UIST ’21, pages 498–510, New York, NY, USA. Association for Computing Machinery.
  * Xiong et al. (2019) Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019.  TWEETQA: A Social Media Focused Question Answering Dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5020–5031, Florence, Italy. Association for Computational Linguistics.
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  WikiQA: A Challenge Dataset for Open-Domain Question Answering.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2013–2018, Lisbon, Portugal. Association for Computational Linguistics.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020.  ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning.  In _International Conference on Learning Representations_.