Figure 1. The Crepe app provides a low-code solution for academic researchers
to collect data displayed on mobile screens. Through a programming by
demonstraion paradigm, a researcher taps on the target data to collect on the
screen (A). Crepe will automatically generate a novel Graph Query we designed
(B), to accurately identify and locate the UI element containing the target
data on the screen’s UI Snapshot (C). The created Graph Query will be shared
with data collection study participants to collect the target data on other
screens (D).

The teaser figure consists of four sections: A, B, C, and D. Section A shows a
screenshot from an Instagram app’s Story advertisement with various UI
elements like buttons, text, and images. The researcher has tapped on the
”Apple” text, and a confirmation dialog appears, asking if they want to
collect this data. Section B presents the automatically generated Graph Query
as a structured expression, capturing the relationships between the selected
“Apple” element and other screen entities like class names, package names, and
locations. Section C displays the original UI snapshot context from the
Instagram app. Section D demonstrates the execution of the generated graph
query across different app UI states, successfully identifying and locating
target data elements like “centierbank” and “duecelove”, illustrating the
precise and flexible data collection capabilities.

#  Crepe: A Mobile Screen Data Collector Using Graph Query

Yuwen Lu  University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Meng Chen
University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Qi Zhao  Indiana
University-Purdue University IndianapolisIndianapolisINUSA<EMAIL_ADDRESS>,
Victor Cox  University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Yang
Yang  University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Meng Jiang
University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Jay Brockman
University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Tamara Kay  University
of Notre DameNotre DameINUSA<EMAIL_ADDRESS>and  Toby Jia-Jun Li  University of
Notre DameNotre DameINUSA<EMAIL_ADDRESS>
(2023)

###### Abstract.

Collecting mobile datasets remains challenging for academic researchers due to
limited data access and technical barriers. Commercial organizations often
possess exclusive access to mobile data, leading to a “data monopoly” that
restricts the independence of academic research. Existing open-source mobile
data collection frameworks primarily focus on mobile sensing data rather than
screen content, which is crucial for various research studies. We present
Crepe, a no-code Android app that enables researchers to collect information
displayed on screen through simple demonstrations of target data. Crepe
utilizes a novel Graph Query technique which augments the structures of mobile
UI screens to support flexible identification, location, and collection of
specific data pieces. The tool emphasizes participants’ privacy and agency by
providing full transparency over collected data and allowing easy opt-out. We
designed and built Crepe for research purposes only and in scenarios where
researchers obtain explicit consent from participants. Code for Crepe will be
open-sourced to support future academic research data collection.

††journalyear: 2023††copyright: rightsretained††journal: PACMHCI††journalyear:
2024††journalvolume: 6††journalnumber: UIST††article: 1††publicationmonth:
11††doi: XXXXXXX.XXXXXXX

##  1\. Introduction

Datasets collected from apps running on mobile devices are valuable resources
in Human Computer Interaction (HCI) and social sciences (Raento et al., 2009),
but collecting mobile app datasets has remained challenging for academic
researchers. In practice, access to data siloed in mobile apps (Chandramouli
et al., 2015) is often exclusively possessed by the smartphone platforms (e.g.
iOS, Google Android) or individual products’ developers (Generosi et al.,
2020). Academic researchers have to either use public developer APIs (Trupthi
et al., 2017) or directly collaborate with products’ creators (Kramer et al.,
2014) to obtain mobile datasets. However, many researchers have limited
resources and technical expertise to build their own data collectors.
Meanwhile, the availability of developer APIs are subject to corporations’
discretionary policies (Reddit, 2023; Barnes, 2023). Also, when researchers
collaborate with commercial organizations to access mobile data (Kramer et
al., 2014; Deka et al., 2017), they often need to align with the
organizations’ business goals and public images, restricting the independence
of academic research (Kramer et al., 2014). These phenomena are recognized as
commercial organizations’ “data monopoly” in previous work (Jia-Jun Li et al.,
2022) and pose challenges for empirical research related to mobile devices.

Many open-source mobile data collection frameworks have been created to
support academics; however, most mature solutions focus on mobile sensing data
(Ferreira et al., 2015; Aharony et al., 2011; Cardone et al., 2013;
Tangmunarunkit et al., 2015; Schueller et al., 2014; Ferreira et al., 2015;
Andone et al., 2016; Cao et al., 2015) instead of screen information
collection. Screen information i.e., the various data displayed on mobile user
interface (UI) screens, is an important data source to enable various research
studies (Thomas et al., 2020; Robinson et al., 2017; Supanitayanon et al.,
2020; Hill and Monroy-Hernández, 2017). A major technical challenge is the
difficulty to reliably and automatically identify which user interface screens
contain the target data, and then locate and collect it. As a result, many
mobile screen collectors require users to actively input the data or upload
screenshots containing the target data (Calacci and Pentland, 2022). Some
collectors continuously record user screens to get the target information,
which is less efficient and poses privacy concerns (Krieter, 2019).

In this work, our goal is to build a low-code tool to support the collection
of screen information and user interaction data on mobile devices. To tackle
the above challenges in mobile data collection, we designed and implemented a
novel query language, Graph Query, which augments mobile UI screen structures
to support the flexible identification, location, and collection of specific
data pieces. On top of Graph Query, we designed and built an Android collector
app named Crepe111Crepe stands for “Collector for Research Experiments of
Participant Experiences”.. Crepe supports researchers to collect screen data
through simple, no-code demonstrations of the target data to collect, as well
as easy, scalable sharing of collectors with participants.

We designed Crepe with a special focus on data collection participants’
privacy. The participants can download Crepe and easily join a data collection
study by entering a collector ID shared by a researcher. Users will have full
transparency over what data is collected from their phones and can leave any
data collection at any time. We intend Crepe to be used for research purposes
only, and require researchers to obtain explicit consent from participants and
fully disclose the data being collected to participants.

In all, our contributions include:

  1. (1)

A novel query language to effectively identify, locate, and collect target
data on UI screens;

  2. (2)

The first Android data collector app, Crepe, to support flexible,
customizable, low-code UI screen data collection using a programming by
demonstration (PBD) paradigm (Cypher and Halbert, 1993);

  3. (3)

A user study with 7 participants in a realistic data collection task that
uncovered the real-world performance, limitations, and future development
directions of Crepe.

##  2\. Background and Related Work

###  2.1. Understanding Human Activities via Mobile Data

Mobile devices, especially smartphones, are now ubiquitous in everyday lives.
Mobile applications harness various information about user social activities,
which are embedded in users’ interactions, app usage patterns, and
preferences. As a result, researchers in HCI and social science have been
leveraging rich data gained from mobile apps to better improve user
experiences (Deka et al., 2017; Wang et al., 2023), uncover societal trends
(Vieweg et al., 2010; Starbird et al., 2014; Starbird and Palen, 2010), and
understanding user behaviors and attitudes (Fakhradyan, 2021; De et al., 2017;
Balaban et al., 2022; Kramer et al., 2014).

Screen data from mobile phones, i.e. information displayed on mobile screens,
are particularly valuable for understanding user behavior and preferences.
Screen data capture visual information and layout of the apps, providing
insights into how users navigate and engage with different features. By
analyzing this rich, granular data, researchers can gain a deeper
understanding of user needs, preferences, and pain points, which can inform
the design of more user-friendly and engaging mobile experiences (Ballard,
2007).

####  2.1.1. Utilizing Behavioral Data Stream for Analysis

Researchers from a diverse background, such as social science or business,
mine mobile data for behavioral patterns and insights. Experience Sampling
Method (ESM) is a research methodology to obtain in situ data for constructing
an understanding of subjects’ daily behaviors, feelings, and thoughts of
participants (van Berkel et al., 2017; Yue et al., 2014). In the survey
conducted by Berkel et al., smartphone-based ESM benefits researchers in
improving data quality through validation, collecting rich multimodal data for
context reconstruction, and enabling real-time data collection and analysis
(van Berkel et al., 2017). ESM can provide social scientists with combined
human and sensor data in addition to background logging (Raento et al., 2009).
Aside from sensor data, ESM questionnaires can also capture external aspects
of experience, such as time, place, activities, and companion (Hektner et al.,
2007). For instance, Yue et al. adopted ESM to study the function of photos.
They found that photos can be used to trigger memory during follow-up
interviews and as a beneficial component in data analysis (Yue et al., 2014).
In this work, our goal is to create a low-code mobile UI screen data
collection tool. Our app Crepe can facilitate academic researchers to
instrument screen data collections without the need to extensively develop
customized data collectors, or have public access to data APIs.

###  2.2. Computational Understanding of Mobile UI Structures and Data

Technical researchers have developed intelligent systems that achieve
computational understanding of mobile user interfaces. Sugilite, a mobile PBD
system that the execution of various tasks through user-driven multimodal
direct manipulations, parses UI elements and layout information to understand
the intentions contained in user activity (Li et al., 2017). The Sugilite
system leverages the accessibility API of Android to encapsulate high-quality
relevant behavioral data on mobile devices. UI understanding and summarization
is another topic that HCI researchers are interested in. An accurate and
succinct description of UI semantics can facilitate seamless interactions as
well as bridge the gap between language and user interface. Prior research
such as Screen2Vec (Li et al., 2021) and Screen2Words (Wang et al., 2021),
adopted Machine Learning (ML) techniques to support UI design. More recent
approaches have also leveraged the advance of Large Language Models (LLMs) and
large multimodal models to enhance conversational interactions on mobile UI
(Wang et al., 2023; Yan et al., 2023). In this work, with our proposed novel
Graph Query, we aim to create a deterministic and reliable query language that
can reliably and deterministically identify and locate a piece of target
information on screen. This compliments existing research on mobile UI
understanding in addition to existing deep-learning-based mechanisms.

###  2.3. State-of-the-art Mobile Data Collection Tools

A variety of data collection tools and frameworks have been proposed to
support academic data collection of various types of data. Aware is a mobile
instrumental tool designed for collecting usage context through sensors on
mobile devices (Ferreira et al., 2015). Rather than being a simple mobile data
collector, Aware creates a collaborative framework for researchers to share
context data with the community. Purple Robot is an extensible and modular
development platform that supports behavioral and clinical intervention
(Schueller et al., 2014). The tool enables Behavioral Intervention Technology
(BIT) stakeholders to create mobile apps that collect user management, content
authorship, and content delivery data, through which physician-scientists can
evaluate and share data resources to increase knowledge finding and
generalization. ShiptCalculator is a research tool designed to track and share
workers’ aggregated data about their pay to empower workers in offering
awareness of wage transparency and advocacy to campaigns. In the data
collection process, Workers send a screenshots of their pay history to
ShiptCalculator, which parses them using OCR and stores structured data. The
system then sends a validation text to workers and allows them to explore
their pay details (Calacci and Pentland, 2022). However, no current tool
supports the flexible collection of data displayed on mobile UIs based on our
review of previous work. Our design of Crepe in this study aims to solve this
exact problem.

##  3\. Proposed Method

Our goal for the Crepe data collector is three-fold. First, we seek to empower
researchers lacking programming expertise to easily instrument data
collections. Second, we prioritize the privacy and agency of individual data
collection participants. Lastly, we also want to address the following
observed technical challenges in reliable mobile screen data collection, which
hinder even researchers with technical skills:

  1. (1)

The difficulty in flexibly and automatically detecting the appearance of
target data on the screen;

  2. (2)

The lack of precision in reliably pinpointing the location of target data;

  3. (3)

The limited generalizability to diverse data types, especially for dynamically
evolving data.

To get around these challenges, previous collectors have asked participants to
either directly input target data or upload screenshots or screen recordings
containing target data (Calacci and Pentland, 2022; Krieter, 2019). However,
these practices significantly increase participants’ efforts and reduce the
reliability of the collected data’s quality.

For the rest of this section, we first give an overview of the experience
design of Crepe, our Android mobile data collection app. Then, we dive into
the technical details of the novel Graph Query we designed and implemented as
the backbone of Crepe.

###  3.1. The Crepe Data Collector User Experience

Figure 2. The workflow of Crepe for our main user groups: data collection
researchers and participants. Researchers create a new data collector by
demonstration (A) and share the collector ID with participants (B).
Participants add the collector to their own devices (C), which runs in the
background to collect the specified target data (D). The collected data is
transmitted to a database (E) for the researcher to analyze and gain insights
(F). The colors in the background indicate each user group’s experience
involved in the holistic Crepe pipeline.

The figure illustrates the workflow of the Crepe system with 6 labeled steps:
A) The researcher creates a new data collector by specifying the time period,
app package, data field and description through a graphical interface on their
device. B) The researcher shares the collector ID with participants. C)
Participants add the collector to their own devices by entering the shared
collector ID into the Instagram app running on their phone. The collector
details like start/end time are shown. D) On the participant’s device, the
Crepe app automatically identifies and locates the target UI data elements
(like ”walmart”, ”deucelove”) using Graph Query, based on the UI screenshot.
E) The target data elements collected by Crepe on participant devices get
transmitted to a database. F) The researcher analyzes the collected data from
the database to gain insights for their study. Arrows show the flow of
information between the researcher device, participant devices, and the data
collection database.

The Crepe data collector has two main target user groups: researchers who want
to instrument a mobile data collection study, and study participants who want
to contribute mobile data to ongoing data collections222We intend Crepe to be
used for research purposes only where participants give their explicit consent
to researchers before data collection begins..

Researchers have two main tasks when using Crepe (Figure 2): creating a
collector (A), and sharing collectors with study participants (B). A
researcher can use Crepe’s graphical user interfaces to define the start and
end dates of a collector, as well as the target app to collect data from.

The most important yet trickiest step in defining a data collector is
specifying the target data on screen to collect. In Crepe, our design goal is
to minimize the complexity of this specification process for researchers, by
only asking them to tap on the screen data to collect in the target app, and
describe their intention to collect these data (Figure 2 (A)). Crepe
automatically translate these two pieces of information into a formal,
executable Graph Query, which includes information about the target data’s
parent app package, component type, and relations with other UI entities on
screen. This Graph Query is the core of a data collector and will be used to
reliably identify and precisely locate target data on the study participants’
devices. This specification process follows a programming-by-demonstration
paradigm (Cypher and Halbert, 1993) and provides a simple, no-code experience
for researchers to specify the target data. (The technical detail will be
further explained in the following subsection on Graph Query.) After the
target data are specified and descriptions are added, a collector is created.
Afterward, the researcher can copy and share the unique ID of this collector
with participants (B). The collector will run on the participants’ device for
the specified data collection periods. Afterwards, the researcher can receive
the data collected in a self-defined database (E) and analyze it to uncover
research insights (F). Our initial implementation of Crepe transmits collected
data to a pre-configured Google Firebase Realtime Database that encrypts data
both in transit and at rest333https://firebase.google.com/support/privacy. We
plan to support researchers’ customized configuration of their own database
servers in the future.

For study participants, they can easily add a collector to Crepe by entering
the unique collector ID shared by the study’s researchers (C), using Crepe’s
graphical user interfaces. After granting Crepe with the necessary Android
Accessibility Service permission, the added collector(s) will start
automatically running in the background of the participants’ devices (D).
Guided by the graph queries associated with the participants’ added
collectors, Crepe will selectively seek the appearance of target data
exclusively when the designated target apps are active. Once the target data
is detected, Crepe will scrape and transmit the relevant information to the
researchers (E). The participant will be able to monitor the active status of
the collector(s) they have joined in Crepe. To drop out of a study,
participants can simply delete a collector from their Crepe app, and the
collector will no longer run in the background.

##### Privacy considerations

Protecting the privacy and agency of study participants is critical in our
design of Crepe. We ensure individual participants have the full control over
their participation status and data contributions using two main measures.
First, we adopt an “opt-in” participation mechanism, where participants
actively add collectors to their Crepe app to join data collections. We intend
this to be combined with the explicit consents that researchers must obtain
before each data collection study starts. Second, each participant can opt out
of any collection, at any time, by deleting the collector from their Crepe
app, which will immediately terminate the collection on their mobile devices.

For full transparency, the participant can in real-time check the examples of
data being collected from their phone in the app’s collector details popup.
Whenever a collector is modified by the researcher (e.g. extended or shortened
collection period, additional or reduced types of data being collected), all
participants will be informed of the changes through push notifications. We
are also continuously rolling out new privacy-centric features, such as a data
visualization dashboard for participants to see their contributed data volume
and content, and the ability to request all their data be deleted when they
withdraw from a study.

###  3.2. Graph Query

####  3.2.1. Building blocks of graph queries: screen entities and their
relations

Figure 3. Details regarding the process of query generation and execution in
Crepe. Section A (1) shows a UI Snapshot with UI elements and their relations,
from which potential graph queries are generated (2) by combining screen
entity relations to uniquely identify the target UI element on this UI
Snapshot. Section B (3) depicts the execution of a chosen graph query on a
different UI Snapshot, where the target element “duecelove” is successfully
located, while the query fails to find a match on another screen.

The figure consists of two main sections, A and B, presented side by side.
Section A, labeled “Query Generation”, displays a UI snapshot from a mobile
app with various GUI elements like buttons, images, and text. It shows the
process of creating graph queries by combining the relations between these
screen entities. Section B, titled “Query Execution”, demonstrates the
execution of a selected graph query on different UI snapshots. The query
successfully identifies and locates the target element “duecelove” on one
snapshot but fails to find a match on another screen. The figure uses visual
cues like arrows and graphs to illustrate the flow of query generation and
execution.

We develop a novel Graph Query in Crepe for accurate and consistent extraction
of various types of screen information (Figure 3). The intuition behind Graph
Query is to uniquely identify and locate the element(s) that contain the
target data using the UI elements’ own attributes and relations with other
elements on screen. Graph Query builds on top of the XML mobile screen
hierarchy, similar to the DOM tree of web HTML, that is accessible through
Android Accessibility Service. This mobile screen hierarchy contains Views on
the current screen (1), each of which contains rich properties including
content, screen location, etc.

First, we enhance the default mobile screen hierarchy provided by Android
Accessibility Service into an augmented UI Snapshot, which contains additional
relations between screen entities (Figure 3, UI Snapshot section). We defined
a screen entity to be either an UI element (i.e., Android View) or an
attribute of the UI element, in data representations such as strings or
integers. The relations between these screen entities include:

  1. (1)

UI elements to their own Android implementation attributes:

     * •

e.g. hasClassName, hasScreenLocation, isEditable

  2. (2)

UI elements to their semantic content:

     * •

e.g. hasText, containsEmailAddress, containsMoney, containsDate

  3. (3)

hierarchical relations between UI elements in the XML screen hierarchy:

     * •

e.g. hasParent, hasChild, hasSibling

  4. (4)

spatial relations between UI elements on screenshot images:

     * •

e.g. right, left, above, below, near

These relations comprehensively encapsulate the various relations and semantic
contents tha are useful for identifying and locating a date on screen. With
such additional information, an UI Snapshot is a collection of subject-
predicate-object triples denoted as (s, o, p), where s and o are two entities
and the p is the relation between s and o.

####  3.2.2. Automatic creation of graph queries

When a researcher specifies a piece of data to collect through the
demonstration of tapping on the target data on screen (e.g. tapping on “apple”
on the screenshot in Figure 3 section 1), we locate the associated UI element
in the UI Snapshot, and generate a list of potential graph queries based on
the target data’s relations with other screen entities in the UI Snapshot (2).

The Graph Query can be connected via 3 logical operators: conj (and), or (or),
and prev (previous). We defined a set of context-free grammars, summarized in
Table 1, to construct formal graph queries based on screen entity relations in
UI Snapshots.

Expression | Rule
---|---
E |  $\rightarrow$ e;
E |  $\rightarrow$ S;
S |  $\rightarrow$ (join r E);
S |  $\rightarrow$ (and S S);
T |  $\rightarrow$ (ARG_MAX r S);
T |  $\rightarrow$ (ARG_MIN r S);
Q |  $\rightarrow$ S — T;
Table 1. Context-free grammars (CFGs) for constructing Graph Query in Crepe. Q
denotes the initial non-terminal symbol, terminal e denotes a GUI object
entity, and terminal r denotes a relation. Other non-terminal symbols are
employed for intermediary steps in the derivation process. We used this set of
CFGs to automatically construct formal Graph Queries from the screen entity
relations in UI Snapshots.

Specifically, Graph Queries encapsulate combinations of the screen entity
relations for the target UI element and Crepe creates combinations can
uniquely identify only the target UI element on screen (Figure 3 (2)). The
combinations of screen entities are empirically set through experimentation.
All graph queries will have hasPackageName and hasClassName to establish the
context. In most cases, more than one graph query can uniquely locate the
target UI element on the screen. In these scenarios, to ensure the final
selected Graph Query reflects the user’s intention of collecting the target
data, we used a large language model444We used GPT-3.5-turbo in our
implementation due to its ease of access, short inference time, and decent
performance observed on our task., to select the Graph Query that best matches
users’ description of target data. The description of the target data to
collect was an input field in Crepe’s graphical user interface (the details of
our prompt can be found in Appendix A.1). Altogether, Crepe can effectively
generate graph queries that can reliably, uniquely identify and locate the
target UI element on screen.

####  3.2.3. Graph Query execution on UI Snapshots: retrieving the target data

After a graph query is created, it can be used to collect target data on any
UI Snapshot with elements sharing the same set of entity relations. In the
background, Crepe runs the Graph Query every time a relevant UI snapshot is
updated by changes in screen content (Figure 3 Section (B)). We also limit the
Graph Query to only run in the target app package, in order to effectively
reduce unnecessary battery usage of Crepe. After idenfitying and locating the
target UI element on screen (3), Crepe can collect the target data to collect
through the element’s attributes such as text and content description.

####  3.2.4. Summary

Graph Query showcases three major strengths in screen data collection. First,
it provides much more flexibility in identifying and locating target UI
elements through their implementation attributes, semantic content,
hierarchical screen structure position, and screen location relations (3.2.1).
For instance, a researcher can utilize the pre-defined hasPrice relation to
collect Uber drivers’ task payment information, a common task on gig worker
support platforms (Calacci and Pentland, 2022). In the future, developers can
also extend existing entity relation types for more customized data collection
tasks. Second, the data collection mechanism using Graph Query is fully
deterministic and more reliable than alternative solutions like using Optical
Character Recognition (OCR) or deep-learning based detection models. Lastly,
by utilizing Android’s Accessibility Service, using graph query can easily
integrate the collection of user interaction data, a feature included in Crepe
yet unsupported in most vision-based data collection tools. In the future,
Crepe can be extended to collect more rich media information such as
screenshots under the study participants’ consent555 Link to the documentation
of Android Accessibility Service’s API for taking screenshots..

###  3.3. Implementation

Crepe was implemented using Android, Google Firebase Realtime Database, OAuth,
and OpenAI API. The app was developed in Java using Android Studio and is
compatible with mobile devices running Android 9.0 or above, which in March
2024 takes up around 90% of the Android mobile phone market. Firebase Realtime
Database was selected due to its high-standard data encryption both in transit
and at rest, real-time data synchronization, and reliable user authentication.
To implement the query language, Crepe leverages the Accessibility Service and
captures Accessibility Events as a result of changes in the content of the
screen. The captured Accessibility Event contains the new UI screen
hierarchy666https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent.
The code for Crepe will be open-sourced to enable easy adoption and further
development within the research community.

##  4\. Example Use Case and User Testing

Participant ID |  Mobile OS |  Device |  Period |  Battery Usage
---|---|---|---|---
P1 |  Android 14 |  Pixel 8 (own) |  72 hours |  1%
P2 |  Android 14 |  Pixel 6 (provided) |  72 hours |  14%
P3 |  Android 14 |  Pixel 6 (provided) |  72 hours |  9%
P4 |  Android 13, OxygenOS 13.1 |  OnePlus 9R (own) |  24 hours |  0.91%
P5 |  Android 12, Oxygen OS 12.1 |  OnePlus Nord (own) |  24 hours |  N/A
P6 |  Android 14 |  Pixel 6 (provided) |  24 hours |  2%
P7 |  Android 9 |  Galaxy S10 (own) |  24 hours |  0%
Table 2. Mobile device usage data for study participants over 24 to 72 hour
periods, including mobile OS, device model, and battery usage percentage.
Three of the participants did not have an Android phone and used our provided
device, with which they were asked to use Instagram following their regular
habits during the study period. The participants decided the time period of
the study based on their time schedule. Note that P5 reported that they could
not found the battery usage of Crepe after the collection. We were not able to
help them find it either, despite our best efforts.

In this section, we describe a potential use case of collecting Instagram
Story advertisements to demonstrate the collection mechanism and utility of
Crepe. We also deployed a user study to understand Crepe’s collection
performances and limitations in the wild.

###  4.1. Example Use Case: Collecting Instagram Story Advertisements

Instagram Story advertisements are sponsored short image/video segments that
appear between an Instagram user’s following content. They usually come with a
“Sponsored” label to be distinguished from users’ regular following content.
Mutiple research studies have been conducted in recent years around Instagram
Story advertisements, to understand how users perceive and understand this
specific form of advertising (Balaban et al., 2022; Haldborg Jørgensen et al.,
2023; Fakhradyan, 2021). However, our the best of our knowledge, no academic
research to date has been able to collect a dataset of Instagram story
advertisements. Such data is also not available from Instagram’s developer API
777https://developers.facebook.com/docs/instagram-api/.

We believe that getting access to a dataset of Instagram’s Story
advertisements for individual users can enable research contributions in
fields including social media research (Hanbazazh and Reeve, 2021), algorithm
auditing (Ulbricht and Yeung, 2022; Eslami et al., 2015, 2016), and usable
privacy (Chen et al., 2024). Specifically, we can empirically observe users’
interactions with Instagram Story ads, as well as inferring dwelling time over
Story ads based on the first and last content change while a specific ad
appears on screen. More generally, some broader research questions we can
answer with the dataset include:

  1. (1)

What empirical understanding of Instagram’s personalized advertisement
algorithms can we obtain from quantitative data analysis?

  2. (2)

How Instagram Story advertisements reveal the platform’s portraits of
individual users’ preferences?

  3. (3)

What are users’ folk theories of Instagram’s personalized advertisement
algorithm?

###  4.2. User Testing

We conducted a user study to evaluate the performance and limitations of Crepe
in real-world scenarios. It is important to note that the primary objective of
this study was not to collect high-quality, immediately usable datasets to
support future research, but rather to assess the real-world performance of
our data collector and its impact on participants’ regular mobile device usage
experience. To this end, the study aimed to address the following research
questions:

  1. (1)

RQ1: How does Crepe perform in a real-world data collection scenario?

  2. (2)

RQ2: What limitations does Crepe have when collecting data in the wild?

  3. (3)

RQ3: How does Crepe impact the regular everyday mobile phone device usage
experience of participants?

##### Participants

We recruited seven participants through social media advertisements and word-
of-mouth. The participants are regular users of Instagram and Instagram Story.
Each participant decided to contribute their data from 24 hours up to 72 hours
based on their preferences and time schedule. We aimed at testing Crepe on
various Android versions and device platforms, while also evaluating Crepe’s
performance over short and longer collection duration. Four participants used
their own Android devices. Since our main goal is not to collect a dataset
that can be directly used in rigorous quantitative analysis but to empirically
test our collector system, we also recruited three participants who are
interested but do not have an Android device (P2, P3, P6). We provided each of
them with an Android phone, with which they were asked to use Instagram during
the period of the study, roughly following their regular Instagram usage
habits. These three participants also contributed to our study by helping us
assess the performance and usability of Crepe across different usage
behaviors.

##### Study procedure

The user study followed a three-stage procedure888The study protocol was
reviewed and approved by the Institutional Review Board (IRB) at our
institution.:

  1. (1)

A 30-minute introductory interview to familiarize participants with the study
and set up the Crepe app on their devices;

  2. (2)

A testing period lasting between 24 to 72 hours, based on the participants’
time schedule and preferences, during which they were encouraged to use
Instagram as usual.

  3. (3)

A 30-minute concluding interview to administer a post-study questionnaire,
collect feedback on participants’ experiences, and address any questions or
concerns.

Throughout the study, we maintained email communications with participants to
ensure Crepe functioned normally and conducted troubleshooting sessions when
necessary. Each participant was compensated with 25 USD digital gift card for
the data they contributed, and 20 USD for each hour of meeting sessions they
attended. The participants’ participation duration, device information, and
Crepe’s battery usage result after the study ended are shown in Table 2.

Figure 4. Number of unique Instagram Story Ads collected by participants using
the Crepe data collector. The top subplot displays data from Participants 1-3
over a 72-hour period, while the bottom subplot shows data from Participants
4-7 over a 24-hour period. The x-axis represents the time intervals in hours,
and the y-axis indicates the number of Instagram Story Ads collected within
each time interval.

The figure consists of two vertically stacked subplots, both with multi-
colored bar graphs. The top subplot, titled ”Participants 1-3 (72 hours),”
displays data for three participants over a 72-hour period. Each participant’s
data is represented by a different color: Participant 1 in blue, Participant 2
in purple, and Participant 3 in green. The bars indicate the number of unique
Instagram Story Ads collected by each participant at various time intervals,
with the time in hours along the x-axis and the number of ads on the y-axis.
The bottom subplot, titled ”Participants 4-7 (24 hours),” presents data for
four participants over a 24-hour period. The participants’ data are
represented by different colors: Participant 4 in yellow, Participant 5 in
orange, Participant 6 in dark blue, and Participant 7 in light green. Similar
to the top subplot, the bars show the number of unique Instagram Story Ads
collected by each participant at different time intervals, with the time in
hours along the x-axis and the number of ads on the y-axis. Both subplots have
clearly labeled x and y axes, and a legend is provided to identify the
participants by their assigned colors. The figure allows for a visual
comparison of the Instagram Story Ad data collected by different participants
over their respective data collection periods.

###  4.3. Study Results

After initial data analysis, Crepe was able to collect 358 Instagram Story
advertisements from all seven participants, with an average of 51 ads per
participant (Figure 4). The collected data reveals the number of unique
Instagram Story Ads encountered by each participant at regular time intervals
throughout their respective data collection periods. The data allow us to
observe the frequency and distribution of Instagram Story Ads over time,
potentially providing insights into the advertising strategies employed by
businesses on the platform. We shared the data we collected from each
participant with them individually upon request.

In the post-study questionnaire, which included two 5-scale questions, all
participants unanimously reported that they felt minimal interruption and
disturbance from Crepe while using their everyday apps and did not notice
significant changes to their devices during the collection study. Here are
details regarding these two questions and all participants’ response:

  * •

Q1: When using my everyday apps, I felt minimal interruption and disturbance
from the Crepe app.

    * –

All participants responded 5 – Strongly agree

  * •

Q2: I noticed significant changes to my device during the data collection
study.

    * –

All participants responded 1 – Strongly disagree

Based on the battery usage data collected in the post-study questionnaire,
Crepe demonstrated excellent energy efficiency on participants’ mobile devices
over the 24 to 72 hour study periods. As shown in Table 2, the battery usage
of Crepe ranged from 0%, 0.91% to 14%, with most participants reporting usage
of 2% or less. This indicates that running Crepe in the background for data
collection has minimal impact on the battery life of participants’
smartphones, making it a power-efficient tool for mobile data collection
studies.

However, we observed a limitation of Crepe during the study: the background
process management on Android can terminate Crepe to conserve resources. This
is a known issue for similar mobile data collectors (Ferreira et al., 2015).
For P1, we suspect the Android operating system terminated Crepe in the
background, resulting in data loss at around hour 30. Only after we noticed it
and prompted P1 to check the running status of Crepe did the data resumed to
be collected. To address this issue, for later participants, we improved
Crepe’s “heartbeat” mechanism and had the app send a heartbeat signal every 10
minutes. Once we noticed a potential inactive status of Crepe, we sent
reminder emails to participants and asked them to open Crepe and check its
running status. In future iterations of Crepe, this heartbeat mechanism could
be combined with a dashboard to alert researchers when data collection becomes
offline, enabling timely interventions to minimize data loss.

##  5\. Discussion

###  5.1. Collection Support Tools for De-centralized Data Access

Crepe complements existing open-source data collection tools like AWARE
(Ferreira et al., 2015) and Purple (Schueller et al., 2014) that focus on
collecting sensor data, and others like ShiptCalculator (Calacci and Pentland,
2022) that enable workers to share screenshots of their pay history for wage
transparency advocacy. Crepe targets the generalized collection of various
types of mobile screen data. This contains great potential for more academic
researchers to more easily access datasets and uncover more quantitative,
analytical insights. Crepe aligns with the growing movement towards a more
open and equitable ecosystem in digital economy (Arrieta-Ibarra et al., 2018).

###  5.2. Careful Considerations for User Privacy

The collection of user data through Crepe raises important privacy concerns
regarding users’ app usage and screen information. Throughout the design and
implementation of Crepe, user privacy has been a top priority, ensuring real-
time transparency of collected data and employing high-standard encryption for
data security. It is crucial to emphasize that Crepe is intended solely for
research purposes, under scenarios where researchers obtain explicit consent
from participants to collect screen data. In the future, we plan to
continuously enhance the privacy-related features in Crepe. For instance,
prior to the public release of Crepe, we intend to include a feature that
hashes personal identifiers before sending them to researchers, thereby
protecting users’ privacy, similar to the approach taken by previous data
collectors (Ferreira et al., 2015). There have also been differential-privacy-
based approaches that obfuscate sensitive data on collected mobile app UI
snapshots (Li et al., 2020), which we can adopt in Crepe. We acknowledge that
protecting users’ privacy in data collection remains an open research
question, and we encourage the research community to collectively work towards
improving our ethical standards in this regard.

###  5.3. Implications for Data Labor and Data Governance

Our approach for the decentralization of data access has important
implications for data governance and the broader ecosystem of data-driven
technologies. The rise of digital platforms has led to the concentration of
user data in the hands of corporations and platforms, creating a “data
monopoly” that limits the potential benefits of academic research and user-
centric data-driven technollgies (Jia-Jun Li et al., 2022). By breaking the
data monopoly and providing users with access to their own data, our approach
can help bridge the data divide and empowers individuals and communities to
leverage their own data for personal and collective benefits, rather than
being solely exploited by corporations.

This increased transparency can help to identify and address biases and
inequalities in the algorithms (Panch et al., 2021; Kordzadeh and
Ghasemaghaei, 2022). By providing access to users’ own data on platforms,
distributed, user-centric data-driven technologies can be developed in user
groups and communities to shift the focus of benefits from corporations and
platforms to end users. This aligns with the vision of a more equitable data
ecosystem, where the value generated by user data is distributed more fairly
among stakeholders (Posner and Weyl, 2018).

##  6\. Limitations and Future Work

While Crepe offers a novel approach to mobile screen data collection, we
acknowledge it still contains a few limitations. One issue is that Crepe can
be terminated by the system in the background after a period of inactivity,
potentially leading to data loss. This is a known issue for existing mobile
data collectors (Ferreira et al., 2015). Additionally, system or app updates
can potentially change the UI hierarchy of app screens, invalidating pre-
defined graph queries and requiring researchers to update them during the
study to ensure compatibility. Running an Android application continuously in
the background without exhibiting malware-like behavior presents a challenge.
Yet, we plan to refine Crepe’s notification system and collector active status
to account for scenarios where Crepe gets killed in the background processes
by the Android system. In addition, a few directions can also be explored to
expand the capabilities of Crepe. Firstly, developing visualization dashboards
for both researchers and participants would enable real-time monitoring of the
collection status and the data contributed. Secondly, empowering users with
the ability to delete all collected data after dropping out of a study would
enhance their control over their data contributions and agency.

Currently, Crepe’s focus is on screen textual data, with great potentials in
incorporating more rich multimodal data, such as screenshots and user
interactions. However, we also acknowledge that there are certain limits to
the types of data our approach cannot collect. For example, information
rendered in game or 3D engines are hard to collect, since they do not show up
in Android Accessibility Services’ responses. In addition, the expressiveness
of our designed Graph Query also has boundaries. In certain screen structures,
Graph Query may default to collecting data based on their absolute screen
location, limiting the flexibility.

We plan to open source Crepe to promote the ethical use of our novel approach
in future academic research data collection. We also welcome contributions
from the research community to Crepe to make it more ethical, robust, and
adaptive to various data collection scenarios.

##  7\. Conclusion

In this paper, we introduce Crepe, a novel mobile data collection tool that
enables researchers to easily define and collect user interface screen data
from participants’ Android devices. Crepe utilizes Graph Query, which allows
flexible identification, location, and extraction of target UI elements. The
tool’s key feature is its low-code, programming by demonstration approach that
lets researchers specify data to collect through simple interactions with
example app screens. A user study with 7 participants demonstrated Crepe’s
ability to effectively collect screen UI data across different devices, while
identifying areas for future improvements. Overall, Crepe represents an
important step towards democratizing mobile app data for research by reducing
technical barriers and giving participants more agency. Code for Crepe will be
open-sourced to support the academic community in future research data
collection.

## References

  * (1)
  * Aharony et al. (2011) Nadav Aharony, Wei Pan, Cory Ip, Inas Khayal, and Alex Pentland. 2011.  The social fMRI: measuring, understanding, and designing social mechanisms in the real world. In _Proceedings of the 13th international conference on Ubiquitous computing_. ACM, Beijing China, 445–454.  https://doi.org/10.1145/2030112.2030171
  * Andone et al. (2016) Ionut Andone, Konrad Błaszkiewicz, Mark Eibes, Boris Trendafilov, Christian Montag, and Alexander Markowetz. 2016.  Menthal: a framework for mobile data collection and analysis.  In _Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct_. Association for Computing Machinery, New York, NY, USA, 624–629.  https://doi.org/10.1145/2968219.2971591
  * Arrieta-Ibarra et al. (2018) Imanol Arrieta-Ibarra, Leonard Goff, Diego Jiménez-Hernández, Jaron Lanier, and E. Glen Weyl. 2018.  Should We Treat Data as Labor? Moving beyond “Free”.  _AEA Papers and Proceedings_ 108 (2018), 38–42.  https://www.jstor.org/stable/26452701 Publisher: American Economic Association.
  * Balaban et al. (2022) Delia Cristina Balaban, Meda Mucundorfeanu, and Larisa Ioana Mureșan. 2022.  Adolescents’ Understanding of the Model of Sponsored Content of Social Media Influencer Instagram Stories.  _Media and Communication_ 10, 1 (March 2022), 305–316.  https://doi.org/10.17645/mac.v10i1.4652
  * Ballard (2007) Barbara Ballard. 2007.  _Designing the mobile user experience_.  John Wiley & Sons.
  * Barnes (2023) Jenae Barnes. 2023.  Twitter Ends Its Free API: Here’s Who Will Be Affected.  https://www.forbes.com/sites/jenaebarnes/2023/02/03/twitter-ends-its-free-api-heres-who-will-be-affected/ Section: Business.
  * Calacci and Pentland (2022) Dan Calacci and Alex Pentland. 2022.  Bargaining with the Black-Box: Designing and Deploying Worker-Centric Tools to Audit Algorithmic Management.  _Proceedings of the ACM on Human-Computer Interaction_ 6, CSCW2 (Nov. 2022), 1–24.  https://doi.org/10.1145/3570601
  * Cao et al. (2015) Paul Y. Cao, Gang Li, Guoxing Chen, and Biao Chen. 2015.  Mobile Data Collection Frameworks: A Survey. In _Proceedings of the 2015 Workshop on Mobile Big Data_ _(Mobidata ’15)_. Association for Computing Machinery, New York, NY, USA, 25–30.  https://doi.org/10.1145/2757384.2757396
  * Cardone et al. (2013) Giuseppe Cardone, Andrea Cirri, Antonio Corradi, Luca Foschini, and Dario Maio. 2013.  MSF: An Efficient Mobile Phone Sensing Framework.  _International Journal of Distributed Sensor Networks_ 9, 3 (March 2013), 538937.  https://doi.org/10.1155/2013/538937 Publisher: SAGE Publications.
  * Chandramouli et al. (2015) Vageesh Chandramouli, Abhijnan Chakraborty, Vishnu Navda, Saikat Guha, Venkata Padmanabhan, and Ramachandran Ramjee. 2015.  Insider: Towards breaking down mobile app silos. In _TRIOS Workshop held in conjunction with the SIGOPS SOSP_. Citeseer.
  * Chen et al. (2024) Chaoran Chen, Weijun Li, Wenxin Song, Yanfang Ye, Yaxing Yao, and Toby Jia-jun Li. 2024.  An Empathy-Based Sandbox Approach to Bridge the Privacy Gap among Attitudes, Goals, Knowledge, and Behaviors.  https://doi.org/10.1145/3613904.3642363 arXiv:2309.14510 [cs].
  * Cypher and Halbert (1993) Allen Cypher and Daniel Conrad Halbert. 1993.  _Watch what I Do: Programming by Demonstration_.  MIT Press.
  * De et al. (2017) Shaunak De, Abhishek Maity, Vritti Goel, Sanjay Shitole, and Avik Bhattacharya. 2017.  _Predicting the Popularity of Instagram Posts for a Lifestyle Magazine Using Deep Learning_.  https://doi.org/10.1109/CSCITA.2017.8066548
  * Deka et al. (2017) Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_. ACM, Québec City QC Canada, 845–854.  https://doi.org/10.1145/3126594.3126651
  * Eslami et al. (2016) Motahhare Eslami, Karrie Karahalios, Christian Sandvig, Kristen Vaccaro, Aimee Rickman, Kevin Hamilton, and Alex Kirlik. 2016.  First I ”like” it, then I hide it: Folk Theories of Social Feeds. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 2371–2382.  https://doi.org/10.1145/2858036.2858494
  * Eslami et al. (2015) Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015.  ”I always assumed that I wasn’t really that close to [her]”: Reasoning about Invisible Algorithms in News Feeds. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 153–162.  https://doi.org/10.1145/2702123.2702556
  * Fakhradyan (2021) Dovan Fakhradyan. 2021.  The Study of Consumer Preferences and Advertising Effectiveness Analysis Towards Studio Hikari Instagram Story Video Ads.  _Jurnal Ilmu Sosial Politik dan Humaniora_ 4, 1 (March 2021), 10–18.  https://doi.org/10.36624/jisora.v4i1.50 Number: 1.
  * Ferreira et al. (2015) Denzil Ferreira, Vassilis Kostakos, and Anind K. Dey. 2015.  AWARE: Mobile Context Instrumentation Framework.  _Frontiers in ICT_ 2 (2015).  https://www.frontiersin.org/articles/10.3389/fict.2015.00006
  * Generosi et al. (2020) Andrea Generosi, Silvia Ceccacci, Samuele Faggiano, Luca Giraldi, and Maura Mengoni. 2020.  A Toolkit for the Automatic Analysis of Human Behavior in HCI Applications in the Wild.  _Advances in Science, Technology and Engineering Systems Journal_ 5, 6 (2020), 185–192.  https://doi.org/10.25046/aj050622
  * Haldborg Jørgensen et al. (2023) René Haldborg Jørgensen, Hilde A.M Voorveld, and Guda van Noort. 2023.  Instagram Stories: How Ephemerality Affects Consumers’ Responses Toward Instagram Content and Advertising.  _Journal of Interactive Advertising_ 23, 3 (July 2023), 187–202.  https://doi.org/10.1080/15252019.2023.2232797 Publisher: Routledge _eprint: https://doi.org/10.1080/15252019.2023.2232797.
  * Hanbazazh and Reeve (2021) Alaa Hanbazazh and Carlton Reeve. 2021.  Pop-up Ads and Behaviour Patterns: A Quantitative Analysis Involving Perception of Saudi Users.  _International Journal of Marketing Studies_ 13, 4 (Nov. 2021), 31.  https://doi.org/10.5539/ijms.v13n4p31
  * Hektner et al. (2007) Joel M. Hektner, Jennifer Anne Schmidt, and Mihaly Csikszentmihalyi. 2007.  _Experience Sampling Method: Measuring the Quality of Everyday Life_.  SAGE.  Google-Books-ID: 05e5d_KBYY0C.
  * Hill and Monroy-Hernández (2017) Benjamin Mako Hill and Andrés Monroy-Hernández. 2017.  A longitudinal dataset of five years of public activity in the Scratch online community.  _Scientific Data_ 4, 1 (Jan. 2017), 170002.  https://doi.org/10.1038/sdata.2017.2 Publisher: Nature Publishing Group.
  * Jia-Jun Li et al. (2022) Toby Jia-Jun Li, Yuwen Lu, Jaylexia Clark, Meng Chen, Victor Cox, Meng Jiang, Yang Yang, Tamara Kay, Danielle Wood, and Jay Brockman. 2022.  A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality.  https://doi.org/10.48550/arXiv.2204.13842 Publication Title: arXiv e-prints ADS Bibcode: 2022arXiv220413842J.
  * Kordzadeh and Ghasemaghaei (2022) Nima Kordzadeh and Maryam Ghasemaghaei. 2022.  Algorithmic bias: review, synthesis, and future research directions.  _European Journal of Information Systems_ 31, 3 (May 2022), 388–409.  https://doi.org/10.1080/0960085X.2021.1927212 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/0960085X.2021.1927212.
  * Kramer et al. (2014) Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. 2014.  Experimental evidence of massive-scale emotional contagion through social networks.  _Proceedings of the National Academy of Sciences of the United States of America_ 111, 24 (June 2014), 8788–8790.  https://doi.org/10.1073/pnas.1320040111
  * Krieter (2019) Philipp Krieter. 2019.  Can I record your screen? mobile screen recordings as a long-term data source for user studies. In _Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia_ _(MUM ’19)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/3365610.3365618
  * Li et al. (2017) Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017.  SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. ACM, Denver Colorado USA, 6038–6049.  https://doi.org/10.1145/3025453.3025483
  * Li et al. (2020) Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, and Brad A Myers. 2020.  Privacy-preserving script sharing in gui-based programming-by-demonstration systems.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW1 (2020), 1–23.
  * Li et al. (2021) Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021.  Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3411764.3445049
  * Panch et al. (2021) Trishan Panch, Heather Mattie, and Rifat Atun. 2021.  Artificial intelligence and algorithmic bias: implications for health systems.  _Journal of Global Health_ 9, 2 (June 2021), 020318.  https://doi.org/10.7189/jogh.09.020318
  * Posner and Weyl (2018) Eric A. Posner and E. Glen Weyl. 2018.  Radical Markets: Uprooting Capitalism and Democracy for a Just Society.  In _Radical Markets_. Princeton University Press.  https://doi.org/10.23943/9781400889457
  * Raento et al. (2009) Mika Raento, Antti Oulasvirta, and Nathan Eagle. 2009.  Smartphones: An emerging tool for social scientists.  _Sociological Methods & Research_ 37, 3 (2009), 426–454.  https://doi.org/10.1177/0049124108330005 Place: US Publisher: Sage Publications.
  * Reddit (2023) Staff Reddit. 2023.  Key Facts to Understanding Reddit’s Recent API Updates - Upvoted.  https://www.redditinc.com/blog/apifacts
  * Robinson et al. (2017) Thomas N. Robinson, Jorge A. Banda, Lauren Hale, Amy Shirong Lu, Frances Fleming-Milici, Sandra L. Calvert, and Ellen Wartella. 2017.  Screen Media Exposure and Obesity in Children and Adolescents.  _Pediatrics_ 140, Supplement_2 (Nov. 2017), S97–S101.  https://doi.org/10.1542/peds.2016-1758K
  * Schueller et al. (2014) Stephen M. Schueller, Mark Begale, Frank J. Penedo, and David C. Mohr. 2014.  Purple: A Modular System for Developing and Deploying Behavioral Intervention Technologies.  _Journal of Medical Internet Research_ 16, 7 (July 2014), e3376.  https://doi.org/10.2196/jmir.3376 Company: Journal of Medical Internet Research Distributor: Journal of Medical Internet Research Institution: Journal of Medical Internet Research Label: Journal of Medical Internet Research Publisher: JMIR Publications Inc., Toronto, Canada.
  * Starbird et al. (2014) Kate Starbird, Jim Maddock, Mania Orand, Peg Achterman, and Robert M. Mason. 2014.  Rumors, False Flags, and Digital Vigilantes: Misinformation on Twitter after the 2013 Boston Marathon Bombing.  _iConference 2014 Proceedings_ (March 2014).  https://doi.org/10.9776/14308 Publisher: iSchools.
  * Starbird and Palen (2010) Kate Starbird and Leysia Palen. 2010.  Pass It On?: Retweeting in Mass Emergency.  _Pass It On_ (2010).
  * Supanitayanon et al. (2020) Sudarat Supanitayanon, Pon Trairatvorakul, and Weerasak Chonchaiya. 2020.  Screen media exposure in the first 2 years of life and preschool cognitive development: a longitudinal study.  _Pediatric Research_ 88, 6 (Dec. 2020), 894–902.  https://doi.org/10.1038/s41390-020-0831-8 Publisher: Nature Publishing Group.
  * Tangmunarunkit et al. (2015) H. Tangmunarunkit, C. K. Hsieh, B. Longstaff, S. Nolen, J. Jenkins, C. Ketcham, J. Selsky, F. Alquaddoomi, D. George, J. Kang, Z. Khalapyan, J. Ooms, N. Ramanathan, and D. Estrin. 2015.  Ohmage: A General and Extensible End-to-End Participatory Sensing Platform.  _ACM Transactions on Intelligent Systems and Technology_ 6, 3 (May 2015), 1–21.  https://doi.org/10.1145/2717318
  * Thomas et al. (2020) George Thomas, Jason A. Bennie, Katrien De Cocker, Michael J. Ireland, and Stuart J. H. Biddle. 2020.  Screen-based behaviors in Australian adolescents: Longitudinal trends from a 4-year follow-up study.  _Preventive Medicine_ 141 (Dec. 2020), 106258.  https://doi.org/10.1016/j.ypmed.2020.106258
  * Trupthi et al. (2017) M. Trupthi, Suresh Pabboju, and G. Narasimha. 2017.  Sentiment Analysis on Twitter Using Streaming API. In _2017 IEEE 7th International Advance Computing Conference (IACC)_. 915–919.  https://doi.org/10.1109/IACC.2017.0186 ISSN: 2473-3571.
  * Ulbricht and Yeung (2022) Lena Ulbricht and Karen Yeung. 2022.  Algorithmic regulation: A maturing concept for investigating regulation of and through algorithms.  _Regulation & Governance_ 16, 1 (2022), 3–22.  https://doi.org/10.1111/rego.12437 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/rego.12437.
  * van Berkel et al. (2017) Niels van Berkel, Denzil Ferreira, and Vassilis Kostakos. 2017.  The Experience Sampling Method on Mobile Devices.  _Comput. Surveys_ 50, 6 (Dec. 2017), 93:1–93:40.  https://doi.org/10.1145/3123988
  * Vieweg et al. (2010) Sarah Vieweg, Amanda L. Hughes, Kate Starbird, and Leysia Palen. 2010.  Microblogging during two natural hazards events: what twitter may contribute to situational awareness. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’10)_. Association for Computing Machinery, New York, NY, USA, 1079–1088.  https://doi.org/10.1145/1753326.1753486
  * Wang et al. (2023) Bryan Wang, Gang Li, and Yang Li. 2023.  Enabling Conversational Interaction with Mobile UI using Large Language Models.  https://doi.org/10.48550/arXiv.2209.08655 arXiv:2209.08655 [cs].
  * Wang et al. (2021) Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021.  Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning.  https://doi.org/10.48550/arXiv.2108.03353 arXiv:2108.03353 [cs].
  * Yan et al. (2023) An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. 2023.  GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation.  https://doi.org/10.48550/arXiv.2311.07562 arXiv:2311.07562 [cs].
  * Yue et al. (2014) Zhen Yue, Eden Litt, Carrie J. Cai, Jeff Stern, Kathy K. Baxter, Zhiwei Guan, Nikhil Sharma, and Guangqiang (George) Zhang. 2014.  Photographing information needs: the role of photos in experience sampling method-style research. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’14)_. Association for Computing Machinery, New York, NY, USA, 1545–1554.  https://doi.org/10.1145/2556288.2557192

##  Appendix A Appendix

###  A.1. Details of Our Best Query Selection Prompt

As described in Section 3.2.2, we used large language models to select the
best Graph Query that most accurately matches the researcher’s data collection
intention. Specifically, our prompt uses the following structure:

We defined this new graph query to locate relevant data on mobile UI
structure. Now, we have generated the following potential queries:

  1. (1)

…(first query)

  2. (2)

…(second query)

  3. (3)

…(etc.)

Which one matches the following description the best?

> [User description of the target data]

Please only return the index number, without any additional information, even
periods or comma.

Example response format: 0

Example generated graph query candidates can be found in Figure 3 section 2.
In the user description of the data to collect, in our user interfaces, we
encourage the user to explain the target data’s relations with other content
on screen, e.g., “Instagram Story advertisement title, the brand name above
text Sponsored”.