# Learning When to Say Goodbye: What Should be the Shelf Life of an Indicator
of Compromise?

Breno Tostes1, Leonardo Ventura1, Enrico Lovat3, Matheus Martins3, Daniel
Menasché1  1Federal University of Rio de Janeiro (UFRJ), 3Siemens Corporation

###### Abstract

Indicators of Compromise (IOCs), such as IP addresses, file hashes, and domain
names associated with known malware or attacks, are cornerstones of
cybersecurity, serving to identify malicious activity on a network. In this
work, we leverage real data to compare different parameterizations of IOC
aging models. Our dataset comprises traffic at a real environment for more
than 1 year. Among our trace-driven findings, we determine thresholds for the
ratio between miss over monitoring costs such that the system benefits from
storing IOCs for a finite time-to-live (TTL) before eviction. To the best of
our knowledge, this is the first real world evaluation of thresholds related
to IOC aging, paving the way towards realistic IOC decaying models.

###### Index Terms:

Threat intelligence; data science; modeling and analysis.

##  I Introduction

Indicators of Compromise (IOCs), such as IP addresses of compromised hosts,
hashes of malware and bodies of emails of phishing campaigns, are the
foundation of cyber threat intelligence (TI). They serve as signatures of
risk, being employed in monitoring systems to generate alerts if a match is
found between known IOCs and data collected at a given environment. In
essence, the larger the IOC base, the greater is the coverage against
previously observed cyber-attacks. However, such coverage is associated with
its own costs [1].

The security of the target environment could be at risk if the number of
monitored IOCs is limited, as it may lead to the omission of crucial
indicators. On the other hand, maintaining too many IOCs is prohibitive due to
intrinsic costs of investigating a large catalog of potential incidents [2].
Understanding the dynamics of IOC creation and sightings is crucial for
addressing the challenges of maintaining and using IOCs, and for developing
effective strategies for monitoring and responding to cyber threats.

Figure 1 shows the typical dynamics of IOC creation and sightings. An IOC is
typically discovered at a vendor or TI source, e.g., in a controlled lab
environment or through a honeypot. Then, the IOC is created and published by
that vendor, and is also propagated to sharing platforms, such as the Malware
Information Sharing Platform (MISP). MISP is a distributed system comprising
multiple instances, run by different organizations and communities, and
allowing them to benefit from the collective knowledge. Finally, Security
Information and Event Management (SIEM) systems monitor the IOCs, and
eventually report sightings for the monitored IOCs at the Security Operation
Centers (SOCs).

Over time, IOCs lose relevance and their monitoring leads to costs due to
outdated information. Indeed, numerous false positives challenges cognitive
limitations of SOC employees, and builds monetary costs that vary depending on
monitoring price models. Azure Sentinel Threat Intelligence, for instance,
offers two pricing alternatives: Capacity Reservations and Pay-As-You-Go. In
the latter, the current cost is of $2.46 per GB-ingested. If the presence of
IOCs is used to pre-filter data to be fed to Azure, the larger the number of
IOCs being monitored, the larger the incurred costs [3].

Figure 1: Ecosystem of vendors, MISP instances and SIEMs sharing IOCs and
their sightings.

To cope with the aging of IOCs, threat intelligence platforms, such as MISP,
have introduced models to determine when an IOC should no longer be monitored.
Those models, referred to as aging models (or decaying models), however, have
a number of parameters whose assignment poses its own challenges, thus
motivating explainable models with simpler and fewer parameters [4]. In
particular, we focus on a model with two main parameters, namely the missing
and monitoring costs, related to the impact of missing a sighting and to the
attention span required to handle alarms, respectively. Our study aims to
answer two key questions: How long should a certain indicator be monitored
for, and what is the optimal aging model parameterization for a given
environment?

To answer the above questions, we leverage real data to compare different
parameterizations of IOC aging models. Our dataset comprises traffic from a
real-world enterprise environment spanning over a year. It contains for each
IOC the instants at which sightings occurred (see Figure 2). Those sightings
are used to parametrize aging models.

Figure 2: Dynamics of IOC creations and sightings.

In the MISP terminology, an event refers to a collection of related IOCs. A
given IOC may be part of multiple events, but each sighting corresponds to a
single event. In summary, our dataset contains, for each sighting towards each
IOC, its 1) timestamp, 2) IOC anonymized identifier, 3) IOC type, e.g.,
domain, IP source, IP destination, email source, email subject, md5, sha1,
sha256, filename, hostname or URL, 4) IOC creation date, and 5) event
identifier. Note that data is anonymized, so the event is characterized by a
non-informative identifier, that serves solely to determine how sightings are
related to each other through their corresponding events.

Using our dataset, we characterize how the number of sightings towards IOCs
varies over time, as a function of IOC types, and assess the impact of aging
model parameters on coverage and corresponding monitoring costs [5]. The _hit
ratio_ corresponds to the fraction of IOC sightings that occur while the
corresponding IOC is being monitored. A sighting to an unmonitored IOC is said
to be uncovered, contributing towards its _miss ratio_. Correspondingly, the
_monitoring cost_ at any given point in time is proportional to the number of
IOCs that are being monitored at that time.

Contributions.  Our key contributions are twofold.

Formulation of the TTL optimization problem.  To each IOC we associate a
corresponding TTL. The TTL is initialized at a constant value, and is
decremented at every time unit. When TTL reaches zero, the corresponding IOC
monitoring is discontinued. Such a TTL decaying model has a number of
different flavors [6, 7, 8]. Under TTL with reset, the TTL is reset to its
initial value whenever a sighting occurs. Under TTL without reset, in
contrast, sightings do not impact the TTL dynamics. In any case, note that the
TTL dynamics are decoupled across multiple IOCs. We let $T$ denote the initial
TTL value, and show how $T$ impacts monitoring costs and miss ratio, under TTL
with and without reset.

Trace driven findings.  Among our trace-driven findings, we discover that if
the cost of missing a sighting is below 2,152 times the daily cost of
monitoring an IOC, it is not worth incurring the monitoring costs for any IOC.
Conversely, if the cost of a miss is beyond $10.5$ million dollars, all IOCs
should be constantly monitored, assuming a unitary daily dollar cost for
monitoring an IOC. For values inbetween those two thresholds, the system
benefits from storing IOCs for a finite time-to-live (TTL), which can be set
according to the IOC category. For instance, if the TTL is set to 248 days
then the sum of miss costs and monitoring costs is minimized when the cost of
missing a sighting equals 10,000 times the daily cost of monitoring an IOC. To
the best of our knowledge, this is the first real world evaluation of IOC
aging thresholds.

Outline. The remainder of this paper is organized as follows. In Section II we
introduce our trace. Then, Sections III and IV report our trace-driven
findings. Section V introduces a utility optimization approach to set TTLs and
reports our model-driven findings. Finally, Section VI concludes.

Figure 3: Cumulative fraction of IOCs created and sightings

##  II Dataset Description

Our trace was collected from a SOC of a large scale company, comprising more
than 300,000 employees and gathering data from more than 12 countries, and
contains 5,789 IOCs with at least one sighting. The first and last IOC
creation dates occurred at 9/9/2018 and 9/2/2020.111For the considered
company, sensitive information becomes unclassified after 2 years, as far as
it is anonymized. The last sighting occurred at the same date at which the
last IOC was created. Figure 3 shows the cumulative fraction of IOCs created
during the interval of our trace, and the cumulative fraction of sightings
issued across that interval. Among all sightings, 66% occurred during the
first three months of our monitoring campaign, between 4/24/2019 and 7/29/2019
(see Table I).

It is worth noting that some IOCs may have sightings before their
corresponding creation dates. This is because once an IOC is created, one may,
in retrospect, detect occurrences of the IOC at system logs. Among the IOCs
with sightings in our trace, 9% (530/5,789) contain sightings one day before
their creation dates (see Figure 4). In Appendix A we provide a more detailed
statistical characterization of time between sightings and creation dates,
leveraging survival analysis for this purpose.

######  Motivating question 1.

How are IOCs related to each other?

In our trace, each IOC belongs to one of the eleven types shown in Figure 5.
Most of the observed indicators and IOCs are associated with domains. IPs also
contribute to a considerable proportion of sightings, whereas hashes have the
second-highest number of IOCs.

Recall that each sighting to an IOC belongs to an event, where an event
consists of a collection of sightings to related IOCs over time. Despite the
fact that the IOCs and events in our trace were anonymized, we are still able
to correlate sighting categories through the corresponding events. To that
aim, in Figure 6 each cell corresponds to the percentage of sightings
pertaining to the row’s category that appear in events that also present IOCs
from the column’s category. From this heat map, we can highlight two clusters.

The first cluster, in the bottom right corner, comprises MD5, SHA-1 and
SHA-256 categories, in which we observe that whenever we have an indicator’s
MD5 hash, it is accompanied 33% of the time by its SHA-1 hash and, 46% of the
time, by its SHA-256 hash. Indeed, it is typical to share different hash
values for a given malware or for its variants. The second cluster, in the top
left, suggests that domain names and IP addresses also tend to be shared
through common events.

Table I: General statistics.

Metric | Value
---|---
# IOCs with at least one sighting | 5,789
# IOCs (total) | $\approx 14,000,000$
# sightings | 892,240
# IOCs with first sighting |
before its creation date | 530
trace duration | 724 days
first creation date | 2018-09-09
last creation date | 2020-09-02
first sighting | 2019-04-24
last sighting | 2020-09-02
Conditional metrics for IOCs with at least one sighting
mean time to upcoming sighting, |
per IOC (multiple sightings |
can occur at same day) | 0.6 days
mean time to first sighting, per IOC |
(after creation) | 83 days
average number of sightings, |
per IOC | 154.13
average number of days |
with sightings, per IOC | 2.72 days

Figure 4: Distribution of first sighting minus creation date. Around 9% of the
IOCs have their first sighting one day before creation date, and 43% at the
same day.

(a) Number of sightings per IOC type

(b) Number of IOCs per IOC type

Figure 5: Number of sightings and IOCs per type. Domains correspond to the
vast majority of sightings and IOCs. IPs also yield a significant fraction of
sightings, while hashes have the second-largest number of IOCs.

######  Finding 1.

Hashes are typically sighted in bundles, and the same occurs for IPs and
domain names. The decaying models considered in this work can be used either
to capture the aging of isolated IOCs or bundles of IOCs.

Figure 6: Correlation matrix of IOC categories, obtained from the relation
between IOC sightings and events. Each cell corresponds to the percentage of
sightings pertaining to the row’s category that appear in events that also
have IOCs from the column’s category.

##  III Deterministic bounds and outliers

######  Motivating question 2.

What is a naive upper bound on TTL?

We begin by estimating an upper bound on the TTL value to cover all sightings
for all IOCs. Indeed, a conservative approach towards IOC monitoring consists
in setting TTL to a large enough value that, in retrospect, would have covered
all sightings.

In our trace, the largest gap between the first and last sightings towards an
IOC equals 724 days. Following a conservative monitoring strategy, i.e.,
$T=724$, and assuming an extension towards an infinite trace wherein IOCs are
created at a rate of $\lambda$ IOCs per time unit, it follows from Little’s
law the expected number of IOCs to be monitored at any point in time, in
steady state, equals $724\cdot\lambda$. This estimate, however, is sensitive
to outliers, motivating the use of statistical tools to parameterize TTL in
order to determine when and if IOCs should be evicted.

######  Finding 2.

Setting TTL to the largest gap between IOC creation date and IOC sighting
serves as a conservative upper bound on the TTL value, but this bound is
sensitive to outliers.

Whereas the above discussion accounted for a deterministic upper bound on the
TTL, in what follows we consider a statistical perspective to account for
outliers.

######  Motivating question 3.

How to cope with outliers while trading between monitoring costs and misses?

To cope with outliers and with the need for allowing a certain level of missed
sightings, we consider statistical approaches to parametrize TTLs. In the
simplest setting, we take as inputs the target hit ratio $t$ (with
corresponding miss ratio $1-t$) and the cumulative distribution function (CDF)
of the time between consecutive sightings, $F(x)=P(X<x)$, where $X$ is a
sample from the distribution of the time between sightings. Then, we let
$T=F^{-1}(t)$.

For large values of $t$, this model clearly degenerates to the simple
deterministic bound discussed in the previous paragraph. Smaller values of $t$
allow us to trade-off between coverage and monitoring costs. In our trace, to
capture 90% of sightings towards IOCs related to emails, we must let $T=38$
days. In this case, a 10% reduction in coverage corresponds to a 95% decrease
($1-38/724=0.95$) in monitoring costs.

######  Finding 3.

By reducing TTL, a small decrease in coverage can yield a significant
reduction in monitoring costs. This occurs in part due to outliers.

##  IV Accounting for categories

######  Motivating question 4.

What is the impact of categories on IOC lifetimes?

In our trace we count with eleven IOC types: md5, sha1, sha256, ip-src, ip-
dst, email-subject, email-dst, domain, hostname, filename and url. Each IOC is
associated to exactly one type. Conditioning TTL values to IOC types allows us
to reduce the impact of outliers, which skew the TTL values for the whole
trace but may not impact certain categories.

The categories discussed above can be split or grouped. As an example, the
eleven categories may be grouped into five coarser clusters (see Figure 7):
hashes (md5, sha1, sha256), IPs (ip-src, ip-dst), email (email-subject, email-
dst), host (domain, hostname, url) and filename. In what follows, we refer to
those five clusters of categories simply as _categories_.

Figure 7a shows the cumulative distribution function (CDF) of the time between
sightings, for the five considered categories, respectively. Hashes tend to
linger longer than IPs, corresponding to larger times between sightings.
Indeed, while IPs are dynamic and should eventually be white-listed, hashes
tend to be more stable over time.

Figure 7b shows the CDF of time from IOC creation until first sighting, per
category. For most categories, roughly 70% of the IOCs have their first
sighting soon after the creation date. However, the remaining 30% of IOCs have
their first sightings uniformly distributed throughout the period of
observation. This means that for a significant fraction of IOCs the first
sighting can take more than one year to occur.

######  Finding 4.

There are at least two motivations to reduce TTL values: 1) reduce monitoring
costs and 2) cope with IOCs that expire. The time between sightings for hashes
tends to be larger than for IPs. This motivates setting TTLs for IPs to
smaller values when compared against hashes, not only to reduce monitoring
costs but also to cope with the fact that IP entries in blacklists expire due
to their ephemeral nature.

(a) Time between sightings. Internal plot: zoom accounting for time between
sightings of up to 30 days.

(b) Time between IOC creation date and first sighting, measured in hours.

Figure 7: Time between sightings and time until first sighting (measured in
hours), over different IOC categories. Hashes tend to linger longer than IPs,
showing larger times between sightings and times to first sightings.

##  V Utilities and costs

######  Motivating question 5.

How to quantitatively capture the monitoring and miss costs while setting TTL?

Next, we consider the availability of information about monitoring costs and
costs associated with missing a sighting, to determine the target hit ratio
$t$. Together, these costs can assist in the flexible monitoring of IOCs.
Indeed, such costs, e.g., measured in dollars per time unit and dollars per
missed sighting, respectively, can be used to establish a utility function to
set TTL values in a principled utility-oriented fashion.

Introducing monetary costs may impose additional challenges, as determining
such costs is non-trivial. However, monetary costs may help to convey the role
of the IOC aging model in the considered organization, bridging the gap
between IOC monitoring strategies and other elements of the business workflow.
Monetary costs can be determined exogenously based on related literature or on
information provided by certain products, such as Azure Sentinel Threat
Intelligence (ASTI).

The ratio between monitoring and missing costs can also be estimated in an
endogenous fashion, using data from the collected traces. Indeed, the trace of
sightings implies that if the cost ratio is above a certain threshold, one
should never monitor any IOC (no-monitoring extreme). At the other extreme of
the spectrum, when the ratio is below a lower threshold all IOCs should be
constantly monitored (always-monitoring extreme). Knowing such two thresholds,
and understanding how the cost ratio impacts monitoring strategies, together
with historical information about monitoring practices in a given business,
provides insights on the current and prospective target cost ratios.

To find the two thresholds referred to in the above paragraph, we define TTLs
ranging between 0 and the maximum interval between sightings (see Section
III). For each TTL value we compute, in retrospect, the corresponding
monitoring and missing costs. The monitoring cost is the number of days we
monitor each IOC in our system multiplied by the cost of each day of
monitoring. The missing cost is the number of missed sightings multiplied by
the cost of each miss.

Let $C$ be the total cost, and $C_{\textrm{mon}}$ and $C_{\textrm{miss}}$ be
the monitoring and missed sighting costs, respectively. $C_{\textrm{mon}}$ is
the monitoring cost per IOC per day, and $C_{\textrm{miss}}$ is the miss cost
per missed sighting. Under the above simple model, the total cost is a linear
function of the time that IOCs were monitored (accounting for IOCs with no
sighting) and the number of missed sightings. Let $I$ be the number of IOCs:

$\displaystyle
C(N_{\textrm{mon}},N_{\textrm{miss}};C_{\textrm{mon}},C_{\textrm{miss}})=$ (1)
$\displaystyle\quad=C_{\textrm{mon}}\sum_{i=1}^{I}N_{\textrm{mon}}^{(i)}+C_{\textrm{miss}}\sum_{i=1}^{I}N_{\textrm{miss}}^{(i)}$
(2)
$\displaystyle\quad=C_{\textrm{mon}}N_{\textrm{mon}}+C_{\textrm{miss}}N_{\textrm{miss}}$
(3)

where $N_{\textrm{mon}}^{(i)}$ and $N_{\textrm{miss}}^{(i)}$ are the
monitoring time and number of missed sightings for the $i$-th IOC, and
$N_{\textrm{mon}}$ and $N_{\textrm{miss}}$ are the corresponding quantities
accounting for all IOCs. Note that $N_{\textrm{mon}}$ and $N_{\textrm{miss}}$
are functions of $T$. Indeed, as many IOCs receive no sightings, we have

$\displaystyle N_{\textrm{mon}}^{(i)}=$
$\displaystyle=\left\\{\begin{array}[]{ll}T,&\textrm{if IOC }i\textrm{ has no
sightings}\\\ T+\Delta T^{(i)},&\textrm{otherwise}\end{array}\right.$ (6)

where $\Delta T^{(i)}$ is the additional number of days at which IOC $i$ is
monitored beyond $T$. Then,

$N_{\textrm{mon}}=T\cdot I+\sum_{i=1}^{I}\Delta T^{(i)}$ (7)

where $\Delta T^{(i)}=0$ for TTL without reset, and can be positive for TTL
with reset.

The dependency of $N_{\textrm{mon}}$ and $N_{\textrm{miss}}$ on $T$ may be
non-trivial, e.g., non-convex. In what follows, we denote the functions that
capture such dependencies and that map $T$ into $N_{\textrm{mon}}$ and
$N_{\textrm{miss}}$ as $g(\cdot)$ and $h(\cdot)$,

$\displaystyle N_{\textrm{mon}}$ $\displaystyle=g(T)$ (8) $\displaystyle
N_{\textrm{miss}}$ $\displaystyle=h(T)$ (9)

We first provide some back-of-the-envelope calculations to get some insights
on how cost impacts the optimal TTL, and the proceed with a trace-driven
exploration to 1) determine the best TTL value, given the costs of monitoring
and missing and 2) search for the two cost ratios that correspond to the
extremal thresholds discussed above.

######  Finding 5.

The impact of monitoring and miss costs can be captured through an utility
function consisting of the sum of two costs, which vary in a non-linear
fashion with respect to TTL values. To deal with such non-linearity, one
alternative is to approach the problem of finding the optimal TTL through a
trace-driven perspective.

###  V-A Back-of-the-envelope calculations

######  Motivating question 6.

Is it feasible to get rough estimates of recommended TTL values without
leveraging the full trace of sightings in its details?

Next, we aim at providing initial insights on the proposed cost model,
leveraging data from our traces. We begin by revisiting Equation (7). In
particular, we consider a simple workload model wherein sightings to IOC $i$
arrive according to a Poisson process with rate $\lambda_{i}$. Then, under TTL
with reset, the probability that the TTL is not reset during an interval of
$T$ seconds is given by $\exp(-\lambda_{i}T)$, which corresponds to the
probability that no sighting arrives during that interval. In that case,
assuming that every reset increases the monitoring period by $T/2$, we have

$\Delta T^{(i)}=Te^{\lambda_{i}T}/2.$ (10)

According to the above model, the cost of monitoring increases exponentially
with $T$. However, our trace-driven analysis shows that in practice, the
monitoring cost increases linearly as $T$ increases. This is because we
observed that sightings of a particular IOC tend to occur in bursts, meaning
that many sightings occur on the same day. To estimate the arrival rate of
sightings that contribute to resets, we count all sightings in a burst as a
single burst arrival. We found that the average number of daily bursts of
sightings per IOC is 2.72 (as shown in the last line of Table I), which is
quite small compared to the overall trace duration. As a result, $\lambda_{i}$
approaches zero, causing the monitoring cost to increase roughly linearly with
respect to $T$ in the scenarios examined in the following section.

Next, we provide rough estimates of two thresholds on the ratio between
monitoring costs and missed sighting costs, according to which it is
beneficial never to monitor any IOC or always to monitor all IOCs. Let

$R=\frac{C_{\textrm{mon}}}{C_{\textrm{miss}}}.$ (11)

For the first threshold, that we refer to as $R_{U}$, we have that

$\displaystyle R\geq R_{U}\Rightarrow T^{\star}=0$ (12)

where $T^{\star}$ is the optimal threshold. Similarly, for the second
threshold, referred to as $R_{L},$

$\displaystyle R\leq R_{L}\Rightarrow T^{\star}=\tilde{T}$ (13)

where $\tilde{T}$ is the maximum admitted threshold.

In the following section, we formally pose optimization problems that yield
the above thresholds, and proceed with a trace-driven simulation to determine
$R_{L}$ and $R_{U}$. The results are shown in Table II. Alternatively, in what
follows we provide some back-of-the-envelope heuristics to approximate those
thresholds.

The total number of sightings in our trace is $892,240$, spread over 724 days
and across roughly 14 million IOCs, most of which having no sightings.
Assuming a normalized unitary monitoring cost per IOC per day, the monitoring
cost to monitor a single IOC during the interval of interest will be of 724
normalized monetary units. The gain, in contrast, will be on average of
$C_{\textrm{miss}}\cdot 892,240/(14\cdot 10^{6})$, assuming that the
probability that the monitored IOC will be sighted is proportional to the
fraction of IOCs that receive at least one sighting. In particular, in our
trace we have an average of $892,240/5,789$ sightings per IOC, conditioned on
IOCs that have at least one sighting, and a fraction of $5,789/(14\cdot
10^{6})$ IOCs that are sighted. Multiplying the two quantities, we obtain the
expected number of sightings covered by an IOC. Therefore, it is worth
monitoring at least one IOC per day if $C_{\textrm{mon}}\cdot
724<C_{\textrm{miss}}\cdot 892,240/(14\cdot 10^{6})$. Letting
$C_{\textrm{mon}}=1$, if $1/C_{\textrm{miss}}>892,240/(14\cdot 10^{6}\cdot
724)=1/11,363$ it is not worth monitoring even one IOC per day. As shown in
Table II, indeed such ballpark value is on the same order of magnitude of
1/2,115 that we found in our trace-driven evaluations as the cost ratio above
which monitoring is not worthy.

On the other extreme of the spectrum, if the cost of a miss at a given day,
$C_{\textrm{miss}}$, is larger than the cost of monitoring all IOCs during
that day, one should always monitor all IOCs. Given that we have roughly $14$
million IOCs, this amounts to always monitoring if $C_{\textrm{miss}}>14\cdot
10^{6}$. Again, the order of magnitude is in agreement with the results
presented in Table II. The difference between the values in Table II and the
above back-of-the-envelope calculations are due to a number of factors,
including 1) the fact that IOCs are created in a non-uniform fashion over the
trace and 2) sightings tend to occur in bursts. To capture those details, we
perform a trace-driven evaluation as further discussed in the next section.

######  Finding 6.

The ratio between monitoring costs and miss costs plays a key role in
determining when it is optimal to never monitor any IOC, or to always monitor
all IOCs. Those two strategies are optimal when the miss cost is on the order
of thousands and millions, respectively, when compared against unitary
monitoring costs.

###  V-B Model and optimization problem

######  Motivating question 7.

How does a detailed trace analysis compare against back of the envelope
calculations to determine optimal TTL values?

The optimization problem corresponding to the optimal TTL estimation is given
as follows:

$\displaystyle T^{*}:\quad$ $\displaystyle\textrm{Argmin}_{T}$ $\displaystyle
C(N_{\textrm{mon}},N_{\textrm{miss}};C_{\textrm{mon}},C_{\textrm{miss}})$
Subject to $\displaystyle N_{\textrm{mon}}=g(T)$ $\displaystyle
N_{\textrm{miss}}=h(T)$

Recall that $R=C_{\textrm{mon}}/C_{\textrm{miss}}$ (see Equation 11). Then,

$\displaystyle\textrm{Argmin}_{T}C(N_{\textrm{mon}},N_{\textrm{miss}};C_{\textrm{mon}},C_{\textrm{miss}})=$
$\displaystyle\quad=\textrm{Argmin}_{T}R\cdot g(T)+h(T)$ (14)

In Appendix B we specialize the above optimization problem to determine
$R_{U}$ and $R_{L}$, i.e., the thresholds beyond which IOCs should never be
monitored, or should be always monitored, respectively.

Note that in the above formulation we assumed that functions $g(\cdot)$ and
$h(\cdot)$ are obtained from traces. Alternatively, approximating those
functions through simple expressions may be instrumental to express the
solutions to the above problems in closed-form, which we leave as subject for
future work.

Table II: Trace-driven bounds. The lower bound $R_{L}$ corresponds to the
ratio monitoring cost:miss cost motivating always monitoring, i.e., setting
$T=724$ days. The upper bound $R_{U}$ corresponds to the ratio motivating
never monitoring, i.e., setting $T=0$.

| TTL
---|---
| with reset | without reset
$R_{L}$ | 1:10,504,881 | 1:843,470
$R_{U}$ | 1:2,152 | 1:2,115

Table II shows the values of $R_{L}$ and $R_{U}$ obtained as solutions of the
above optimization problems. Leveraging our traces, we learn that if the cost
of a miss is on the order of millions when compared to the monitoring cost per
IOC per day, it is worth always monitoring all IOCs. Alternatively, if the
miss cost is around 2,000 times the monitoring cost per IOC per day, it is not
worth monitoring any IOC. For intermediary cost values, Figure 8 shows how the
cost varies as a function of TTL.

Figure 8 shows how the monitoring cost, $C_{\textrm{mon}}N_{\textrm{mon}}$,
miss cost, $C_{\textrm{miss}}N_{\textrm{miss}}$, and total cost, $C$, varies
as a function of TTL, $T$. We let $C_{\textrm{mon}}=1$ USD and
$C_{\textrm{miss}}=10,000$ USD, i.e., $R={1}/{10,000}$. The curve is obtained
using TTL with reset. Note that the monitoring cost grows roughly linearly
with respect to $T$, as previously discussed under our back-of-the-envelope
calculations. In addition, note that the miss cost decreases with respect to
$T$, showing a steep decrease around $T=200$. Finally, the optimal TTL equals
248, and occurs around the point where the cost curves corresponding to
monitoring and miss costs cross each other.

Figure 8: Costs as a function of $T$ for
$R=C_{\textrm{mon}}/C_{\textrm{miss}}=1/10,000$. The monitoring cost
$C_{\textrm{mon}}\cdot N_{\textrm{mon}}$ grows with respect to $T$, whereas
the miss cost $C_{\textrm{miss}}\cdot N_{\textrm{miss}}$ decreases. The sum of
the two costs, $C$, has its minimum at $T=248$.

We repeat the above methodology for $C_{\textrm{miss}}$ varying between 1 and
$10^{8}$, keeping $C_{\textrm{mon}}=1$. The results are reported in Figure 9,
that shows how the best $T$ varies as a function of the ratio
$R^{-1}=C_{\textrm{miss}}/C_{\textrm{mon}}$. The figure shows that as
$C_{\textrm{miss}}$ increases from 10,000 to 100,000, the optimal TTL value
rapidly grows from 248 (as discussed in the previous paragraph) to 500 (as
shown in the inside zoom box in Figure 9). The maximum TTL value is 724 days,
which corresponds to the trace duration, and is reached when the miss cost
reaches roughly 10 million times the monitoring costs (as previously reported
in Table II).

######  Finding 7.

The trace-driven analysis is in agreement with the back-of-the-envelope
calculations, and further allows us to assess the optimal TTL values in the
range between 0 and 724 days when the miss costs vary from the order of
thousands to the order of million times the normalized monitoring costs.

Figure 9: Best TTL as a function of miss over monitoring cost ratio
($R^{-1}=C_{\textrm{miss}}/C_{\textrm{mon}}$), under TTL with reset.

##  VI Conclusion

Platforms for information sharing are key elements of the cyber-security
ecosystem [9, 10, 11]. By facilitating the exchange of IOCs and sightings
among stakeholders, such platforms allow SOC employees to prioritize which
IOCs must be monitored and which alerts require immediate attention. Despite
the tremendous success of platforms of that sort, such as MISP, and the
proposed features to capture the aging of IOCs, their users are still
challenged by the parametrization of the aging models. In particular, such
platforms allow users to share sightings, but sensitive information about how
those sightings impact monitoring is typically kept private.

In this work, we provide initial insights obtained from a flow of sightings
towards IOCs in a real world environment. By leveraging a trace of sightings
towards IOCs, we consider the problem of determining optimal schedules for the
monitoring of IOCs. While considering the costs of monitoring IOCs and of
missed sightings, we provide a methodology to determine lower and upper bounds
($R_{L}$ and $R_{U}$, respectively) on the ratio between such costs so that
IOCs should be monitored forever or not monitored at all. Such ratio, together
with traces collected from a SOC, allow operators to tune TTL values according
to their needs. We illustrate the methodology using a trace collected from a
real world environment, and envision that the methodology is applicable across
different businesses.

This work opens up a number of avenues related to IOC decaying models. First,
we simplified the parameterization of an IOC decaying model, mapping it into
the assessment of the ratio between miss costs and monitoring costs, which we
believe is close to the reality of operators. Further assessing how that ratio
varies across environments is left as subject for future work. Second, we
provided a trace-driven solution to the problem of determining the best TTL.
Analytical solutions, e.g., leveraging special properties on how the number of
misses or the monitoring costs vary as a function of the TTL, are also left as
subject for future work.

Acknowledgment:  This work was supported in part by SIEMENS, CAPES, CNPq, and
FAPERJ under Grants 315110/2020-1, E-26/211.144/2019, and E-26/201.376/2021.

## References

  * [1] X. Bouwman, H. Griffioen, J. Egbers, C. Doerr, B. Klievink, and M. Van Eeten, “A different cup of TI? the added value of commercial threat intelligence,” in _USENIX security_ , 2020, pp. 433–450.
  * [2] X. Liao, K. Yuan, X. Wang, Z. Li, L. Xing, and R. Beyah, “Acing the IOC game: Toward automatic discovery and analysis of open-source cyber threat intelligence,” in _SIGSAC Conference on Computer and Communications Security_ , 2016, pp. 755–766.
  * [3] Microsoft, “Azure sentinel,” https://docs.microsoft.com/en-us/azure/architecture/example-scenario/data/sentinel-threat-intelligence and https://azure.microsoft.com/en-us/pricing/details/azure-sentinel/.
  * [4] A. Iklody, G. Wagener, A. Dulaunoy, S. Mokaddem, and C. Wagner, “Decaying indicators of compromise,” _arXiv:1803.11052_ , 2018.
  * [5] S. Romanosky, “Examining the costs and causes of cyber incidents,” _Journal of Cybersecurity_ , vol. 2, no. 2, pp. 121–135, 2016.
  * [6] M. Dehghan, L. Massoulie, D. Towsley, D. S. Menasche, and Y. C. Tay, “A utility optimization approach to network cache design,” _IEEE/ACM Transactions on Networking_ , vol. 27, no. 3, pp. 1013–1027, 2019.
  * [7] S. Mokaddem, G. Wagener, A. Dulaunoy, and A. Iklody, “Taxonomy driven indicator scoring in misp threat intelligence platforms,” _arXiv preprint arXiv:1902.03914_ , 2019.
  * [8] J. Ermerins, N. van Noort, J. de Novais Marques, and L. Velasco, “Scoring model for IoCs by combining open intelligence feeds to reduce false positives,” 2020.
  * [9] L. Miranda, D. Vieira, L. P. de Aguiar, D. S. Menasché, M. A. Bicudo, M. S. Nogueira, M. Martins, L. Ventura, L. Senos, and E. Lovat, “On the flow of software security advisories,” _IEEE Transactions on Network and Service Management_ , vol. 18, no. 2, pp. 1305–1320, 2021.
  * [10] D. Preuveneers and W. Joosen, “Sharing machine learning models as indicators of compromise for cyber threat intelligence,” _Journal of Cybersecurity and Privacy_ , vol. 1, no. 1, pp. 140–163, 2021.
  * [11] K. Sridhar, A. Householder, J. Spring, and D. W. Woods, “Cybersecurity information sharing: Analysing an email corpus of coordinated vulnerability disclosure,” in _The 20th Annual Workshop on the Economics of Information Security_ , 2021.
  * [12] M. Newman, “Power laws, Pareto distributions and Zipf's law,” _Contemporary Physics_ , vol. 46, no. 5, pp. 323–351, sep 2005. [Online]. Available: https://doi.org/10.1080%2F00107510500052444
  * [13] S. Frei, M. May, U. Fiedler, and B. Plattner, “Large-scale vulnerability analysis,” in _Proceedings of the 2006 SIGCOMM Workshop on Large-Scale Attack Defense - LSAD ’06_ , 2006, pp. 131–138.

##  Appendix A Workload Characterization

We provide further insights on the workload of our traces.

###  A-A Curve fitting

######  Motivating question 8.

Does the time between sightings follow a power law?

We plotted the complementary cumulative distribution function (CCDF) of time
between all sightings in our trace, and observed a linear decay typical of a
power law [12]. A similar power law behavior for time between security
incidents was observed in [13]. Motivated by these observations, we selected
Pareto and Weibull distributions to fit our data. The Pareto and Weibull CCDFs
are given by $\overline{F}(x)=\left({k}/{x}\right)^{\alpha}$ and
$\overline{F}(x)=e^{-(x/k)^{\lambda}}$, respectively. The parameters of the
best fits for each category are listed in Table III, for the Pareto and
Weibull fits. Table III also provides insight on the quality of fits. For
instance, IOCs corresponding to hashes (MD5, SHA-1 and SHA-256) tend to be
better captured through a Weibull distribution as opposed to other IOCs
(domains and IPs) that find a good fit under both Pareto and Weibull.

Table III:  Pareto and Weibull fits to time between sightings.

Category | Weibull Parameters | RMSE | Pareto Parameters | RMSE
---|---|---|---|---
Domain | $\lambda=0.2057,k=0.0004$ | $1e^{-5}$ | $\alpha=1.2,k=0.0199$ | $8e^{-6}$
IP Source | $\lambda=0.1191,k=9.65e^{-7}$ | $1e^{-5}$ | $\alpha=0.8,k=0.0020$ | $2e^{-5}$
IP Destination | $\lambda=0.1354,k=3.01e^{-5}$ | $5e^{-5}$ | $\alpha=0.7,k=0.0067$ | $6e^{-5}$
MD5 | $\lambda=0.2731,k=0.2983$ | $1e^{-3}$ | $\alpha=0.7,k=0.1914$ | $2e^{-3}$
SHA1 | $\lambda=0.2735,k=1.3974$ | $3e^{-3}$ | $\alpha=0.6,k=0.3322$ | $7e^{-3}$
SHA256 | $\lambda=0.2990,k=0.6838$ | $1e^{-3}$ | $\alpha=0.7,k=0.2623$ | $3e^{-3}$
Filename | $\lambda=0.1375,k=0.0003$ | $5e^{-4}$ | $\alpha=0.6,k=0.0097$ | $7e^{-4}$
URL | $\lambda=0.2978,k=2.2679$ | $1e^{-5}$ | $\alpha=0.6,k=0.4199$ | $9e^{-3}$

######  Finding 8.

The time between sightings follows a power law, whose parameters are presented
in Table III.

###  A-B Survival analysis

######  Motivating question 9.

How to cope with the fact that many IOCs, for which the exact creation date is
unknown, have their creation dates set at the very beginning of our trace?

The creation dates of IOCs in our trace are lower bounded by September 9,
2018. Around 10% of the IOCs (533 IOCs) were created before that date, but are
attributed such lower bound as a reference (see Figure 3). Under survival
analysis, this sort of reference attribution is referred to as censoring. In
what follows, we apply the standard Kaplan-Meier (KM) survival estimator to
adjust the dataset, filling up missing data for creation dates that occurred
before the reference.

Figure 10 shows the results obtained through the KM estimator. The dotted blue
line shows the empirical CDF of time to first sighting, and the red line shows
the adjusted CDF obtained using the KM estimator. The green line corresponds
to a simple heuristic, wherein for each IOC $i$ for which the creation date
equals the reference value of September 9, 2018, we replace its creation date
by its first sighting minus 724 days, which corresponds to our trace duration.
The rationale is that, for the oldest IOCs, the trace duration is a good proxy
for the time until first sighting. Indeed, Figure 10 indicates that this
heuristic closely captures the behavior of KM, and we adopt it to reassess
optimal TTL values in light of survival analysis, referring to it simply as
KM.

Figure 9 indicates how the adjustment of creation dates impacts optimal TTLs.
Note that after distancing creation dates from the IOC first sighting,
additional time is spent monitoring IOCs while no sightings are observed.
This, in turn, makes monitoring less rewarding, favoring smaller TTL values.

######  Finding 9.

Survival analysis provides a methodology to cope with missing data, and
alllows us to revisit the optimal TTL values in light of missing data. In
particular, after filling up missing data, optimal TTL values increase, given
that unknown creation dates occurred before the beginning of the trace.

##  Appendix B Details on optimization problem

First, we consider the problem of determining an upper bound on the cost ratio
$R_{U}$ beyond which IOCs should never be monitored. Given (14), the
optimization problem is:

$\displaystyle R_{U}:$ $\displaystyle\textrm{Min}\quad R$ Subject to
$\displaystyle\quad\textrm{Argmin}_{T}R\cdot g(T)+h(T)=0$

Note that $g(0)=0$, therefore for $R>R_{U}$ we have $C=C_{\textrm{miss}}\cdot
N_{\textrm{miss}}=C_{\textrm{miss}}\cdot h(T)$. In the regime wherein
monitoring costs are high, no IOCs are monitored and the ultimate cost depends
only on the number of missed sightings.

Figure 10: CDF of time between creation and first sighting. In dotted blue,
the empirical CDF; in red, the Kaplan-Meier (KM) estimator; in blue, a simple
adjustment to the empirical values that produces results similar to KM.

Correspondingly, the optimization problem to determine a lower bound on the
cost ratio $R_{L}$ below which IOCs should always be monitored is:

$\displaystyle R_{L}:$ $\displaystyle\textrm{Max}\quad R$ Subject to
$\displaystyle\quad\textrm{Argmin}_{T}R\cdot g(T)+h(T)=\widetilde{T}$

where $\widetilde{T}$ is the maximum feasible TTL value. Under a trace-driven
approach, $\widetilde{T}$ can be set as the maximum interval between sightings
(see Section III). Assuming $h(\widetilde{T})=0$, i.e., no sightings are
missed when $T{}$ is set to $\widetilde{T}$, condition $R<R_{L}$ implies that
$C/C_{\textrm{miss}}=R\cdot g(T)$. In the regime wherein monitoring costs are
low, all IOCs are monitored and the ultimate cost depends only on the product
$C_{\textrm{mon}}\cdot N_{\textrm{mon}}=C_{\textrm{mon}}\cdot g(T)$.

|  Breno Tostes received his B.Sc. degree in Computer Science from Federal
University of Rio de Janeiro (UFRJ) in 2022. His interests include modeling
and analysis of computer systems, with focus on performance evaluation and
cybersecurity.
---|---
|  Leonardo Ventura received his B.Sc. degree in Computer Science from Federal
University of Rio de Janeiro (UFRJ) in 2022. He is currently a software
engineer at Uber, with focus on cybersecurity.
---|---
|  Enrico Lovat received his PhD from the Technical University of Munich for
his research on the topics of usage control and information flow tracking. He
joined Siemens CERT in 2016 as incident handler. Currently, he is responsible
for Cyber Threat Intelligence at Siemens CERT.
---|---
|  Matheus Martins received the B.S. and M.Sc. degrees in Computer Science and
Informatics from Federal University of Rio de Janeiro (UFRJ), in 2018 and
2020, respectively. His research focuses on data science to create
cybersecurity risk models for patch management. Currently, he is working with
cyber threat intelligence and incident response at Siemens Technology. His
interests include modeling and analysis of systems, with focus on
cybersecurity.
---|---
|  Daniel Sadoc Menasché received the Ph.D. degree in Computer Science from
the University of Massachusetts, Amherst, in 2011. Currently, he is an
Associate Professor with the Computer Science Department, Federal University
of Rio de Janeiro, Brazil. His interests are in modeling, analysis, security
and performance evaluation of computer systems, being a recipient of best
paper awards at GLOBECOM 2007, CoNEXT 2009, INFOCOM 2013 and ICGSE 2015. He is
an alumnus affiliated member of the Brazilian Academy of Sciences.
---|---


# Mobile User Interface Element Detection Via Adaptively Prompt Tuning

Zhangxuan Gu, Zhuoer Xu, Haoxing Chen, Jun Lan, Changhua Meng, Weiqiang Wang
Tiansuan Lab, Ant Group
{guzhangxuan.gzx,xuzhuoer.xze,chenhaoxing.chx,yelan.lj,changhua.mch,weiqiang.wwq}
@antgroup.com

###### Abstract

Recent object detection approaches rely on pretrained vision-language models
for image-text alignment. However, they fail to detect the Mobile User
Interface (MUI) element since it contains additional OCR information, which
describes its content and function but is often ignored. In this paper, we
develop a new MUI element detection dataset named MUI-zh and propose an
Adaptively Prompt Tuning (APT) module to take advantage of discriminating OCR
information. APT is a lightweight and effective module to jointly optimize
category prompts across different modalities. For every element, APT uniformly
encodes its visual features and OCR descriptions to dynamically adjust the
representation of frozen category prompts. We evaluate the effectiveness of
our plug-and-play APT upon several existing CLIP-based detectors for both
standard and open-vocabulary MUI element detection. Extensive experiments show
that our method achieves considerable improvements on two datasets. The
datasets is available at github.com/antmachineintelligence/MUI-zh.

##  1 Introduction

While significant progress has been made in object detection[24, 2, 23, 28,
17], with the development of deep neural networks, less attention has been
paid to its challenging variant in the Mobile User Interface (MUI) domain[1].
Instead of personal computers and books, people nowadays spend more time on
mobile phones due to the convenience of various apps for daily life. However,
there may exist some risks, including illegal gambling[10, 19], malware[32,
31], security[8, 4], privacy[15, 14], copy/fake[27] and fraudulent
behaviors[6, 13] in apps, which need to be detected and alarmed as required by
government authorities and app markets. In apps, these risks may occur in one
element or even hide in the subpage after clicking one element. As a result,
it is in great need of an accurate, robust, and even open-vocabulary MUI
element detection approach in practice. Such technology can benefit a great
variety of scenarios as mentioned above, towards building a better mobile
ecosystem[13, 30].

This paper proposes MUI element detection as a variant object detection task
and develops a corresponding dataset named MUI-zh. In general, object
detection aims to classify and locate each object, such as an animal or a
tool, in one raw image. While in MUI data, our primary concern is detecting
elements, _e.g._ , products and clickable buttons in the screenshots. The main
difference between the two tasks is that MUI data often have discriminative
OCR descriptions as supplemental information for every element, significantly
influencing detection results. To better explain it, we put two MUI data
examples from VINS[1] and our MUI-zh in Figure 1. VINS only provides the
category annotation and bounding box for every element, as the object
detection dataset does. At the same time, MUI-zh additionally obtains the OCR
descriptions and links them with elements for further usage. Since the OCR
descriptions are texts and will be an additional input modality, it is natural
to leverage recent Open-Vocabulary object Detection (OVD) models[22, 37, 40,
36, 3, 11, 20] as the MUI element detection baseline because of their rich
vision-language knowledge learned from pretrained CLIP[21].

Figure 1: Two MUI samples from VINS and MUI-zh dataset. Compared to VINS, we
additionally obtain the OCR descriptions as supplemental information in MUI-
zh. Moreover, we further link OCR descriptions and element annotations with
the same color.

OVD detectors usually detect and classify objects by calculating the
similarity between visual embeddings and textual concepts split from captions.
However, according to our experiments, existing OVD methods can not achieve
satisfactory performances on MUI datasets. The reason mainly comes from two
aspects: Firstly, the samples for training OVD detectors are appearance-
centric, while MUI data is not. Besides the appearance, the category of one
MUI element is often closely related to its textual explanations obtained by
OCR tools. Thus, OCR descriptions of one element can be viewed as a
discriminative modality to distinguish itself from other categories, but
neither exists nor is used in OVD models; Secondly, the category prompts with
only category name is not optimal for vision-language alignment since they may
not be precise enough to describe an MUI element. For example, we show four
buttons (blue) and one icon (red) in Figure 2. The baseline (OVD detector)
only uses “a photo of category name” to perform alignment and misclassify
button 1 as an icon.

Figure 2: Decision boundaries of baseline and adding APT during vision-
language alignment. The stars are category prompts, and the circles are
element vision embeddings. Element 1 is misclassified by baseline while our
APT tunes its category prompts adaptively and thus successfully matches it and
its category.

To alleviate the above issues, we propose a novel lightweight and plug-and-
play Adaptively Prompt Tuning (APT) module in MUI element detection. Firstly,
it takes OCR descriptions as input, using a unimodal block to obtain rich
elements’ information (_e.g._ , content and function) for vision-language
alignment; Secondly, it adaptive encodes vision and OCR description features
into embeddings to adjust the representation of frozen category prompts, which
further reduces the impact of language ambiguity during matching. As shown in
Figure 2, the gray dotted lines indicate the decision boundaries of the OVD
baseline and its variant with APT during the recognizing phase. Element 1 is
misclassified by the baseline since its embedding is close to the frozen
category prompt of “icon” and far away from its groundtruth “button”. Our APT
adaptively tunes two category prompts (noted by the green arrow) for every
element and successfully recognizes element 1. As a result, we demonstrate
that the APT can achieve noticeable performance gains based on previous OVD
detectors, which will benefit many mobile layout analyses[35, 34] and risk
hunters[4, 10]. We summarize our contributions as follows.

  * •

We develop a high-quality MUI dataset (called MUI-zh) containing 18 common
categories with OCR descriptions as the supplemental information. Besides MUI-
zh, we will also provide the OCR descriptions of the existing dataset VINS to
facilitate future research.

  * •

Inspired by the MUI data characteristics, we further proposed a novel Adaptive
Prompt Tuning (APT) module to finetune category prompts for standard and open-
vocabulary MUI element detection.

  * •

Experiments on two datasets demonstrate that our APT, as a plug-and-play
module, achieves competitive improvements upon four recent CLIP-based
detectors.

##  2 Related Works

###  2.1 Object Detection

Object detection aims to detect and represent objects at a bounding box level.
There are two kinds of object detection methods, i.e., two-stage[24, 2], and
single-stage[23, 28, 17]. Two-stage methods first detect objects, then crop
their region features to further classify them into the foreground or
background. In contrast, the one-stage detectors directly predict the category
and bounding box at each location.

###  2.2 Open-vocabulary Object Detection

Relying heavily on visual-language pretrained models[21], open-vocabulary
object detection approaches aim to locate and classify novel objects that are
not included in the training data. Recently, OVD methods[7, 22, 37, 40, 36, 3,
11, 20] follow two-stage fashion: class-agnostic proposals are firstly
generated by RPN[24] trained on base categories, then the classification head
is required to recognize novel classes with the knowledge from pretrained
CLIP[21].

The representative solutions include OVR-CNN[33] and ViLD[11]. Taking Faster
RCNN[24] as the backbone, OVR-CNN[33] trains a projection layer on image-text
pairs with contrastive learning, while ViLD[11] proposes to explicitly distill
the knowledge from the pretrained CLIP visual encoder. Advanced to them,
Detic[40] tries to self-train the detector on ImageNet21K[25] for OVD.
Recently, VL-PLM[36] use self-training in both two stages on unlabeled data
and MEDet[3] proposes an online proposal mining method to refine the vision-
language alignment. Following Detic[40], Object-centric OVD[22] combines
knowledge distillation and contrastive learning, achieving the best
performance on COCO[16] with the extra weakly supervised data from
ImageNet21K. One closely related work is RegionCLIP[37], which leverages a
CLIP model to match image regions with template texts on large-scale data from
the web and then uses pseudo pairs to train the fine-grained alignment between
image regions and text spans.

###  2.3 Prompts Learning

The large vision-language model, _e.g._ , CLIP[21], has significantly improved
many few-shot or zero-shot computer vision tasks. They are often pretrained on
a large amount of image-text pairs collected from the web and can be easily
transferred to numerous downstream tasks with either finetuning[18, 26] or
prompt learning[39]. From [21], we can observe that a task-specific prompt can
boost performance significantly but needs carefully tuning prompts by humans.
As its extension, CoOp[39] proposes context optimization with learnable
vectors for automating prompt learning in few-shot classification, relieving
the burden of designing hand-craft prompts by humans. Moreover, its further
extension CoCoOp[38] learns a lightweight neural network to generate for each
image an input-conditional token, which improves the generalization ability to
wider novel categories in image classification tasks.

Recently, DetPro[7] and PromptDet[9] adapt CoOp[39] to OVD by designing
particular strategies to handle foreground and background proposals within
images. Although the vision embeddings learned in our APT are somehow inspired
by CoCoOp[38], we are the first to propose a unified module for tuning prompts
on two modalities, _i.e._ , OCR descriptions and vision features.

##  3 Mobile User Interface Dataset

Figure 3: Overview of MUI element detection pipeline associated with our
proposed APT. We first use OCR tools and class-agnostic RPN for input images
to obtain OCR descriptions and element proposals. An IoM module matches and
links the elements and OCR descriptions in the preprocessing phase. Existing
CLIP-based detectors usually encode element proposals and category prompts
into vision ( green) and text ( blue) embeddings by image encoder $\bm{I}$ and
text encoder $\bm{T}$ for similarity calculation. Our APT additionally uses
OCR descriptions ( red) and vision embeddings to tune the text embedding for
better alignment. Best viewed in color.

In this section, we first introduce the existing MUI datasets and our
developed MUI-zh. Then we briefly describe how to match OCR descriptions and
elements.

###  3.1 Dataset Preparation

Early work on the MUI dataset explored how to support humans in designing
applications. For example, Rico[5], a dataset of Android apps, was released
five years ago. It consists of 72k MUI examples from 9722 apps, spanning 27
categories in the Google Play Store. However, the annotations of Rico are
noisy, sometimes even incorrect, according to [1]. As its extension, VINS[1]
uses MUI designs and wireframes to enable element detection and retrieval.

Nowadays, more and more tinyapps (in apps) are developed by merchants, and
their elements, also as MUI data, have a noticeable domain gap with the
elements in Rico and VINS. In order to fully understand MUI data, we develop
MUI-zh, an MUI detection dataset from tinyapp screenshots. MUI-zh has 5769
images of tinyapp screenshots, including 50k elements within 18 categories.
Besides element location and category, we also provide essential OCR
descriptions and locations for every screenshot as supplemental information
for classification. Another reason for developing MUI-zh is that the existing
language of MUI datasets is English. Detectors trained on them can not be used
in another language, such as Chinese, due to the domain gap/bias during
vision-language alignment. Our MUI-zh collects high-quality tinyapp
screenshots in Chinese, which enriches the MUI data for different languages.

###  3.2 OCR Descriptions Matching

After we collect and annotate enough MUI screenshots, we have to link the OCR
descriptions and elements for further usage. How to relate OCR and elements
with their locations is an open question. Intuitively, it is possible to link
them by calculating and ranking their Intersection Over Union (IoU) according
to two series bounding boxes inspired by non-maximum suppression (NMS). For
every element box, we select the OCR boxes whose IoU scores are larger than a
threshold (_e.g._ , 0.5) as its descriptions without replacement. Note that
OCR tools may separate one sentence into many phrases, and as a result, an
element may also be linked to more than one OCR description. Another special
case is when an element box does not have any description, we assign it an
empty word.

Generally speaking, IoU measures how much two proposals overlap and whether
they can be assigned with the same instance in the object detection task.
However, MUI elements like products and buttons are more likely to include
their OCR descriptions (often occupy only a small region, _e.g._ , $10\%$ of
element) within the box. In this case, the IoU (0.1) is smaller than the
threshold and this element fails to match its description, which is
unacceptable. To tackle this problem, we utilize Intersection Over Minimum
(IoM) instead of IoU during OCR matching. IoM replacing the area of union with
the area of the minimum box in IoU is suitable for MUI data. For the case
mentioned above, the IoM is $1$, which means we successfully link the element
and its OCR descriptions. Note that we also conduct OCR matching on VINS and
release the results.

##  4 Methodology

In this section, we first briefly present how existing OVD models detect
elements of MUI data in Section 4.1 and then show the architecture of APT and
how it works in Section 4.2. Finally, we claim how to assemble APT on four
existing detectors in Section 4.3. The whole pipeline of MUI element detection
is shown in Figure 3.

###  4.1 Detectors on MUI

Preprocessing: Given a batch of MUI data, the training pipeline of recent two-
stage CLIP-based detectors follows almost the same scheme (detect-then-
classify). They first use a class-agnostic RPN[24] to obtain element proposals
and perform their innovations and improvements during the classification step.
Our APT additionally considers OCR descriptions while aligning the element
with categories.

Training: Existing CLIP-based detectors mainly focus on the training of the
classifier. Specifically, they first construct human-made prompts (_e.g._ , “a
photo of category name”) and feed them to the frozen language encoder
$\bm{T}(\cdot)$ of pretrained CLIP[21] as the text embeddings. At the same
time, a trainable CLIP visual encoder $\bm{I}(\cdot)$ is adapted to the
detector for encoding element proposals into vision embeddings. Finally, the
classifier learns to match these pair-wise embeddings via contrastive learning
and cross-entropy loss.

Specifically, assuming an image $I$ has $n$ element proposals obtained by RPN,
and we notate their features as
$\\{\mathbf{r}_{i}\\}_{i=1}^{n}\in\mathcal{R}^{d}$. The classifier’s goal is
to match the element proposals with category prompts $\\{{c}_{j}\\}_{j=1}^{m}$
for $m$ different categories. Relying on the powerful CLIP, the text (vision)
embedding $\mathbf{t}_{j}$ ($\mathbf{f}_{i}$) of category $j$ (proposal $i$)
is generated by feeding $c_{j}$ ($\mathbf{r}_{i}$) into the encoders,
respectively:

$\mathbf{t}_{j}=\bm{T}({c}_{j});\mathbf{f}_{i}=\bm{I}(\mathbf{r}_{i}).$ (1)

For a paired proposal $i$ and its groundtruth category $j$ during training, we
can calculate the predicted probability as:

$p_{ij}=\frac{\exp(\cos(\mathbf{t}_{j},\mathbf{f}_{i})/\tau)}{\sum_{k=1}^{m}\exp(\cos(\mathbf{t}_{k},\mathbf{f}_{i})/\tau)},$
(2)

where $\tau$ is a temperature hyper-parameter. Finally, the cross-entropy loss
is applied to optimize the network parameters except for $\bm{T}$ on proposal
$i$:

$L_{i}=-\log(p_{ij}).$ (3)

The reason for freezing $\bm{T}$ is to fully utilize the knowledge learned by
the CLIP pretrained on large-scale data according to [7, 22]. We also conduct
experiments to verify it.

Inference: OVD models predict the category with the probability obtained by
Equation 2 for the element detection task. While performing experiments in the
open-vocabulary setting, we extend the category prompts to cover both base and
novel classes following [37].

###  4.2 Adaptively Prompt Tuning

As we mentioned in Section 1, existing CLIP-based detectors are not
generalizable to MUI categories due to the ignorance of OCR descriptions and
the difficulty of aligning various-appearance elements to one frozen manual
category prompts. To deal with these two weaknesses, we propose an Adaptively
Prompt Tuning (APT) module by mapping OCR descriptions (red) and vision
embeddings (green) into the space of text embeddings to adaptively tune the
category prompts for every element proposal as shown in Figure 3. The figure
shows that the mapped embeddings (red and white) are fused to adjust the
frozen text embeddings (blue) for final alignment with vision embeddings
(green).

For simplicity, we use $\varphi(\cdot)$ to denote the APT and formulate the
training pipeline for image $I$ as:

$\mathbf{o}_{i}=\varphi(\bm{T}({o}_{i}));\mathbf{v}_{i}=\varphi(\mathbf{f}_{i});\mathbf{\hat{t}}_{ji}=\mathbf{t}_{j}+\mathbf{o}_{i}+\mathbf{v}_{i};$
(4)

$\hat{p}_{ij}=\frac{\exp(\cos(\mathbf{\hat{t}}_{ji},\mathbf{f}_{i})/\tau)}{\sum_{k=1}^{m}\exp(\cos(\mathbf{\hat{t}}_{ki},\mathbf{f}_{i})/\tau)},$
(5)

where $\\{o_{i}\\}_{i=1}^{n}$ are the OCR descriptions for $n$ proposals. In
this way, we can optimize the whole model except for $\bm{T}$ with cross-
entropy loss:

$\hat{L}_{i}=-\log(\hat{p}_{ij}).$ (6)

Note that during inference, we also tune the text embeddings in the same way
as training with Equation 4.

Since our goal is to map supplemental information into the embedding space for
prompt tuning, it is natural to uniformly encode OCR descriptions and vision
embeddings to encourage knowledge sharing and interaction from different
modalities. As we know, APT is the first unimodal prompt tuning method,
holding higher performances than individually encoding two modalities with
different network parameters, as shown in our experiments.

Inspired by CoCoOp[38], we construct APT as a lightweight network with only
two bottlenecks, which contains a fully-connected layer (fc) associated with a
batch norm (bn) and a relu activation. It follows standard encoder-decoder
fashion, and the fc is utilized to reduce/enlarge the number of feature
channels (16x). Since the input channel of the visual feature is 1024, the
total number of parameters of APT is about 128k, including the weights and
bias, which have little influence on training and inference speed.

At the end of APT, we also explore how to fuse modality information in three
ways: element-wise sum, element-wise multiply, and fusion with fc. Recall that
the attention mechanism[29] is also influential in modality fusion and feature
extraction. When we choose element-wise sum as the fusion function, our APT
works as an attention layer for different modalities except for the self-
attention part calculated on $\mathbf{t}_{j}$ in equation 4, which is a
constant. If we use fc to learn the weights for fusion, then $\mathbf{t}_{j}$
can also be learned, which means our APT, in this case, has the same function
of attention layers. According to our experiments, we use element-wise sum as
the fusion function due to the slightly higher performance and lower
calculating complexity.

In conclusion, we highlight that our goal of APT is _to adaptively tune frozen
category prompts with the context from every element’s OCR description and
specific vision information_. Another interesting thing is that there exist
many variants of APT. For example, what if we tune the category prompts only
with vision embeddings and tune vision embeddings with OCR descriptions?
Moreover, can we tune vision embeddings by self-attention and OCR descriptions
while leaving category prompts fixed? To explore the influence of different
tuning methods mentioned above, we conduct experiments in Section 5.3.

###  4.3 Assembling APT to CLIP-based Detectors

DetPro[7], PromptDet[9], Object-centric[22], and RegionCLIP[37] are recent
CLIP-based frameworks for OVD. As we mentioned, our APT tunes category prompts
without changing model architectures and thus can be used directly by many OVD
methods. Here we explain how and where to equip them with APT in detail.
Firstly, RegionCLIP[37] and Object-centric[22] use the fixed manual prompts,
and we can easily add APT upon them at the end of the network during
classification. For PromptDet[9] and DetPro[7], they both use the CoOp[39] to
generate trainable category prompts instead of manual ones. Our APT adjusts
that trainable category prompts for fair comparisons in this case.

##  5 Experiments

In this section, we first introduce the implementation details for datasets
and models in Section 5.1. Our main results are APT upon CLIP-based detectors
for both standard and open-vocabulary MUI element detection as shown in
Section 5.2. Moreover, we evaluate the ablations to study model components in
Section 5.3. Since the bounding boxes annotated by Rico[5] are noisy according
to [1], we only conduct experiments on MUI-zh and VINS for comparison.
Finally, we evaluate our APT for the object detection task on COCO[16] in
Section 5.4.

###  5.1 Implementation Details

Datasets. We evaluate our method on two MUI element detection datasets, namely
MUI-zh and VINS[1]. MUI-zh is a high-quality MUI element detection dataset
with screenshots collected from mobile tinyapps. Its training set contains
4769 images and 41k elements, while the validation set has 1000 images and 9k
elements within 18 categories. Another popular MUI dataset is VINS[1], which
contains 3826 training and 981 validation images with 20 categories. For open-
vocabulary element detection, we set the product, icon, button, card, tips,
and menu as the base categories and the remaining 12 elements as novel ones on
MUI-zh. As for VINS, we set background-image, card, text and spinner as four
novel categories and others as base categories.

Training details and metrics. We evaluate MUI element detection performance on
MUI-zh and VINS for both standard and open-vocabulary settings. During
training, the default visual encoder of all models we used in the experiments
is ResNet50[12] from pretrained CLIP[21]. Note that the language encoder is
frozen following [7, 22]. For MUI element detection, SGD is used with a batch
size of 64, an initial learning rate of 0.002, and a maximum iteration of 12
epochs on 8 A100 GPUs. For open-vocabulary element detection, RPN is trained
with the base categories of two datasets. The temperature $\tau$ is 0.01. The
widely-used object detection metrics, including Mean Average Precision (mAP)
for novel and all categories are used.

###  5.2 Main Results of MUI Element Detection

MUI element detection. As shown in Table 1, we list two groups of detection
approaches on both MUI-zh and VINS. The first group is standard object
detection methods like Faster RCNN[24] and Cascaded RCNN[2], while the second
group contains four CLIP-based models.

The table shows that recently proposed Object-centric OVD[22] and
RegionCLIP[37] achieve much better performances than standard object detection
models since MUI data need more attention on vision-language alignment.
Moreover, our APT improves about 4-5% (6-9%) mAP on CLIP-based detectors on
MUI-zh (VINS), which is a significant enhancement and shows APT’s
effectiveness in the MUI element detection task. Among these detectors,
RegionCLIP[37] equipped with APT achieves the best performances (51.23% and
80.84% on MUI-zh and VINS).

Methods | Publication | MUI-zh | VINS
---|---|---|---
VINS[1] | CHI’21 | - | 63.21
Faster RCNN[24] | NeurIPS’15 | 44.63 | 68.89
Cascaded RCNN[2] | CVPR’18 | 46.76 | 72.85
\+ COCO pretrain | 48.80 | 75.77
DetPro[7] | CVPR’22 | 44.55 | 71.67
[7]+APT | - | 48.62(+4.07) | 77.73(+6.06)
PromptDet[9] | ECCV’22 | 40.14 | 68.94
[9]+APT | - | 45.07(+4.93) | 76.43(+7.49)
Object-centric[22] | NeurIPS’22 | 45.87 | 72.36
[22] +APT | - | 50.78(+4.91) | 79.48(+7.12)
RegionCLIP[37] | CVPR’22 | 45.51 | 71.53
[37]+APT | - |  51.23(+5.72) |  80.84(+9.31)

Table 1: Results (mAP%) of MUI element detection. We list the performance of
six popular object detection approaches based on ResNet50. Besides them, we
additionally report the performance gains of our APT module over four recent
CLIP-based models.

Open-vocabulary MUI element detection. One more advantage of CLIP-based
detectors compared to object detection ones is that they can detect objects
not in the predefined categories. To this end, we also conduct experiments on
open-vocabulary MUI element detection, and the results are in Table 2. Here we
compare four recent methods with and without our APT on two datasets. The
table shows that APT achieves noticeable improvements upon the listed methods.
More specifically, among four CLIP-based methods, Object-centric OVD[22] with
APT outperforms others on the MUI-zh, while RegionCLIP associated with APT
gets the best performance on VINS.

Note that even though we have 80% (16/20) base categories on VINS, the
performances of these methods on novel categories still need improvement
compared to OVD detectors on COCO novel categories. There are mainly two
reasons. Firstly, compared to COCO, the category names of MUI data have much
less relation, which causes difficulty for knowledge transfer and embedding
alignment. For example, the knowledge of recognizing cats can be easily
transferred to classify dogs, while it is challenging to utilize the knowledge
of recognizing cards for classifying icons in MUI data. For example, Drawer
and Switch are two MUI categories in VINS. However, they have at least two
meanings (polysemy words) with various appearances (real-world object and MUI
element), making the transfer difficult. Another limitation is the size of the
dataset. For open-vocabulary detection on COCO, there are thousands of
($>$110k) annotated object-caption pairs, not to mention numerous unpaired
data from the web for self-supervised training. At the same time, the recent
MUI datasets only have less than 10k samples for training.

Methods | MUI-zh | VINS
---|---|---
Novel(12) | All(18) | Novel(4) | All(20)
DetPro[7] | 0.54 | 15.99 | 2.03 | 54.68
[7]+APT | 1.41 (+0.87) | 16.93 (+0.94) | 2.74(+0.71) | 54.95(+0.27)
PromptDet[9] | 0.78 | 17.02 | 2.59 | 54.86
[9] +APT | 1.83 (+1.05) | 18.12(+1.10) | 3.02 (+0.43) | 55.18(+0.32)
Object-centric[22] | 1.31 | 17.28 | 3.19 | 55.34
[22]+APT |  2.36(+1.05) |  18.36(+1.08) | 3.76 (+0.57) | 55.49(+0.15)
RegionCLIP[37] | 1.06 | 17.34 | 3.61 | 55.79
[37] +APT | 2.10 (+1.04) | 18.23 (+0.89) |  4.23(+0.62) | 56.80(+1.01)

Table 2: Results (mAP%) of open-vocabulary MUI element detection. We report
the performances of four CLIP-based methods on two datasets. Note that the
number of novel categories of MUI-zh is 12, while VINS has four novel classes.
Our APT improves the results of both novel and all categories.

###  5.3 Ablation Studies

Various APT architectures | MUI-zh | VINS
---|---|---
Ablation | APT($\mathbf{v}_{i}+\mathbf{o}_{i}$) | 51.23 | 80.84
w/o $\mathbf{o}_{i}$ | 47.96 (-3.27) | 75.97(-4.87)
w/o $\mathbf{v}_{i}$ | 48.91 (-2.32) | 76.32(-4.52)
Weights | Share weights | 51.23 | 80.84
Individual weights | 51.01(-0.22) | 79.65 (-1.19)
Layers | 2 (fc+bn+relu) | 51.23 | 80.84
3 (fc+bn+relu) | 51.19 (-0.04) | 80.79 (-0.05)
Tuning |  $\mathbf{t}_{j}+\mathbf{v}_{i}+\mathbf{o}_{i}$ vs. $\mathbf{f}_{i}$ | 51.23 | 80.84
$\mathbf{t}_{j}+\mathbf{o}_{i}$ vs. $\mathbf{f}_{i}+\mathbf{v}_{i}$ | 51.08(-0.15) | 78.23 (-2.61)
$\mathbf{t}_{j}+\mathbf{v}_{i}$ vs. $\mathbf{f}_{i}+\mathbf{o}_{i}$ | 50.23(-0.10) | 78.35(-2.49)
$\mathbf{t}_{j}$ vs. $\mathbf{f}_{i}+\mathbf{v}_{i}+\mathbf{o}_{i}$ | 51.00 (-0.23) | 77.97 (-2.87)
Fusion | Element-wise sum | 51.23 | 80.84
Element-wise multi | 48.83 (-2.40) | 77.65 (-3.19)
Attention(Concat + fc) | 51.17 (-0.06) | 80.59 (-0.25)
Encoder | Freeze $\bm{T}$ | 51.23 | 80.84
Trainable $\bm{T}$ | 45.11(-6.12) | 73.13(-7.71)

Table 3: Ablation studies of APT and its variants. We evaluate APT variants
from several views, including their architecture, ablation and tuning methods.

We perform experiments for the ablation studies on two datasets. First, we
show the impact of progressively integrating our two tuning modalities: the
OCR descriptions $\mathbf{o}_{i}$ and vision embeddings $\mathbf{v}_{i}$, to
the baseline RegionCLIP in Table 3. Then we explore different settings of
weights, layers, tuning methods, and fusion functions, respectively.

Figure 4: T-SNE visualizations. We perform RegionCLIP without and with APT on
two datasets. T-SNE is utilized to visualize their region embeddings. It shows
that APT contributes a lot to vision-language alignment. Best viewed in color
and in-zoom.

Analysis for Components. As shown in Table 3, we first use only the vision
embeddings $\mathbf{v}_{i}$ to tune the category prompts, which decreases
about 3.3% (4.9%) mAP on MUI-zh (VINS). It means that the OCR descriptions of
one element contribute a lot to its classification result. In the next row, we
only equip RegionCLIP with OCR descriptions $\mathbf{o}_{i}$. Removing
$\mathbf{v}_{i}$ leads to a 2.3% (4.5%) decrease in mAP for two datasets,
which means adaptively tuning prompts according to the appearance is also
crucial to the final performance. Overall, the whole improvements of APT upon
baseline RegionCLIP indicate its effectiveness.

Method | Novel | Base | Generalized(17+48)
---|---|---|---
(17) | (48) | Novel | Base | All
RegionCLIP[37] | 35.2 | 57.6 | 31.4 | 57.1 | 50.4
[37] +APT($\mathbf{v}_{i}$) | 36.3 | 57.3 | 32.1 | 57.2 | 50.0
[37] +APT($\mathbf{v}_{i}+\mathbf{o}_{i}$) | 35.9 | 57.7 | 31.8 | 57.3 | 50.6

Table 4: Results (mAP%) of OVD on COCO dataset. We evaluate APT upon
RegionCLIP (backbone ResNet50) following the standard base/novel split setting
for a fair comparison.

Analysis for weights sharing. Our APT is suitable for two different
modalities, as verified in Table 3. We can observe that using a unified
network for encoding OCR and vision embeddings is slightly better than two
individual ones with the same architecture. The reason may be that OCR
descriptions of one element often describe its appearance. Thus, a lightweight
unified network can naturally map two modalities into one semantic space for
prompt tuning.

Figure 5: Visualizations of MUI element detection. We successively visualize
the images and element bounding boxes of ground truth, RegionCLIP and with our
APT. Note that we highlight the differences with the red dotted circle. Best
viewed in color.

Analysis for layers. Besides the weights of APT, we also want to explore its
layer numbers. We compare two settings: 2 or 3 bottlenecks (fc+bn+relu) as
presented in Table 3. With one extra layer, the network performance decreases
a little. So our APT can be lightweight (only two layers with 128k parameters)
and not time-consuming.

Analysis for tuning methods. An important part of APT is how to tune the
prompts. Since our objective function is the similarity between category
prompts and vision embeddings, there are four main ways: tuning only category
prompts, tuning only vision embeddings and tuning both as shown in Table 3. We
choose only to tune the category prompts with both OCR and vision embeddings,
which gets the best performance. We believe the frozen category prompts rather
than the trainable vision embeddings should be tuned adaptively in the MUI
data domain.

Analysis for fusion functions. How to fuse the embeddings from different
modalities also impacts element detection results. We compare element-wise
sum, multiply and concentration with fc. Among them, the element-wise sum
obtains the best performance with no extra parameters. We also believe
employing element-wise sum for embedding fusion makes our APT work like an
attention layer, excluding the self-attention part calculated on fixed text
embeddings.

Analysis for text encoder. We also show the results of whether to freeze the
text encoder $\bm{T}$ in this table. While we train $\bm{T}$ with MUI data, a
large performance drop appears. As a result, we follow [7, 22] to freeze
$\bm{T}$ in this paper.

###  5.4 Generalization on Object Detection

Although our APT is specially designed for MUI element detection with extra
OCR information, it can also be modified to tune the category prompts on
object detection tasks. To this end, we additionally conduct OVD experiments
on COCO[16]. We follow the data split of [37] with 48 base categories and 17
novel categories and we also use the processed data from [37] with 110k
training images and 4836 test images. Since objects in COCO usually have no
OCR descriptions, we directly use their category names as the OCR
descriptions, and thus we can build APT on RegionCLIP for the OVD task.

As shown in Table 4, our APT slightly outperforms RegionCLIP on all metrics
(_e.g._ , 31.7 vs. 31.4 on novel categories) in the generalized setting.
Compared with RegionCLIP in the standard OVD setting, our APT improves novel
categories by about 0.7 mAP but only helps a little on the base categories. We
find that the improvements in novel categories are larger than base ones,
which indicates the effectiveness of APT in knowledge transfer. With these
studies, we conclude that our APT positively impacts MUI element detection and
object detection tasks.

###  5.5 Visualizations

T-SNE plots of region vision embeddings. We have shown that APT can
significantly improve performance over the baseline RegionCLIP. However,
because the CLIP-based models implicitly learn the alignments by calculating
similarity, it is interesting to see their region vision embeddings after
training. We show the t-SNE plots of RegionCLIP and APT region embeddings
(after non-linear dimensionality reduction) of MUI categories on two
validation datasets in Figure 4. We can observe that APT promotes intra-class
compactness and inter-class separation, which benefits the vision-language
alignment. For example, in MUI-zh, our APT separates products and banners
better than RegionCLIP. As for VINS, our model can successfully classify
edittexts and textbuttons, while RegionCLIP can not.

Detection on MUI data. The detection visualizations of RegionCLIP and our APT
on two MUI datasets are shown in Figure 5. We successively visualize the
images, ground truth element boxes, RegionCLIP and ours. The red dotted
circles in this figure highlight the differences. For example, RegionCLIP
misclassifies texts in the fifth column and modal in the sixth column, while
ours does not. It shows that our APT can better detect elements in MUI
datasets.

##  6 Conclusion

In this work, we introduced APT, a lightweight and effective prompts tuning
module for MUI element detection. Our APT contains two modality inputs, _i.e._
, element-wise OCR descriptions and visual features. They are fused and
encoded within the APT to obtain embeddings for category prompt tuning. It
significantly improves performance on existing CLIP-based models and achieves
competitive results on two MUI datasets. We also released MUI-zh, a new MUI
dataset with matched OCR descriptions. In summary, our model and dataset can
benefit various real-world domains, such as robot interaction, information
retrieval, targeted advertising, and attribute extraction on mobile phones. We
hope our work could inspire designing new frameworks to tackle the challenging
MUI element detection tasks.

Limitations. Our work has several limitations that can be further
investigated. (1) The open-vocabulary capabilities of existing models on the
MUI data could be further improved compared to the results on OVD datasets as
mentioned in Section 5.2. (2) Existing methods all rely on the frozen language
encoder from CLIP. We believe the performance drop of unfreezing the language
encoder may be due to the small dataset size.

##  7 Acknowledgements

We thank the contribution of Qiangqiangzhu and his team Tinyapp Ecological
Business Team in Ant Group for establishing MUI-zh dataset. Besides, we also
thank the Business Risk Management—eKYB Team from the Ant Group for their help
in this paper.

## References

  * [1] Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr.  Vins: Visual search for mobile user interface design.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.
  * [2] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In CVPR, 2018.
  * [3] Peixian Chen, Kekai Sheng, Mengdan Zhang, Yunhang Shen, Ke Li, and Chunhua Shen.  Open vocabulary object detection with proposal mining and prediction equalization.  arXiv preprint arXiv:2206.11134, 2022.
  * [4] Zhuo Chen, Jie Liu, Yubo Hu, Lei Wu, Yajin Zhou, Xianhao Liao, and Ke Wang.  Illegal but not malware: An underground economy app detection system based on usage scenario.  arXiv preprint arXiv:2209.01317, 2022.
  * [5] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar.  Rico: A mobile app dataset for building data-driven design applications.  In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 2017.
  * [6] Feng Dong, Haoyu Wang, Li Li, Yao Guo, Tegawendé F Bissyandé, Tianming Liu, Guoai Xu, and Jacques Klein.  Frauddroid: Automated ad fraud detection for android apps.  In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 257–268, 2018.
  * [7] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In CVPR, 2022.
  * [8] Parvez Faruki, Ammar Bharmal, Vijay Laxmi, Vijay Ganmoor, Manoj Singh Gaur, Mauro Conti, and Muttukrishnan Rajarajan.  Android security: a survey of issues, malware penetration, and defenses.  IEEE communications surveys & tutorials, 17(2):998–1022, 2014\.
  * [9] Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma.  Promptdet: Towards open-vocabulary detection using uncurated images.  In ECCV, 2022.
  * [10] Yuhao Gao, Haoyu Wang, Li Li, Xiapu Luo, Guoai Xu, and Xuanzhe Liu.  Demystifying illegal mobile gambling apps.  In Proceedings of the Web Conference 2021, pages 1447–1458, 2021\.
  * [11] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  arXiv preprint arXiv:2104.13921, 2021.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [13] Yangyu Hu, Haoyu Wang, Ren He, Li Li, Gareth Tyson, Ignacio Castro, Yao Guo, Lei Wu, and Guoai Xu.  Mobile app squatting.  In Proceedings of The Web Conference 2020, pages 1727–1738, 2020\.
  * [14] Eunhoe Kim, Sungmin Kim, and Jaeyoung Choi.  Detecting illegally-copied apps on android devices.  In 2013 International Conference on IT Convergence and Security (ICITCS), pages 1–4. IEEE, 2013.
  * [15] Jialiu Lin, Shahriyar Amini, Jason I Hong, Norman Sadeh, Janne Lindqvist, and Joy Zhang.  Expectation and purpose: understanding users’ mental models of mobile app privacy through crowdsourcing.  In Proceedings of the 2012 ACM conference on ubiquitous computing, pages 501–510, 2012.
  * [16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [17] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In ECCV, 2016.
  * [18] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [19] Qian Luo, Jiajia Liu, Jiadai Wang, Yawen Tan, Yurui Cao, and Nei Kato.  Automatic content inspection and forensics for children android apps.  IEEE Internet of Things Journal, 7(8):7123–7134, 2020.
  * [20] Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu.  Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation.  In CVPR, 2022.
  * [21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [22] Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan.  Bridging the gap between object and image-level representations for open-vocabulary detection.  In NIPS, 2022.
  * [23] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In CVPR, 2016.
  * [24] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NIPS, 2015.
  * [25] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115(3):211–252, 2015\.
  * [26] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv preprint arXiv:1908.08530, 2019.
  * [27] Chongbin Tang, Sen Chen, Lingling Fan, Lihua Xu, Yang Liu, Zhushou Tang, and Liang Dou.  A large-scale empirical study on industrial fake apps.  In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 183–192, 2019\.
  * [28] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In ICCV, 2019.
  * [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [30] Nicolas Viennot, Edward Garcia, and Jason Nieh.  A measurement study of google play.  In The 2014 ACM international conference on Measurement and modeling of computer systems, pages 221–233, 2014.
  * [31] Haoyu Wang, Junjun Si, Hao Li, and Yao Guo.  Rmvdroid: towards a reliable android malware dataset with app metadata.  In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pages 404–408. IEEE, 2019.
  * [32] Liu Wang, Ren He, Haoyu Wang, Pengcheng Xia, Yuanchun Li, Lei Wu, Yajin Zhou, Xiapu Luo, Yulei Sui, Yao Guo, et al.  Beyond the virus: A first look at coronavirus-themed mobile malware.  arXiv preprint arXiv:2005.14619, 2020.
  * [33] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In CVPR, 2021.
  * [34] Haoran Zhang, Xuan Song, Yin Long, Tianqi Xia, Kai Fang, Jianqin Zheng, Dou Huang, Ryosuke Shibasaki, and Yongtu Liang.  Mobile phone gps data in urban bicycle-sharing: Layout optimization and emissions reduction analysis.  Applied Energy, 242:138–147, 2019.
  * [35] Mingming Zhang, Guanhua Hou, and Yeh-Cheng Chen.  Effects of interface layout design on mobile learning efficiency: a comparison of interface layouts for mobile learning platform.  Library Hi Tech, (ahead-of-print), 2022.
  * [36] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, Anastasis Stathopoulos, Manmohan Chandraker, Dimitris Metaxas, et al.  Exploiting unlabeled data with vision and language models for object detection.  arXiv preprint arXiv:2207.08954, 2022.
  * [37] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In CVPR, 2022.
  * [38] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In CVPR, 2022.
  * [39] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130(9):2337–2348, 2022\.
  * [40] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In ECCV, 2022.