# Dumviri: Detecting Trackers and Mixed Trackers with a Breakage Detector

He Shuang
University of Toronto  Lianying Zhao
Carleton university  David Lie
University of Toronto

###### Abstract

Previous automatic tracker detection work lacks features to recognize web page
breakage and often resort to manual analysis to assess the breakage caused by
blocking trackers.

We introduce Dumviri, which incorporates a breakage detector that can
automatically detect web page breakage caused by erroneously blocking a
resource that is needed by the page to function properly. This addition allows
Dumviri to prevent functional resources from being misclassified as trackers
and increases overall detection accuracy. We designed Dumviri to take
differential features. We further find that these features are agnostic to
analysis granularity and enable Dumviri to predict tracking resources at the
request field granularity, allowing Dumviri to handle some mixed trackers.

Evaluating Dumviri on 15K pages shows its ability to replicate the labels of
human-generated filter lists with an accuracy of 97.44%. Through a manual
analysis, we found that Dumviri identified previously unreported trackers and
its breakage detector can identify rules that cause web page breakage in
commonly used filter lists like EasyPrivacy. In the case of mixed trackers,
Dumviri, being the first automated mixed tracker detector, achieves a 79.09%
accuracy. We have confirmed 22 previously unreported unique trackers and 26
unique mixed trackers. We promptly reported these findings to privacy
developers, and we will publish our filter lists in uBlock Origin’s extended
syntax.

##  1 Introduction

A significant and ever-growing percentage (42% [6]) of Internet users utilize
content blockers to block trackers and safeguard their privacy. Extensions
like uBlock Origin [19] and AdBlock Plus [37] rank among the most frequently
downloaded browser extensions. However, existing content blockers are far from
flawless; they rely on human-constructed filter lists, such as
EasyPrivacy[15], for effectiveness. Yet, maintaining filter lists is slow [44]
and error-prone [2]. Erroneous rules in filter lists can cause web page
breakage.

Previous proposals have attempted to train machine learning models to
automatically detect trackers [38, 24, 23]. However, these approaches are
prone to overestimating trackers due to their inability to recognize “when a
web page is broken” as a consequence of such overestimation. Consequently, a
functional resource that behave similarly to a tracker is likely to be
misclassified as one, leading to potential web page breakage. Moreover,
previous proposals rely on manual analysis to assess the breakage they cause,
lacking precision and scalability.

Although other works [41] have aimed to automatically detect web page
breakages, to the best of our knowledge, no work has combined automatic
breakage detection and tracker detection. We propose Dumviri 111Dumviri is the
Latin word for “two men” referring to a pair of officials who shared power and
responsibility., which incorporates a breakage detector capable of
automatically detecting web page breakage resulting from tracker blocking.
When given a resource-under-analysis, Dumviri simultaneously addresses the
following questions: 1) “is it used for tracking?” and 2) “does blocking it
lead to web page breakage?”. This combination of detectors enables Dumviri to
make informed choices, minimizing the chance that a functional resource is
misclassified as a tracker.

In designing Dumviri, we overcame the following challenges :  1. No single
dataset contains both tracker samples and breakage samples for the same
website. In fact it is challenging to find relevant breakage samples at all.
We solve this by a) training two machine learning models, each making
individual prediction on a resource-under-analysis b) leveraging exception
rules in filter lists for up-to-date collection of breakage samples that we
can reconstruct.  2. Previous breakage detection proposals do not contain all
the features necessary for accurate breakage detection. We show that features
we employ, which were not previously used, contribute to increased accuracy.
3. We leverage differential features for both detectors, which are features
extracted by comparing the original web page with the same web page with a
resource-under-analysis blocked. We show that differential features can
accurately capture breakage and tracking activities.

Furthermore, we evaluate our tool on mixed trackers, which are trackers that
also perform legitimate web page functionality [40, 4, 3]. They differ from
non-mixed trackers, whose sole purpose is to conduct tracking activities. One
challenge is the absence of a dataset specific to mixed trackers. Despite
this, we find that our models trained on traditional non-mixed trackers
effectively identify tracking request fields.

This paper makes the following contributions:

  * •

It is the first work that incorporates a breakage detector that is capable of
automatically detecting web page breakage into the tracker detection pipeline.
The combination of a tracker detector and a breakage detector enables Dumviri
to maximize tracker detection while minimizing breakages caused by tracker
detection leading to a higher overall tracker detection rate.

  * •

We evaluate the Dumviri’s ability to detect non-mixed trackers on 15K pages.
The result shows that Dumviri can reproduce the labels from human-generated
filter lists with a 97.44% accuracy. Through our manual analysis of the
disagreements between Dumviri and filter lists, we found 55% of instances to
be previously unreported (by filter lists) new trackers. Conversely, 10% of
analyzed cases are filter list-caused web page breakage that Dumviri’s
breakage detector found. We have reported 22 cases of confirmed new tracker
with 175 occurrences in our dataset and 2 instances of confirmed filter list-
caused breakage to the community.

  * •

It is the first work that can automatically detect mixed trackers. Dumviri
makes predictions from the differences of the page execution and states
between a vanilla page and another page with some resource blocked. Such
differential features is analysis granularity agnostic allowing Dumviri to
analyze the impact of blocking tracking request fields and thereby detecting
mixed trackers.

  * •

To find out the prevalence of mixed trackers, we evaluated Dumviri on all
resources that filter lists deemed as non-trackers. Through a manual analysis,
we found that Dumviri can achieve 79.09% accuracy in detecting tracking
parameters. In this process, we have found a total of 26 mixed trackers with
83 occurrences in our evaluation dataset, which we have reported to the
community.

Upon acceptance of this paper, we will make our data, code, models and the
filter lists that Dumviri automatically created for non-mixed and mixed
trackers on 15K pages available.

##  2 Background

Figure 1: Dumviri shares an identical workflow as a human developer: 1) they
both visit the site to start investigating a potential tracker, 2) they both
pick a potential tracker for analysis. Human developer uses expertise while
Dumviri, as an automatic tool, can analyze all resources, 3) they both
evaluate a potential tracker by blocking it, 4) they both evaluate the
resulting page for a) the absence of page breakage and b) the absence of
tracking. Human developers rely on expertise while Dumviri relies its breakage
and tracking detectors. 5) the result is written to a filter list.

We adopt a generic definition of trackers: a tracker is any software piece
(i.e., JavaScript code, cookies, network requests) that enables a server to
(re-)identify users in different computation contexts [42]. While the
computation contexts can be, and are not limited to browsing modes (e.g.,
incognito and regular), sites (i.e., cross-sites tracking) and device (i.e.,
cross-device tracking), the user identification process relies on unique
identifiers (IDs) assigned to users. Traditionally, this ID is stored in
cookies or generated dynamically like a fingerprint. The re-identification of
users is accomplished by sharing user-specific identifiers back to the
tracking server. The transmission process can be done automatically by the
browser (browser-attached cookies) or programmatically through JavaScript
(e.g., using request parameters). In either case, the user-specific identifier
must be unique and thus high-entropy to avoid collision.

Content Blockers and Filter Lists. Content blockers, such as AdBlock Plus
[37], are designed to block unwanted web content. They work by checking
resources against filter lists, which are curated collections of rules that
dictate which resource should be blocked or allowed. EasyList and EasyPrivacy
are the most popular rule lists for ads and trackers, respectively [15].

Figure 2: An example of an exception rule used to ‘fix’ page breakage is when
‘fender.com’ fetches ‘gretel.min.js’ from the domain ‘cdn.cquotient.com’. This
request is blocked as the domain is listed on EasyPrivacy as a tracking
server. However, the particular resource is used for legitimate web page
functionality: it loads the page content. Ultimately, EasyList developers fix
this issue by adding an exception rule allowing the fetch (by disabling the
rule) on ‘fender.com’.

Breakage. Filter lists consist of blacklist rules for blocking matched
resources and exception rules to spare matched resources. A web page breakage
occurs if a user-desired resource is blocked by a blacklist rule. Privacy
developers commonly write exception rules to fix filter list-caused breakages
temporally. We show one example in Fig 2.

One line of work [28, 41] tries to automatically predict web page breakage as
a tool for the manual rule curation pipeline. Blocked or broken [41] draws
features from the differences in PageGraphs [9] in the following categories:
DOM, graph, request, JavaScript DOM accesses and JavaScript dynamic traces.
However, the features they use tend to model trackers than breakages. In this
work, we model both tracker and breakage to improve tracker blocking accuracy
without causing breakage.

Privacy/compatibility Trade-off is a term used in previous work [40, 26] where
they face a choice between maximizing tracker detection and minimizing
breakage. They face such choice, as opposed to achieving both at the same
time, because previous work only considers how similar a resource is to
trackers; they fail to consider the impact of blocking that resource has on
the web page. Such one-sided consideration will inevitably include functional
resources that behave like a tracker.

We believe it is necessary to consider the impact applied to a web page when
blocking a resource. An ideal tracker detector needs to know 1) what resources
look like a tracker and, more importantly, 2) what the impact on the page is
when that resource is blocked. This solves the trade-off as it can maximize
privacy gain by blocking trackers and minimize breakage by not blocking any
resources that are functional.

Mixed Trackers. From a different perspective, we can define traditional non-
mixed trackers to be resources that solely conduct tracking. Another type of
tracker is mixed tracker, which involves dual-purposed resources where
tracking information and functionality-related information are mixed in a
single network request. Existing tracker defenses such as AdBlock Plus are not
designed to defend against mixed trackers, as they make binary decisions to
either block or allow a request at the request granularity. Therefore, they
face a dilemma between page functionality and user privacy. We will further
detail mixed trackers in §5.

Ideal Solution. We outline the properties of an ideal solution. The solution
should  1. maximize tracker blocking,  2. minimize web page breakage,  3. be
automated in detecting trackers, constructing fixes and verifying the fix, and
4. be compatible with the existing AdBlock extension with a large user base.

##  3 The Design of Dumviri

Figure 3: 1) Dumviri crawls the web using two web page instances. Each runs
Dumviri’s instrumented browser with a clean rendering environment and
PageGraph tracing. Both instances share a network cache that minimizes
differences at the network level. 2) Dumviri conducts differential analysis on
the page instances and draws differential features independently for its
detectors. 3) the detectors take the features and make predictions. The
predictions determine if the potential tracker is added to the filter list.

We begin by discussing Dumviri’s general design that enables it to detect non-
mixed trackers. Dumviri mimics how human privacy developers work with trackers
(Fig 1). First, Dumviri visits a target page in its instrumented browser. The
instrumented browser, when rendering a page, produces a detailed trace of the
page rendering process, which Dumviri uses for feature extraction. The trace
includes JavaScript execution control flow, data flow, DOM states (e.g., event
listeners) in a graph representation similar to PageGraph [9], and additional
statistics such as page load time, a full-sized page screenshot.

For every network request that the page sends out, Dumviri asks two questions:
1) does blocking it lead to web page breakage? and 2) is it used for tracking?
Requests used for tracking and whose blocking does not cause page breakage are
considered tracking requests. Conversely, requests causing page breakage upon
blocking or those not used for tracking are not trackers and left as is. This
process produces a list of (request, page) tuples where Dumviri automatically
converts them into rules in EasyPrivacy syntax, facilitating adoption by
existing content blocking extension users.

To answer these two questions, we developed a breakage detector and a tracking
detector. Both detectors are trained machine learning models that take
differential features as inputs. These features are derived from the
differences in the captured traces which includes page states (e.g., visual
changes, event handlers, outgoing requests), JavaScript control flow (e.g.,
invocation of sensitive APIs), and data flow (e.g., flow from cookie jar to
the request) between two page instances: a vanilla page and a page with a
request-under-analysis blocked. The reason behind using differential features,
as opposed to focusing on the blocked request, is twofold: 1) It is easier to
detect breakage when comparing the potentially broken page with a known
working page (i.e., the vanilla page) and 2) Looking at the changed behavior
allows Dumviri to attribute tracking activities to the blocked request. We
show Dumviri’s components in Fig 3 and describe them in detail in the rest of
this section.

###  3.1 Breakage Detector

The breakage detector predicts whether the blocking of a request leads to web
page breakage. In this work, we model breakages from the user’s perspective,
similar to previous works [43]: we focus on modeling web page breakages with
client-side symptoms. Such breakages can be perceived by a human user while
breakages that only have server-side symptoms cannot and thus we exclude such
breakages from our modeling.

Predicting breakage poses the following challenges: 1) how to define breakage
and how to design features to model breakage (§3.1.1) 2) how to distinguish
page breakage from a web page’s non-deterministic behavior (e.g., dynamically
loaded ads, slide shows) (§3.1.2, and finally 3) how to find samples of page
breakage (§3.1.3). We answer these questions in the rest of this section.

Breakage detector | Tracking detector
---|---
Feature | Importance | Uniq |  Feature | Importance | Uniq
$\Delta$ console logs | 3.2 | $\checkmark$ |  $\Delta$ parameters of the blocked request | 14.8 |
$\Delta$ page load time | 2.81 | $\checkmark$ |  $\Delta$ URL length of the blocked request | 8.08 |
$\Delta$ event listeners | 2.25 | $\checkmark$ |  $\Delta$ response size of the blocked request | 3.87 |
$\Delta$ cookies values | 1.07 | $\checkmark$ |  $\Delta$ times first party appear in the blocked request | 3.42 |
$\Delta$ document height | 0.79 | $\checkmark$ |  $\Delta$ ‘eval’ appear in the response of the blocked request | 3.23 |
$\Delta$ CSS classes | 0.73 | $\checkmark$ |  $\Delta$ high entropy fingerprinting function executed | 0.54 |
$\Delta$ DOM tree | 0.38 | $\checkmark$ |  $\Delta$ third party requests blocked | 1.03 |
$\Delta$ listeners on interactable elements such as buttons, inputs | 0.35 | $\checkmark$ |  $\Delta$ requests blocked | 0.3 |
$\Delta$ HTML tag sequences | 0.33 |  |  $\Delta$ third party requests with sensitive information such as identifier-like strings | 0.29 | $\checkmark$
$\Delta$ full-paged screenshot as a feature vector | 0.31 | $\checkmark$ |  $\Delta$ ‘eval’ in the ancestors nodes of the blocked request | 0.24 |
Table 1: Top-10 most important features of Dumviri’s detectors. “Importance”
shows the permutation feature importance using accuracy as the metric,
repeated 5 times. “Uniq” illustrates whether the feature is proposed by and
thus unique to Dumviri.

####  3.1.1 Differential Features

We model web page breakage as changes in browser behavior in at least one
externally visible channel compared to the vanilla functional page. This can
be, for instance, the lack of some user interface (UI) element on the
appearance or the lack of an event listener causing unresponsiveness. We
consider as channels: 1) web page appearance for user perception, 2) user
input handling for interactiveness, 3) network requests for server side
states, 4) writes to persistent storage for client side states, and 5)
temporal performance for user experience.

We draw features from the differences between the vanilla page and a potential
breakage on those channels. Differential features contain more information
than just drawing features from the potential breakage as they will not reveal
the change of functionality. We developed a total of 63 differential features.
We show the top 10 most important features of the breakage detector in Table
1. 9 out of the 10 most important features are unique to Dumviri. We further
show that our breakage detector, which relies on these features, contributes
to Dumviri’s accuracy through an ablation analysis in §4.2.2.

One challenge in designing differential features is the trade-off between
high-level aggregate and low-level features. While this trade-off is
extensively discussed in the literature [7], Dumviri utilizes both types of
features. For instance, regarding appearance change, one aggregate feature of
Dumviri is the screenshot similarity which answers “how similar do the pages
look?”. This is crucial as one of the main symptoms of breakage is a page
failing to load, leading to error pages. However, this feature can be overly
general, as it treats all visual HTML elements equally. For instance, the
absence of a few lines of critical text only has a slight difference in
similarity, but it can have a higher severity than two videos playing at
different positions which causes a larger difference in similarity. Therefore,
Dumviri, in addition to the aggregate features, incorporates low-level
specific features that provide fine-grained information. Again, using
appearance change as an example, one low-level feature is the change in the
number of textual elements.

####  3.1.2 Obtaining Reliable Features

One problem in drawing differential features is the web page non-determinism
behavior, which can arise due to various factors involved in the web page
rendering process such as server-state (e.g., network responses), client-state
(e.g., existing cookies) and characteristics (e.g., user agent) and the web
page itself (e.g., time-sensitive content). Non-determinism behavior adds
noises to differential features that are not caused by the blocking of the
request increasing the likelihood that the breakage detector will mistakenly
predict a breakage. Such erroneous breakage prediction then lowers Dumviri’s
overall accuracy by predicting trackers as non-trackers. To mitigate this one-
way impact on accuracy, Dumviri implements the following measures to minimize
web page non-deterministic:  1. Specialized Environment: Servers may react
based on client states such as existing cookies, user agents (e.g., desktop,
mobile), window sizes, and client locations (i.e., IP addresses). Dumviri
ensures a consistent and clean (e.g., no cookies) pre-loading environment. We
tunnel all our traffic through Bright Data’s [13] network proxy to utilize its
static and data center IP addresses.  2. Network Responses: Some servers may
send back non-deterministic responses, such as third-party ads. This non-
determinism can affect page appearance. To deal with this, Dumviri uses a
network cache to record the communication (e.g., requests and responses) to
the server when rendering the vanilla page and replays the same response when
rendering the second page when an identical request passes through the cache.
3. Network Requests: There may be session-specific requests. For instance,
requests may contain random values as a browser cache buster [30], making
requests appear different across page loads. Dumviri needs to consider such
requests as the same request for 1) response replay and 2) differential
feature construction (e.g., network request related). Thus, Dumviri deploys
fuzzy request matching based on a similarity function that considers the URL,
request type, header fields, and request body.  4. Page Elements: There may be
non-deterministic HTML elements such as time-sensitive elements (e.g., slide
shows, videos). We observed that the layout of the page is usually
deterministic, we use the following method to address non-deterministic
element appearance: for every screenshot, Dumviri takes another screenshot
after a configurable amount of time. The difference between the two
screenshots forms a dynamism mask — these are the time-sensitive and non-
deterministic areas. These areas are excluded from appearance comparison
during feature extraction.  Still, Dumviri cannot eliminate non-determinism
behavior — we found cases where non-determinism behavior causes
mispredictions, which we further discuss in §7.

####  3.1.3 Breakage and Non-breakage Samples

One problem in training the breakage detector is the lack of positive data
points – instances of breakage. While previous works have studied user reports
of breakage on filter lists [33] and commit messages [41], they are unreliable
sources as they can be outdated. Dumviri gathers breakage samples from the
exception rules in current filter lists.

We gather exception rules to reconstruct broken pages. Exception rules are
EasyList-compatible rules that exempt certain requests from being blocked,
which are often used by privacy developers to temporarily "fix" broken pages.
We illustrate one such instance in Fig 2. In this example, the exception rule
being added contains a domain modifier indicating the active domain. We
harvest this domain modifier as a pointer to where we can replicate the
breakage.

Dumviri reconstructs a breakage by flipping the exception rule. Specifically,
Dumviri: 1) navigates to the URL indicated by the domain modifier, and 2)
flips the exception rule from a whitelist rule to a blacklist rule, thereby
blocking the previously spared content. This process enables Dumviri to
reconstruct the breakage that the developer intended to fix.

Not all exception rules can reconstruct the breakage. This can occur when the
rule becomes stale (e.g., breakage fixed by the site but not updated in the
filter list) or if the domain specifier is ambiguous (e.g., the specifier
points to a domain, but the breakage only occurs on a specific page within
that domain). When reconstructing breakage, Dumviri identifies such cases by
monitoring the number of resources blocked by the flipped rule. Flipped
exception rules that do not block any resources are discarded. We gathered a
total of 13,921 exception rules and constructed 2,308 breakages, which were
sampled to ensure correctness.

For the non-breakage samples, we need page changes that do not disrupt
functionality. We want the breakage detector to “accept” legitimate changes.
Therefore, we utilize EasyList and EasyPrivacy to construct non-breakage
samples from pages with these filter lists enabled. While these filter lists
themselves may occasionally be inaccurate and cause breakage, we only collect
non-breakage samples from the top 5K sites on the Alexa Top List. Popular
sites tend to have issues discovered and resolved more quickly due to their
large visitor numbers.

###  3.2 Tracking Detector

The tracking detector predicts whether a request is used for tracking. We
designed differential features in four broad categories: DOM states, requests,
JavaScript control flow, and data flow. DOM state features capture tracking
activities reflected in the DOM tree, such as installing event listeners to
capture user behavior. Request features capture communication between a
tracker and the remote server, as trackers usually rely on network requests to
share sensitive information like user identifiers. Control flow features
capture unique tracking activities, such as invoking high-entropy APIs, which
differ from functional scripts. Data flow features capture information flow,
such as cookies, from one actor (e.g., a network request) to local storage or
other actors (e.g., scripts), which is crucial as many tracking information
(e.g., user identifiers) must be shared with remote servers to complete
tracking. We present the most important features of the tracking detector in
Table 1.

We designed our tracking detector with detecting both types of trackers in
mind. It captures the essence of tracking activities shared by both types of
trackers. We believe that non-mixed trackers solely perform activities (e.g.,
API invocations, DOM accesses) for tracking purposes, which is at the core of
what mixed trackers must also perform. By our definition, mixed trackers
perform tracking along with additional activities related to functionality. We
train our tracking detector using non-mixed trackers, where filter lists
serves as a reliable ground truth, and we believe it should also be able to
predict tracking activities performed by mixed trackers.

##  4 Compare Dumviri to Filter Lists

We evaluate Dumviri to answer the following questions:

  1. 1.

Q1: Do the detectors have predictive power? (§4.1)

  2. 2.

Q2: As a base test, how is Dumviri’s accuracy compared to manually constructed
filter lists such as EasyList and EasyPrivacy? Can it identify additional
trackers? can it identify breakage caused by filter lists? (§4.2)

  3. 3.

Q3: Does Dumviri bring benefit to the existing tracker blocking community?
(§4.3)

###  4.1 Q1 Detector Accuracy

| Training | Test | Cross-validation
---|---|---|---
Detector | Size | Size | Accuracy | F1 | STD
Breakage | 15,854 | 3,171 | 98.30 | 0.9591 | 0.0028
Tracking | 27,721 | 5,545 | 93.62 | 0.9268 | 0.0026
Table 2: The training size, test accuracy, and cross-validation (5 folds)
accuracy of the detectors.

We train our detectors using requests encountered while crawling the top 5K
sites from the Alexa Top List. For each site, we visit the URL, collect all
JavaScript fetching requests, and requests containing query string parameters,
then invoke Dumviri on these requests. The tracking detector was trained on
27,721 requests. Using EasyList and EasyPrivacy (referred to as filter lists
henceforth) as ground truth, we have 12,936 (46.66%) cases of trackers and
14,785 (53.34%) cases of non-trackers. The breakage detector was trained on
15,854 data points, comprising 2,308 reconstructed breakages and 13,546
negative data points of regular trackers. Both detectors were trained using
XGBoost [12].

We perform standard 5-fold cross-validation on the detectors to obtain a
baseline on their classification accuracy on the training set. We report the
mean f1 scores and standard deviations in the last two columns of Table 2.
With a mean f1 score of 0.9591 and 0.9268 for the breakage and tracking
detectors, respectively, we conclude that the models are correctly trained and
have predictive power.

###  4.2 Q2 Comparison to Filter Lists

One problem with manually-constructed filter lists is that they are imperfect:
they can contain false positives (i.e., breakages) and false negatives (i.e.,
missed trackers). Thus, we will first compute a tentative accuracy, followed
by a manual analysis over the disagreements between Dumviri and the filter
lists. In this process, we will verify and demonstrate the new trackers and
filter lists-caused web page breakages. Finally, using the manual analysis
result, we will calculate an adjusted accuracy accounting for filter lists’s
inaccuracies.

Measurement period | 10/22 - 10/27/2023
---|---
# requests analyzed | 53,217
Total space on disk | 8.491 TB
Avg # requests a page sends | 142.13
Avg page load time | 18.06 s
Table 3: Summary of the crawl used to evaluate Dumviri on non-mixed trackers.

Dataset. We construct our dataset using requests observed while crawling 15K
pages from the top, middle, and bottom 5K of the Alexa Top List, covering a
variety of tracking methods. We apply Dumviri to all requests fetching
JavaScript or containing request parameters, as they may transmit tracking
identifiers. This process yields 53,217 labels. We summarize the crawling
statistics in Table 3.

Tentative Accuracy. We compare Dumviri’s prediction to the label from filter
lists. Dumviri achieves a tentative accuracy of 96.53%. There are 1905 cases
of disagreements: 898 cases of “raw false positives” and 1007 cases of “raw
false negatives” We use the term ‘raw’ as they are pending manual
investigation meaning that they may contain true false positives or true false
negatives where filter lists mistake.

In the rest of this section, we will perform a manual analysis over the
disagreements to identify cases of trackers discovered by Dumviri and web page
breakage caused by filter lists, and demonstrate an adjusted accuracy.
Additionally, we will conduct an ablation analysis to demonstrate that Dumviri
cannot achieve its accuracy without its breakage detector.

Request Purpose | Cause Breakage | Label
---|---|---
Functionality | Yes | Functionality
Functionality | No | Stale request
Tracking | Yes | Mixed Tracker
Tracking | No | Tracker
Undecided | Undecided
Table 4: Labels in non-mixed tracker disagreements analysis.

####  4.2.1 Disagreements Analysis

We manually label the disagreements. We manually label the disagreements and
find that Dumviri is capable of identifying trackers missed by filter lists
and instances where filter lists causes page breakage. Using the ratio of true
positives and true negatives in the analyzed disagreements, we estimate that
Dumviri has an accuracy of 97.44%, which we refer to as adjusted accuracy.

Our goal is to assign a request to one of the four labels below: 1) Breakage:
when the request serves for functionality and removing it causes site
breakage, 2) Stale request: when the request serves for functionality yet
removing does not cause immediate breakage, 3) Mixed Tracker: when the request
is a tracker, but its removal breaks the page 4) Tracker: when the request is
a tracker, and removing it does not lead to breakage. 5) Undecided: when we
cannot decide if the request is a tracker or part of the functionality. We
summarize our labels in Table 4. Due to the dataset’s size, we cannot analyze
all disagreements. We sample 40 cases from each type of disagreement, totaling
80 cases for manual labeling.

Figure 4: A Dumviri identified tracker on focus.de with file-level header
comment that leads to external documentation.

Methodology. We describe our approach in detail. To determine if a request is
a tracker, we gather information from online documentation, request initiator,
URL, headers, and response. First of all, we search for any official
documentation of the request. Tracking requests, especially third party, tend
to have official (e.g., Twitter [45], Adobe [1]) or unofficial documentation
(e.g., google gen_204 [21]). We decide on a request to be a tracker if the
documentation clearly states the request’s purpose. Additionally, we refrain
from labeling a request as a tracker if the documentation indicates its
involvement in A/B testing, integration, or optimizations, as we found three
such cases Secondly, we look at the request initiator, if the request
initiator is a known tracking script by filter lists, we deem the request as
tracking. We also look for request-level information such as tracking-related
keywords such as ‘track’ and ‘analytics’. Lastly, if a request has a response,
we look into the response content. Specifically, 1) if the response is a
Javascript and has a file-level header comment, we read the comment as it
often includes references to documentation that describe the file’s purpose
(see Fig 4 for an example), 2) we manually read code if it is not obfuscated
or minimized. We determine the request as tracking if its response contains
fetching or sharing requests to known tracking URLs identified by filter lists
and 3) we check the other resources hosted on the same domain using
DuckDuckGo’s Tracker Radar Wiki [39]. Sometimes, we find documentation in
files on the same domain (e.g., a non-minimized version of an otherwise
minimized file). If, after these steps, the label remains undetermined, we
categorize the request as undecided.

To determine if a request is required for functionality, we follow this
procedure: We first try to find documentation related to the request. Many
functional requests are self-explanatory, and their uses are well documented.
We label a request as functional if we can find a documentation page that
exactly describes the request. While documentation most accurately describes a
request, we could not find documentation for all requests, thus we perform a
second step: using differential analysis to infer the request’s purpose. We
use two page instances: we block the request in one and leave it untouched in
the other. We compare the behavior of the two page instances, any difference
that impacts our, as the user, ability to use the page is breakage.
Specifically, we take the following approach: 1) we observe the page for
visual breakages, such as missing content, iframes, images, and text, 2) we
interact with the pages, including scrolling, clicking on links and buttons,
resizing, hover over buttons and providing inputs to input fields to test
input suggestions. Although we test hyperlinks, we only check if the link
works or not and do not check the referenced page. 3) if the request has a
response, we look into the response for hints on how a page may be impacted.
For instance, if the response JavaScript specifically interacts with a share
button, we look for that share button on the page and see if it is broken. We
decide that a request is required for functionality if there is a clear
association between the request and the change in page behavior between the
two instances. If the purpose of the request remains indeterminate after all
procedures, we designate it as undecided.

Raw False Positives Analysis. We first show the results of analyzing “raw
false positives”, where Dumviri flags a request as tracker but filter lists
say otherwise, summarized in Table 5. In cases where Dumviri flags a request
as tracker and filter lists does not, Dumviri was correct 55% of the time. In
our process of reporting all newly discovered trackers to filter lists
developers, we found that 30% (of all manually analyzed requests) of the cases
are‘ new trackers never unreported before, and we cannot reproduce 25% (of all
manually analyzed requests) of the trackers with filter lists turned on. We
have two explanations for the non-reproducible trackers: 1) they are
probabilistic requests, meaning that they do not always get transmitted (e.g.,
a server samples a subset of their visitors), and 2) they are requests that
are dependent on existing known trackers. We reported all confirmed new
trackers to the privacy developers, who have confirmed and added corresponding
rules to filter lists. To avoid wasting the community’s effort, we did not
report the non-reproducible trackers.

| # | % | Estimated
---|---|---|---
Tracker | 22 | 55% | 494
Breakage | 4 | 10% | 90
Stale requests | 8 | 20% | 180
Undecided | 6 | 15% | 135
Table 5: Manual analysis results for a sample of “raw false positives” where
Dumviri flagged as tracker and filter lists flagged as non-tracker.

Raw False Negatives Analysis. We then show the result of analyzing “raw false
negatives”, where Dumviri flags as non-tracker but filter lists says tracker,
summarized in Table 6. In cases whereDumviri flags as a non-tracker and filter
lists flags as a tracker, 70% are actual trackers contributing to a relatively
high false negative rate. It is not surprising as filter lists are used by
billions of users, and breakages are promptly reported by users and are fixed
quickly. We investigated the predictions from the false negatives, and we
found that they are due to higher than expected predictions by the breakage
detector, which mistakenly flagged a request as functional when it is not and
does not break the page. Since Dumviri has a relatively low false positive
rate, we claim that Dumviri and filter lists find different sets of trackers
and can complement each other. Still, Dumviri was able to find cases of filter
lists-caused page breakage which we detail in §6.

| # | % | Estimated
---|---|---|---
Tracker | 28 | 70% | 705
Breakage | 2 | 5% | 50
Stale requests | 3 | 7.5% | 76
Undecided | 7 | 17.5% | 176
Table 6: Manual analysis results for a sample of “raw false negatives” where
Dumviri flagged as non-tracker and filter lists flagged as tracker.

Adjusted Accuracy. Based on the numbers in Table 5 and 6, we can calculate an
upper bound of the number of false positives to be the sum of breakage, stale
requests and undecided, which is 405 (90+180+135). We can also estimate an
upper bound of the false negatives, counting all known trackers, stale
requests and undecided as false negatives, to be 957 (705+76+176). We can
estimate Dumviri accuracy rate to be 97.44% (((22839+494)+(28473+50))/53217)
which we refer to as the post-adj accuracy, and a f1 score to be 0.9716. The
previous best tracker classifier, AdGraph achieves and accuracy of 95.33%,
which we surpass. Furthermore, AdGraph does not detect breakage caused by
filter lists meaning that their accuracy number is inflated by filter lists-
caused breakages. Dumviri is able to detect breakage in EasyList. We refrain
from direct comparisons with other works due to differences in their recency
and the generality of their approaches. Additionally, in §4.2.2, we
demonstrate the significant contribution of our breakage detector to our
overall accuracy.

Dumviri Findings. In this section, we detail Dumviri’s findings. Due to space
constrains, we cannot discuss all of Dumviri identified tracker. Instead, we
measured the impact of 22 cases of new trackers. We found, by URL, a total of
175 occurrences in our evaluation dataset. This number is a lower bound as
many trackers disguise themselves with different URLs. This shows that the
trackers we found are not rare instances and are used by other sites.

We detail two cases of filter-list caused breakages. One case happens on ero-
advertising.com, where the script www.eroadvertising.com/js/controllers.js is
blocked by EasyPrivacy rule /eroadvertising, this rule is originally designed
to block trackers served from eroadvertising domain, but it is overly
generally as it blocks the functional script controllers.js which loads the
main body content on the site. With EasyPrivacy on, the page is broken as the
main body of the page is missing — the page contains only the header and the
footer as shown on the left side of Fig 8 in Appendix.

Another example is on seznam.cz where script
ssp.seznam.cz/static/js/ssp.js?nocache=1 is blocked by EasyPrivacy rule
||ssp.seznam.cz^. This rule is originally designed to block some trackers from
ssp.seznam.cz domain but is overly generally as ssp.js is responsible for
loading and supporting the cookie consent dialog. With EasyPrivacy on, the
user will not be presented this dialog. A screenshot illustrating the breakage
is shown in Fig 7 in Appendix.

####  4.2.2 Ablation Analysis

In this section, our goal is to show that, without the breakage detector,
Dumviri cannot achieve its accuracy. To demonstrate this, we manually analyze
the top 20 requests where the breakage detector has the highest prediction. We
manually confirm if they are functional resources. Our manual analysis shows
that 7 requests are fetching general purpose JavaScript libraries such as
jquery and setting up lazy loading. We found 7 requests are fetching
JavaScripts that supports specific functionality such as push notification,
payment, font loading, scrolling support, lightbox gallery. We found 6
JavaScript requests that are responsible for content on the page. For each
request, we confirmed, through a method similar to that in §4.2.1, that
excluding the request leads to missing content on the page. The missed content
ranges from icons to sub documents. We are unable to confirm the remaining two
cases. Through this study, we found that all are actually functional resources
meaning that our breakage detector has sufficient prediction power for
detecting resources for web page functionality. We believe that, when our
breakage detector is used with the tracking detector, it can pull back the
misprediction by the tracking detector on functional resources.

###  4.3 Q3 Speed and Cost

Speed. On average, to analyze a single resource, Dumviri needs 363.64 seconds.
This includes performing differential analysis (225.66s, 62.06%), feature
extraction (137.93s, 37.93%) and invoking the detectors (0.05s, 0.01%). The
differential analysis step took the longest as this step is mostly CPU-bound:
Dumviri needs to launch two browser instances and load the page. Dumviri is
single-threaded, and thus, the computation time applies to our 2.4 GHz vCPU
without any parallelism or GPU. We believe this setup is average, and the
computation time is repeatable by others. In the evaluation, we parallelized
72 instances of Dumviri on 72 cores and analyzed 53,217 requests in slightly
less than four days. This evaluation generated 23,737 rules, which translates
to 5,934 (23,737 / 4 days) rules per day. In comparison, filter lists updated
29.8 rules daily [44].

Cost. Based on current spot rate of USD $0.00269/vCPU hour on Google Cloud at
Las Vegas [18], it takes USD $0.02663 ($0.00269*60*60 / 363.64s) to analyze a
single candidate with Dumviri. We estimate that it could cost USD $ 1785.036
(0.02663*67,031) to analyze the whole dataset if ran on Google Cloud. This
translates to an hourly cost of USD $18.59. Dumviri is cost-effective compared
to human developers that manually identify trackers with an average hourly
worker wage of $48.32 [34].

##  5 Adopting Dumviri for Mixed Trackers

Mixed tracker is a type of tracker that is also used for functionality. We
will begin with a mixed tracker example in the wild. After clicking on a
specific product on Duckduckgo’s search page, the page sends out a request
with the following URL: https://www.temu.com/subject/n9/googleshopping-
landingpage-a-psurl.html?goods_id=6010995260893
85&sku_id=17592258865022&_x_ns_msclkid=eeec99c8
3e911b00583ffc4bc3e34060 (additional query string parameters are omitted for
brevity). This URL redirects the user to the desired page. The query string
parameters goods_id and sku_id are legitimate parameters that help redirecting
to the specific product. The final parameter _x_ns_msclkid is a tracking
parameter (assigned by Bing) that records the specific id of the click. In
this example, removing _x_ns_msclkid has no impact on the redirection, but
removing the functional parameters (goods_id and sku_id) will break the
redirection: the URL will land on a generic URL as opposed to the specific
product that the user intends.

This example is a typical use case of mixed tracker where the user is crossing
domains and different network parties (i.e., DuckDuckGo and Temu) need to
communicate (about the click ID). The communication parties have no prior
knowledge of each other, and thus, all information (to be communicated) must
be contained inside a single functional request (i.e., redirection request).

In this section, we will discuss how to extend Dumviri’s methods to detect
mixed trackers. However, one assumption that we have to make in predicting
mixed trackers is that mixed trackers contain single-purposed request fields
(either for tracking or functionality). If all request fields within a request
serve dual purposes, safely removing tracking without compromising web page
functionality—Dumviri’s ultimate goal—becomes unfeasible.

The core property that allows Dumviri to detect mixed trackers is that we can
draw differential features despite the granularity of the blocked resource.
This means that the detectors can make predictions based on differences caused
by the blocking of a request or that caused by the blocking of request fields.
Blocking request fields means removing the fields from the request so that no
more information is transmitted to the server. Dumviri detects mixed trackers
by performing the following steps.  1. It selects request fields of a request
that is potentially used for tracking  2. It draws differential features using
two page instances: one vanilla page and one with the request fields blocked
of the request.  3. It answers two questions: 3a) whether the request fields
are used for tracking and 3b) whether the blocking the request fields lead to
web page breakage. Dumviri answers these questions by invoking its detectors
with differential features.  4. It automatically creates filter rules based on
the detector outputs. We use Ublock Origin’s syntax, which enables us to block
request fields.  ,

Still, one problem Dumviri must solve in the mixed tracker identifications
pipeline concerns step #1: how to identify potential mixed trackers and
potential tracking request fields. While the search space for requests is in
the order of low hundreds, this search space needs to be multiplied by the
search space of all combinations of request fields of a single request.
Assuming, on average, a page sends out $n$ requests and each request has $k$
request fields for analysis, the total search space of tracking request fields
can be modeled as $n*2^{k}$. This number scales quickly with the number of
request fields, which can also be in tens (with, for example, customized
request headers, cookies, query string parameters). This search space, which
can be in the high thousands, is too expensive to exhaustively analyze.

Therefore, Dumviri’s current implementation focuses on analyzing requests that
contain identifier-like strings — a subset of all mixed trackers. Many
previous work [48, 16] and commercial tools [17] have adopted the heuristics
that tracking relies on transmitting user identifiers to remote servers since,
ultimately, tracking servers need to correlate collected information with a
specific user, who is represented as an identifier. Since mixed tracker, in
essence, is a type of tracker, we make use of the same heuristic to narrow
down the potential tracking requests: a request with identifier-like strings
is a potential mixed tracker.  In our current implementation, we designed
Dumviri to analyze requests with identifier-like strings in URL parameters
(i.e., query string parameters and URL parameters) and cookies. While not all
mixed trackers transmit identifiers in those fields, Dumviri can be tuned to
analyze any request fields; as privacy developers discover the pattern of
mixed trackers, they can tune Dumviri to perform analysis on the corresponding
fields. In this aspect, Dumviri does not improve nor is it worse than the
current practice by human privacy developers.

Although our previous text implied that Dumviri performs analysis at request
field granularity (e.g., a single query string parameter), Dumviri can detect
mixing at at sub-field granularity. Dumviri automatically detects if a request
field (i.e., a cookie value, a query string parameter) contains nested key-
value pairs. It will automatically split such values to perform analysis at
the finest granularity possible.

Entropy. Dumviri uses entropy to select identifiers. One problem is that there
are various ways to transmit tracking identifiers. For instance, a tracking
identifier may be transmitted as a single value in a single request field
(e.g., uid=aX13nL) or it can be transmitted as query string parameters in
several requests and only assembled on the server side. Dumviri developers two
methods to entropy calculation: 1) per field: this method calculates the total
entropy of a single request field by estimating the total number of
combinations that the field can hold based on the character set size and the
field length. 2) per server: this method calculates the total entropy of query
string parameters and cookies of all requests sent to a single server to
determine if the server can potentially assemble a high-entropy identifier.

Dumviri uses two configurable thresholds for the entropy calculation and
returns all fields that identified by both methods. The threshold setting
poses a trade-off between performance and precision. A low threshold produces
a larger set of identifiers for analysis with more irrelevant ones, and thus
Dumviri runs slower, while a high threshold produces a smaller set of more
relevant identifiers for faster analysis time but it is at the risk of false
negatives where mixed trackers are never detected. We picked our thresholds
based on the identifiers observed in current trackers. If we, or the privacy
developers after deployment, observe that tracking servers try to circumvent
Dumviri, we can adopt a lower threshold.

| Both-N | Both-P | Dumviri-only | Filter list-only
---|---|---|---|---
Per field | 21.85 | 38.51 | 32.76 | 6.87
Per server | 3.73 | 43.37 | 50.88 | 2.02
Table 7: Compare percentages of requests that Dumviri decides to analyze to
that of identified by filter lists such as EasyList and EasyPrivacy.

To justify the thresholds we picked, we show the percentage of requests that
Dumviri flags for analysis and compare that to the trackers flagged by the
human-constructed filter lists. Our goal is to demonstrate that Dumviri’s
entropy can identify a close superset of trackers and can remove non-trackers
for effectiveness. We collected all requests, for a total of 4658, from the
top 100 pages of the Alexa Top List and show the results in Table 7. In this
table, Both-N and Both-P refer to requests that are flagged by Dumviri and
filter lists to be negative or positive respectively. Dumviri-only and filter
list-only refer to requests that each flags as positive. We see that there is
a minimum percent of requests in the filter list-only category, which means
that the thresholds can produce a close superset of all trackers.

##  6 Identify Mixed Trackers in the Wild

In this section, we evaluate Dumviri’s unique ability to identify mixed
trackers and show the impact of the mixed tracker we identified. Our analysis
shows that Dumviri can identify 26 mixed trackers with 83 occurrences in our
dataset.

Dataset. To demonstrate the prevalence of mixed trackers, we run Dumviri on
requests the same dataset in q2 evaluation of requests observed on 15K pages
and filter out all requests that filter lists flag as trackers. To ensure the
legitimacy of our findings, we filter out all requests that are not observable
when filter lists are enabled: they are either probabilistic or dependent on
filter lists-flagged requests. We end up with a dataset of 17,087 unique
requests and parameter pairs in our dataset. We invoke Dumviri on this dataset
to analyze all tracking parameters which returned 7,776 tracking parameters
and 9,311 non-tracking parameters. Due to the lack of reliable ground truth
for mixed trackers (§3.2), we perform a manual analysis on the result.

Parameter | Parameter Impact | Name
---|---|---
- | Breakage | Breakage
Functional | No impact | Stale parameters
Tracking | No impact | Tracking parameter
Undecided | Undecided
Table 8: Labels used in mixed tracker evaluation.

Manual Analysis. Our goal is to assign each request-parameter pair to one of
the following labels: 1) Breakage: if parameter removal causes page breakage.
2) Stale parameter: if the parameter serves functionality but removing it has
no impact on the page. 3) Tracking parameter: if the parameter serves
tracking, and parameter removal does not break the page. 4) Undecided: if we
cannot decide the purpose of the request or the parameter. We summarize the
label definitions in Table 8.

Methodology. We detail the procedure to assign labels manually. While we
follow the same method to assign request purpose as in §4.2.1, we detail how
we assign parameter purpose and impact. Determining parameter purposes is
difficult as they are server set and server used; it is possible that a server
names a parameter in a common way and uses it for a different purpose. We
determine parameter purpose based on the following. 1) documentation. We first
try to search for any documentation on the parameter if we can find the
documentation on the request. 2) parameter name and value. If the parameter
has a common name, we look at how other parties use the same parameter if the
parameter value format also matches (e.g., date, hash value). While this
process is relatively easy for some parameters such as ‘hash’ for hash value
and ‘v’ for version number, it is difficult to draw any conclusion for
parameters like ‘token’ or ‘sid’. We take a conservative approach and assign a
label of undecided if we have no concrete evidence to assign the parameter
into other labels. To assign parameter impact, we block the parameter using
the ‘remoeveparam’ rule in uBlock Origin and use the same method described in
§4.2.1 to evaluate if the web page is broken.

Positive Case Analysis. We summarize the analysis result in Table 9. Out of
the positive cases that we analyzed, 50% parameters are for tracking purposes.
Specifically, one of the tracking parameters we found is the ‘sessionId’
parameter in Twitter’s profile fetching and Tweet fetching API. While we can
never confirm whether ‘sessionId’ from this specific API is used for tracking,
we, through our manual analysis, found that this parameter is Twitter-set, has
high entropy and removing it does not break Twitter’s API on the page. All the
other parameters in the API are not deemed for tracking purpose. We detail
additional findings at the end of this section.

We found 10% cases are actually page breakage. For instance, kyoto-
ryokan.co.jp sends a request to fetch TripAdvisor’s certificate of excellence
logo with a parameter ‘wtype=certificateOfExcellence’. While blocking this
parameter changes the logo’s appearance slightly, we deemed this as not
tracking due to the nature of the logo. We found 12.5% to be parameters that
have no immediate impact on the page but define an action that is executed
conditionally at a later time. For instance, YouTube automatically tries to
sign in to the user’s Google account, and the request
‘https://accounts.google.com/ServiceLogin’ contains a continuation parameter
that details the subsequent action after successful sign-in
(continue=‘https://youtube.com/signin’). Blocking this parameter has no impact
on the immediate page but will break page functionality when it is triggered
by user interaction. Blocking such parameter causes a web page breakage that
is only discoverable after user interactions, which Dumviri currently cannot
model. One way to solve this is to enhance Dumviri’s features to capture the
state of the page after certain user interactions (e.g., how the page looks
after user login), which we leave as future work.

We also found 15% of the parameters to be stale parameters. Blocking such
parameters have no impact on the server response. One may argue that such
parameter is time-sensitive, meaning that they may be functionality-dependent
at a different timestamp (e.g., in the future); Dumviri is able to capture its
impact on the page whenever it becomes functional.

| # | % | Estimated
---|---|---|---
Tracking parameter | 26 | 65% | 5,054
Breakage | 5 | 12.5% | 972
Stale parameter | 4 | 10% | 778
Undecided | 5 | 12.5% | 972
Table 9: Manual analysis result of positive cases of mixed trackers.

Negative Case Analysis. We summarize our analysis result in Tab 10. We found
5% cases are tracking parameters that Dumviri mistakenly predicted as
functional parameters. 45% cases break the page when the parameter is removed.
35% cases are stale parameters and 17.5% cases are undecided.

Using these numbers, we calculate Dumviri’s accuracy on mixed tracker
identification to be 79.09% ((5054+778+4190+3492)/17,087). Since stale
parameters can be either blocked or allowed and have no impact on the page, we
count either label as correct.

| # | % | Estimated
---|---|---|---
Tracking parameter | 2 | 5% | 466
Breakage | 18 | 45% | 4,190
Stale parameter | 15 | 37.5% | 3,492
Undecided | 6 | 15% | 1,397
Table 10: Manual analysis result of negative cases of mixed trackers.

Dumviri Findings. While we believe that there are more trackers to be
discovered in our dataset, we measure the occurrence of the confirmed trackers
since our dataset consists of unique trackers. We observed a total of 83
occurrences in our evaluation dataset. Again, this is a lower bound estimate
as we match by URL only. This shows that our identified trackers are not rare
instances. We detail additional findings of mixed trackers.

One case happens on yandex.com, it uses yandex.com/portal/set/any to store
user session data such as configuration data which we believe is legitimate
site functionality. However, Dumviri determined one query string parameter
szm=1:800x600:780x476 to be tracking as string containing resolutions is
commonly used for device fingerprinting. Also, removing such parameter has no
impact on the user experience across session in our manual analysis.

Another case happens on aaalife.com, it loads
siteintercept.qualtrics.com/.../FeedbackButtonM
odule.js?Q_CLIENTVERSION=1.106.0&Q_CLIENTTYPE=
web&Q_BRANDID=aaalife which renders and supports the feedback button which we
deemed as legitimate page functionality. Taking a closer look at its query
string parameters, Dumviri deemed Q_BRANDID as a tracking parameter as it
contain the source domain of the page. This is additional information that the
server tracks and removing it has no impact on submitting feedback on the
page.

##  7 Discussion / Limitations

We perform a cause analysis to see why Dumviri mispredicts and, more
importantly, how we can address those to improve Dumviri’s prediction. We
explore two techniques: feature re-extraction and continued model training. We
show that non-determinism in web page causes Dumviri to fail but these these
two techniques can improve Dumviri’s overall accuracy.

We specifically looked at the cases where Dumviri mispredicts in §4.2.1.
Specifically, we looked at the 4 cases of breakages where Dumviri labels as
tracker in False Positives Analysis and 28 cases of trackers where Dumviri
labels as non-tracker in False Negative Analysis, for a total of 32 cases. We
reran Dumviri on those cases five times to see if rerunning feature extraction
fixes the labels prediction as. We see that 8/32 cases (2 false positives and
6 false negatives) flipped from the incorrect label to the correct label. Our
analysis shows the following source of errors:  1. Silent page load failure
during differential analysis. When this happens, Dumviri will extract a more
significant difference, which causes the breakage detector to predict breakage
and leads to false negatives. To solve this, Dumviri needs to programmatically
know when a page load fails. Although there are simple cases when the network
response has a typical status code (e.g. 400-599), the challenge is to detect
loading failures without the status code. Dumviri currently implements a
heuristic on the size of the response object to guess if a page is loaded
correctly. One of our future work is to adopt a more accurate check.  2.
Excessive dynamism adds noise to Dumviri’s features. While we designed Dumviri
to handle page dynamism like a slide show, excessive dynamism, such as
multiple concurrent animations at different playback rates, makes Dumviri’s
dynamism mask invalid. To address this issue, Dumviri’s detectors need to
exclude elements from feature extraction based on element types (e.g.,
excluding video content from appearance features).  Reducing non-determinism
is a common research problem that others are actively working on. This means
that, Dumviri benefits from the advances in the area.

For the remaining 24 cases, we opted for continuing training where we
retrained the model with the manually specified correct labels. Dumviri was
able to produce the proper labels afterwards. As a sanity check to ensure this
change does not drastically decrease the overall accuracy, we rerun the
evaluation and observed no significant change in the overall accuracy. We
believe that with feature extraction rerun (on predicted data) and continue
training (on more data samples), Dumviri can further improve its accuracy.

##  8 Related Work

We compare Dumviri to related work in Table 11 and we begin by discussing
proposals based on their feature source.

| Feature source |  |
---|---|---|---
| Requests | Response JavaScript | Rendering graph | Breakage analysis | Mixed tracker
AdBlock Plus [37] | Manual | Manual | No
UbO [19] | Manual | Manual | Manual
Request-based [8, 20, 35, 17, 48, 16] | $\checkmark$ |  |  | Manual | No
Code-features [22, 29, 27, 46] |  | $\checkmark$ |  | Manual | No
Graph-based [24, 38, 47, 25], |  |  | $\checkmark$ | Manual | No
Blocked-or-broken [41] AutoFR [28] | NA | Auto | No
SugarCoat [40], ASTracker [10] | NA | No | Semi-auto
Dumviri | $\checkmark$ | $\checkmark$ | $\checkmark$ | Auto | Auto
Table 11: Compare Dumviri to related work. A check mark means a specific work
uses a specific set of features, NA means the work does not identify trackers.
Semi-auto means human developers’ effort is required.

Based on Network-level Information. Several works extract features from
network requests and responses, hypothesizing that trackers exhibit
distinctive characteristics at the network level. Bhagavatula et al.[8] focus
on request query parameters, while Gugelmann et al.[20] construct features
from both requests (e.g., partiness) and responses (e.g., response size).
Additionally, the detection of cookie syncing has employed request and
response features [35].

A core insight used in many tracker identification works, including Dumviri
for its mixed tracker detection, revolves around the identification of high-
entropy identifiers in requests and responses. Yu et al.[48] emphasize the
crucial role of such identifiers in tracking. The flow of unique client values
enables a server to distinguish clients[17, 48] or engage in cookie syncing
[16]. Privacy Badger [17] employs a heuristic: a third-party domain is
classified as a tracking domain if it sets unique identifier-like strings on
over three other domains.

Based on JavaScript Features. Another set of works extracts features based on
the static or dynamic features of JavaScript code. Intuitively, trackers
exhibit different behavior (e.g., API invocations, DOM accesses) compared to
functional JavaScript. Ikram et al.[22] leverage syntactic and semantic
JavaScript code features to detect trackers. Static analysis is vulnerable to
obfuscation[29], making this approach less effective. On the other hand,
Kaizer et al.[27] and DMTrackerDetector[46] rely on dynamic features, such as
accessed properties, to build classifiers.

Based on Graph Representation of Page Rendering. Furthermore, some works
capture contextual information of requests, treating the web page rendering
process as a graph and extracting features from nodes (e.g., DOM elements,
network resources) and edges (e.g., network activity, DOM access)[24, 38, 47,
25]. However, to the best of our knowledge, ML-based classifiers have not been
widely adopted as online tools for tracker blockers due to performance and
robustness concerns. State-of-the-art ML-based classifiers can add over 500ms
of overhead[47], significantly more than the 0.023ms overhead of filter lists
[5]. We designed Dumviri to be used as an offline tool.

None of the existing works considers the impact of tracker removal on the
resulting page, making them 1) prone to break pages and 2) unsuitable for
detecting mixed trackers. Dumviri addresses this gap by implementing a
breakage detector that not only enables the prediction of mixed trackers but
also enhances the overall system accuracy when predicting non-mixed trackers.

Mixed Tracker and Defenses. Amjad et al. [3, 4] conducted the first analysis
of the prevalence of mixed trackers, revealing that 15% of scripts are mixed
trackers, causing breakage when blocked at the file level.

SugarCoat [40] takes on the task of sanitizing mixed code trackers by
generating a resource replacement with sensitive APIs patched. SugarCoat still
requires privacy developers to 1) identify possible mixed trackers in the
first place and 2) perform post-processing inspection to ensure the absence of
page breakage as targeted privacy-sensitive APIs may be used for web page
functionality. In contrast, Dumviri automatically detects mixed trackers and
produce a patched request that does not break the page with confidence
(through its breakage detector).

ASTracker [10] operates on the observation that much third-party tracking code
is embedded into first-party contexts, resulting in shared code structures
across many mixed trackers. By leveraging patterns identified by human
developers in tracking code, ASTracker utilizes the abstract syntax tree to
identify similar code patterns in other presumably mixed scripts. We argue
that the assumption of a shared tracking component may be limiting since
roughly 90% of first parties develop their own trackers [14], suggesting they
are likely to create unique mixed trackers distinct from others.

Dumviri, although only stops mixed trackers with tracking identifiers, stands
out as the first work, to the best of our knowledge, that is capable of
automatically predicting and addressing mixed trackers without developer
effort.

One reason for web page breakage is the use of content blockers, relying on
human-generated filter lists that demand constant maintenance to remain
effective [44]. Breakage can occur due to developer errors in maintenance or
when the page undergoes changes. Additionally, intentional breakage may be
introduced by web pages mixing trackers and functionality to evade detection,
as found by Amjad et al. [3].

Breakage Study. Very few works have systematically studied the symptoms of
breakage. Mathur et al. [31] conducted a survey on content blocker usage,
aiming to evaluate why users adopt or avoid content blockers and how
frequently they encounter blocker-induced breakage along with its symptoms.
While the responses from users say that content blocked rarely break sites,
it’s noteworthy that users might lack the technical background to identify
certain types of breakage.

Nisenoff et al. [33] systematically studied breakages caused by content
blockers, analyzing user reports of broken pages and providing statistics on
broken symptoms (e.g., missing content, non-responsiveness). We designed our
breakage detector by encompassing their comprehensive set of symptoms.

Automatic Breakage Analysis. Breakage analysis in current tracker blocker
implementations is predominantly manual. Content blockers like Adblock Plus
and uBlock Origin depend on user reports of page breakages [36], requiring
privacy developers to verify the reported issues. This process can be time-
consuming and error-prone.

AutoFR [28] uses changes in the number of images and text before and after a
page change to determine breakage. However, its heuristic is oversimplified
and incomplete as it lacks modeling of other ways that a user can perceive
page breakages (e.g., non-responsiveness).

Blocked or Broken [41] attempts to automatically predict broken pages based on
features extracted from PageGraphs [9]. While its goal is similar to Dumviri’s
breakage detector (one of Dumviri’s component), the approaches differ in the
following ways: 1) Blocked or Broken collects broken page samples through
commit messages, which is less relevant (as the commit may be outdated) and
standardized (not all commit messages follow the conventions) compared to
Dumviri’s exception rules and 2) Dumviri uses different features for
prediction. Dumviri’s breakage detector achieves a higher test accuracy (shown
in Table 2 ) than Blocked or Broken’s model. Another set of work aim to self-
recovery from breakages based on heuristics [11, 32].

##  9 Conclusion

In this paper, we introduced Dumviri, a framework designed to identify both
non-mixed and mixed trackers. Dumviri relies on two detectors that
independently predict page breakage and tracking behavior. The combination of
the two detectors allows Dumviri to detect when functional resources are
mispredicted by the tracking detector, thus improving overall accuracy.
Notably, we crafted differential features for our detectors that are analysis
granularity agnostic. They can capture differences due to the blocking of
requests or at request field granularity, allowing Dumviri to analyze tracking
request fields — a type of mixed tracker.

Our results demonstrate Dumviri’s effectiveness, achieving a 97.44% accuracy
in detecting non-mixed trackers when compared to the labels provided by filter
lists across 15K pages from the Alexa Top List. Moreover, we uncovered 22
newly unreported trackers and identified 2 cases of page breakage due to
filter lists. In mixed tracker evaluation, Dumviri achieves an accuracy of
79.09%. Our manual analysis led to the discovery of 26 instances of mixed
trackers. By promptly reporting our newly identified trackers and breakages to
developers, we anticipate that Dumviri will prove valuable to the community.

## References

  * [1] Adobe.  Adobe analytics 2.0 api reference.  https://developer.adobe.com/analytics-apis/docs/2.0/apis/.  (Accessed on 02/07/2024).
  * [2] Mshabab Alrizah, Sencun Zhu, Xinyu Xing, and Gang Wang.  Errors, misunderstandings, and attacks: Analyzing the crowdsourcing process of ad-blocking systems.  In Proceedings of the Internet Measurement Conference, IMC ’19, page 230–244, New York, NY, USA, 2019. Association for Computing Machinery.
  * [3] Abdul Haddi Amjad, Danial Saleem, Muhammad Ali Gulzar, Zubair Shafiq, and Fareed Zaffar.  Trackersift: Untangling mixed tracking and functional web resources.  In Proceedings of the 21st ACM Internet Measurement Conference, pages 569–576, 2021.
  * [4] Abdul Haddi Amjad, Zubair Shafiq, and Muhammad Ali Gulzar.  Blocking javascript without breaking the web: An empirical investigation.  arXiv preprint arXiv:2302.01182, 2023.
  * [5] Vastel Antoine, Snyder Peter, and Ben Livshits.  The mounting cost of stale ad blocking rules, 2018.
  * [6] Backlinko.  Ad blockers usage and demographic statistics in 2023.  https://backlinko.com/ad-blockers-users.  (Accessed on 02/07/2024).
  * [7] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [8] Sruti Bhagavatula, Christopher Dunn, Chris Kanich, Minaxi Gupta, and Brian Ziebart.  Leveraging machine learning to improve unwanted resource filtering.  In Proceedings of the 2014 Workshop on Artificial Intelligent and Security Workshop, pages 95–102, 2014.
  * [9] Brave.  Pagegraph · brave/brave-browser wiki · github.  https://github.com/brave/brave-browser/wiki/PageGraph.  (Accessed on 02/07/2024).
  * [10] Ismael Castell-Uroz, Kensuke Fukuda, and Pere Barlet-Ros.  Astrack: Automatic detection and removal of web tracking code with minimal functionality loss.  arXiv preprint arXiv:2301.10895, 2023.
  * [11] Quan Chen, Peter Snyder, Ben Livshits, and Alexandros Kapravelos.  Detecting filter list evasion with event-loop-turn granularity javascript signatures.  In 2021 IEEE Symposium on Security and Privacy (SP), pages 1715–1729. IEEE, 2021.
  * [12] Distributed (Deep) Machine Learning Community.  Xgboost documentation — xgboost 2.0.3 documentation.  https://xgboost.readthedocs.io/en/stable/.  (Accessed on 02/04/2024).
  * [13] Bright Data.  Bright data - the world’s #1 web data platform.  https://brightdata.com/.  (Accessed on 02/07/2024).
  * [14] Nurullah Demir, Daniel Theis, Tobias Urban, and Norbert Pohlmann.  Towards understanding first-party cookie tracking in the field, 2022.
  * [15] EasyList.  Easylist - overview.  https://easylist.to/.  (Accessed on 02/07/2024).
  * [16] Steven Englehardt and Arvind Narayanan.  Online tracking: A 1-million-site measurement and analysis.  In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 1388–1401, 2016.
  * [17] Electronic Frontier Foundation.  Privacy badger.  https://privacybadger.org/.  (Accessed on 02/07/2024).
  * [18] Google.  Pricing | compute engine: Virtual machines (vms) | google cloud.  https://cloud.google.com/compute/all-pricing.  (Accessed on 02/07/2024).
  * [19] gorhill.  Resources library · gorhill/ublock wiki · github.  https://github.com/gorhill/uBlock/wiki/Resources-Library.  (Accessed on 02/07/2024).
  * [20] David Gugelmann, Markus Happe, Bernhard Ager, and Vincent Lenders.  An automated approach for complementing ad blockers’ blacklists.  Proc. Priv. Enhancing Technol., 2015(2):282–298, 2015.
  * [21] Troy Hunt.  Why do i keep seeing requests for “gen204” in my web logs? because of foreigners!  https://www.troyhunt.com/why-do-i-keep-seeing-requests-for/.  (Accessed on 02/07/2024).
  * [22] Muhammad Ikram, Hassan Jameel Asghar, Mohamed Ali Kâafar, Anirban Mahanti, and Balachandar Krishnamurthy.  Towards seamless tracking-free web: Improved detection of trackers via one-class learning.  Proceedings on Privacy Enhancing Technologies, 2017(1):79–99, 2017.
  * [23] Umar Iqbal, Steven Englehardt, and Zubair Shafiq.  Fingerprinting the fingerprinters: Learning to detect browser fingerprinting behaviors.  In 2021 IEEE Symposium on Security and Privacy (SP), pages 1143–1161. IEEE, 2021.
  * [24] Umar Iqbal, Peter Snyder, Shitong Zhu, Benjamin Livshits, Zhiyun Qian, and Zubair Shafiq.  Adgraph: A graph-based approach to ad and tracker blocking.  In 2020 IEEE Symposium on Security and Privacy (SP), pages 763–776. IEEE, 2020.
  * [25] Umar Iqbal, Charlie Wolfe, Charles Nguyen, Steven Englehardt, and Zubair Shafiq.  Khaleesi: Breaker of advertising and tracking request chains.  In 31st USENIX Security Symposium (USENIX Security 22), pages 2911–2928, 2022.
  * [26] Jordan Jueckstock, Peter Snyder, Shaown Sarker, Alexandros Kapravelos, and Benjamin Livshits.  Measuring the privacy vs. compatibility trade-off in preventing third-party stateful tracking.  In Proceedings of the ACM Web Conference 2022, WWW ’22, page 710–720, New York, NY, USA, 2022. Association for Computing Machinery.
  * [27] Andrew J Kaizer and Minaxi Gupta.  Towards automatic identification of javascript-oriented machine-based tracking.  In Proceedings of the 2016 ACM on International Workshop on Security And Privacy Analytics, pages 33–40, 2016.
  * [28] Hieu Le, Salma Elmalaki, Athina Markopoulou, and Zubair Shafiq.  Autofr: Automated filter rule generation for adblocking.  arXiv preprint arXiv:2202.12872, 2022.
  * [29] Hoan Le, Federico Fallace, and Pere Barlet-Ros.  Towards accurate detection of obfuscated web tracking.  In 2017 IEEE International Workshop on Measurement and Networking (M&N), pages 1–6. IEEE, 2017.
  * [30] Analytics Market.  How google analytics collects data - analyticsmarket.  https://www.analyticsmarket.com/blog/how-google-analytics-collects-data/.  (Accessed on 02/07/2024).
  * [31] Arunesh Mathur, Jessica Vitak, Arvind Narayanan, and Marshini Chetty.  Characterizing the use of $\\{$Browser-Based$\\}$ blocking extensions to prevent online tracking.  In Fourteenth symposium on usable privacy and security (SOUPS 2018), pages 103–116, 2018.
  * [32] Miguel Müller.  Self-healing javascript errors caused by the browser extension privacy badger, 2021.
  * [33] Alexandra Nisenoff, Arthur Borem, Madison Pickering, Grant Nakanishi, Maya Thumpasery, and Blase Ur.  Defining “broken”: User experiences and remediation tactics when ad-blocking or tracking-protection tools break a website’s user experience.  In Proceedings of the 32nd USENIX Security Symposium, 2023.
  * [34] U.S. Bureau of Labor Statistics.  Table b-3. average hourly and weekly earnings of all employees on private nonfarm payrolls by industry sector, seasonally adjusted - 2023 m10 results.  https://www.bls.gov/news.release/empsit.t19.htm.  (Accessed on 02/07/2024).
  * [35] Panagiotis Papadopoulos, Nicolas Kourtellis, and Evangelos Markatos.  Cookie synchronization: Everything you always wanted to know but were afraid to ask.  In The World Wide Web Conference, pages 1432–1442, 2019.
  * [36] Adblock Plus.  Adblock plus - open adblock plus forums.  https://forum.adblockplus.org/viewforum.php?f=6&sid=687bc0da62aa1dcc59b1393439808def.  (Accessed on 02/07/2024).
  * [37] Adblock Plus.  Adblock plus | the world’s #1 free ad blocker.  https://adblockplus.org/.  (Accessed on 02/08/2024).
  * [38] Sandra Siby, Umar Iqbal, Steven Englehardt, Zubair Shafiq, and Carmela Troncoso.  Webgraph: Capturing advertising and tracking information flows for robust blocking.  arXiv preprint arXiv:2107.11309, 2021.
  * [39] Brad Slayter.  Tracker radar wiki.  https://slayterdev.github.io/tracker-radar-wiki/.  (Accessed on 02/07/2024).
  * [40] Michael Smith, Pete Snyder, Benjamin Livshits, and Deian Stefan.  Sugarcoat: Programmatically generating privacy-preserving, web-compatible resource replacements for content blocking.  In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2844–2857, 2021.
  * [41] Michael Smith, Peter Snyder, Moritz Haller, Benjamin Livshits, Deian Stefan, and Hamed Haddadi.  Blocked or broken? automatically detecting when privacy interventions break websites.  arXiv preprint arXiv:2203.03528, 2022.
  * [42] Peter Snyder, Soroush Karami, Arthur Edelstein, Benjamin Livshits, and Hamed Haddadi.  Pool-Party: Exploiting browser resource pools for web tracking.  In 32nd USENIX Security Symposium (USENIX Security 23), pages 7091–7105, 2023.
  * [43] Peter Snyder, Cynthia Taylor, and Chris Kanich.  Most websites don’t need to vibrate: A cost-benefit approach to improving browser security.  In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 179–194, 2017.
  * [44] Peter Snyder, Antoine Vastel, and Benjamin Livshits.  Who filters the filters: Understanding the growth, usefulness and efficiency of crowdsourced ad blocking.  arXiv preprint arXiv:1810.09160, 2018.
  * [45] Twitter.  Tracking tags | docs | twitter developer platform.  https://developer.twitter.com/en/docs/twitter-ads-api/campaign-management/api-reference/tracking-tags.  (Accessed on 02/07/2024).
  * [46] Qianru Wu, Qixu Liu, Yuqing Zhang, Peng Liu, and Guanxing Wen.  A machine learning approach for detecting third-party trackers on the web.  In European Symposium on Research in Computer Security, pages 238–258. Springer, 2016.
  * [47] Zhiju Yang, Weiping Pei, Monchu Chen, and Chuan Yue.  Wtagraph: Web tracking and advertising detection using graph neural networks.  In IEEE Symposium on Security and Privacy, 2022.
  * [48] Zhonghao Yu, Sam Macbeth, Konark Modi, and Josep M. Pujol.  Tracking the trackers.  In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 121–132, Republic and Canton of Geneva, CHE, 2016. International World Wide Web Conferences Steering Committee.

##  Appendix A Breakage Examples

We show example cases of Dumviri’s false positive breaking sites in Fig 5 and
Fig 6, example cases of filter lists-caused page breakage in Fig 7 and Fig 8
and example cases of blocking the entire mixed tracker in Fig 9.

Figure 5: An example of Dumviri’s false postivie breaking site. On tradus.com,
Dumviri mistakenly flags otTCF.js from cdn.cookielaw.org as a tracking script,
and causes the cookie notice to disappear. The broken page without the cookie
notice is on the left while vanilla page with the cookie notice is on the
right.

Figure 6: An example of Dumviri’s false positive breaking site. On haco.nu,
Dumviri mistakenly flags bundle.42ecd298.js as a tracking script breaking the
carousel. The broken page without the carousel is shown on the left while the
functioning carousel is shown on the right.

Figure 7: An example of Dumviri discovered EasyPrivacy-caused site breakage.
The broken page with missing cookie notice is shown on the left while the
functional web page appearance is shown on the right.

Figure 8: An example of Dumviri discovered EasyPrivacy-caused site breakage.
This example happens on ero-advertising.com, where the script
www.eroadvertising.com/js/controllers.js is blocked by EasyPrivacy. This
script loads the main body content on the site as shown on the right side of
the figure. The broken page is shown on the left side where it shows the page
footer right after the header.

Figure 9: An example of Dumviri discovered mixed tracker. We found the
‘sessionId’ parameter in the requst
https://platform.twitter.com/embed/Tweet.html to be tracking. Removing this
request breaks the page. The functional page with the tweet display is shown
on the left, while the broken page with the illegible Tweet is shown on the
right.