# LastResort at SemEval-2024 Task 3: Exploring Multimodal Emotion Cause Pair
Extraction as Sequence Labelling Task

Suyash Vardhan Mathur
IIIT Hyderabad
<EMAIL_ADDRESS>
Akshett Rai Jindal11footnotemark: 1
IIIT Hyderabad
<EMAIL_ADDRESS>
Hardik Mittal
IIIT Hyderabad
<EMAIL_ADDRESS>
Manish Shrivastava
IIIT Hyderabad
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Conversation is the most natural form of human communication, where each
utterance can range over a variety of possible emotions. While significant
work has been done towards the detection of emotions in text, relatively
little work has been done towards finding the cause of the said emotions,
especially in multimodal settings. SemEval 2024 introduces the task of
Multimodal Emotion Cause Analysis in Conversations, which aims to extract
emotions reflected in individual utterances in a conversation involving
multiple modalities (textual, audio, and visual modalities) along with the
corresponding utterances that were the cause for the emotion. In this paper,
we propose models that tackle this task as an utterance labeling and a
sequence labeling problem and perform a comparative study of these models,
involving baselines using different encoders, using BiLSTM for adding
contextual information of the conversation, and finally adding a CRF layer to
try to model the inter-dependencies between adjacent utterances more
effectively. In the official leaderboard for the task, our architecture was
ranked 8th, achieving an F1-score of 0.1759 on the leaderboard. We also
release our code here111github.com/akshettrj/semeval2024_task03.

LastResort at SemEval-2024 Task 3: Exploring Multimodal Emotion Cause Pair
Extraction as Sequence Labelling Task

Suyash Vardhan Mathur††thanks: Equal contribution. IIIT Hyderabad
<EMAIL_ADDRESS>Akshett Rai Jindal11footnotemark: 1 IIIT
Hyderabad<EMAIL_ADDRESS>

Hardik Mittal IIIT Hyderabad<EMAIL_ADDRESS>Manish
Shrivastava IIIT Hyderabad<EMAIL_ADDRESS>

##  1 Introduction

Emotion Analysis is one of the fundamental and earliest sub-fields of NLP that
focus on identifying and categorizing emotions that are expressed in text.
Earlier, research in this domain focused on Emotion Detection in news articles
and headlines (Lei et al., 2014; Abdul-Mageed and Ungar, 2017). However, later
Emotion Recognition in Conversation gained popularity due to the widespread
availability of public conversation data (Gupta et al., 2017). Recently, the
task of emotion cause analysis has gained traction, which tries to identify
the causes behind certain emotions (Xia and Ding, 2019a). This has widespread
application such as building chatbots that can identify the emotions of the
user and even identify the cause behind the emotions to perform certain
actions (Pamungkas, 2019). For instance, companies can identify causes behind
dissatisfaction in customer interactions and take appropriate measures (Yun
and Park, 2022), AI-driven therapeutic insights can be gained using such
models (D’Alfonso, 2020), social media content moderation can be better done
(Sawhney et al., 2021), work management and team management by companies can
be improved (Benke et al., 2020).

In the task Wang et al., 2024, we tackle the problem of Multimodal Emotion
Cause Pair Extraction, where given a set of utterances in a conversation, we
must identify the following:

1\. Emotion of every utterance (if any). These emotions can be one of Ekman’s
six basic emotions (Ekman et al., 1999).

2\. Cause of these emotions, which is considered as the utterance that
explicitly expresses an event or argument that is highly linked to the
corresponding emotion.

Our proposed system tackles the task in a 3-step fashion – (a) First, we train
a model to identify the emotions that are expressed in individual utterances
in a conversation. (b) Next, we train a model to identify whether an utterance
can be a cause of an emotion expressed in another/same utterance (candidate
causes). (c) Finally, we train a model to pair emotion-utterances with their
causes among the possible candidate causes. For both the (a) and (b) models we
experiment with 3 basic architectures – (i) a simple Neural Network to
determine the class of emotion (N-class classifier) and another Neural Network
to identify whether the utterance is a candidate cause or not (binary
classification). (ii) A BiLSTM (Sak et al., 2014) architecture that accounts
for the surrounding context of the conversation while doing the N-class and
binary-classification. (iii) A BiLSTM CRF (Lafferty et al., 2001) architecture
which accounts for the surrounding emotions as well while doing the N-class
classification. We also experiment with different encoders for the three
modalities.

##  2 Background

###  2.1 Dataset

The dataset used for this problem is Emotion-Cause-in-Friends prepared by Wang
et al., 2023 specifically for this task. It has been prepared using
conversations from the popular 1994 sitcom Friends as the source. This dataset
contains 1,344 conversations made up of a total of 13,509 utterances, each
conversation containing an average of 10 utterances. For each utterance, the
dataset has an annotated transcript (covering text modality) and the
corresponding video clip (covering visual and auditory modalities) from the
show.

Each utterance is annotated with the emotion depicted by it, which is one of:
anger, disgust, fear, joy, neutral, sadness and surprise. The dataset is
highly skewed in terms of the frequency of different emotions in the dataset
(see Figure 2). Further, the emotion-causes pairs for all the non-neutral
utterances are provided in the dataset in a separate list.

Figure 1: Model Architecture Figure 2: Emotion frequency in the dataset

The task MC-ECPE expects the model to take a list of such conversations and
predict the emotion and emotion-cause pairs labels.

###  2.2 Related Work

A lot of work has been done in the field of emotion analysis in textual
settings. Soon, work began on extracting not only the emotion but also the
cause of that extracted emotion. People employed mainly two approaches for
emotion cause analysis - 1. Extracting the potential causes given an emotion
(Lee et al., 2010; Chen et al., 2010; Gui et al., 2016b) and 2. Extracting the
emotion-cause pairs jointly (Xia and Ding, 2019b; Ding et al., 2020; Wei et
al., 2020).

Poria et al., 2020 was the first to introduce the task of extracting emotion-
cause in conversations but their focus was also only on the textual dialogues.
However, in our natural way of conversation, we rely on things like facial
expressions, voice intonations for determining the emotion of the speaker. We
also rely on auditory and visual scenes to determine the cause of the
speaker’s emotions. Hence, it is clear that Emotion-Cause Pair Extraction
(ECPE) is a multimodal task requiring at least three modalities: text, audio
and video. Busso et al., 2008; McKeown et al., 2012; Li et al., 2022a and
Poria et al., 2019 worked in the field of multimodal emotion analysis in
conversations but they did not consider the emotion causes.

The task of MC-ECPE was first worked on by Wang et al., 2021.

##  3 System Overview

###  3.1 Baseline I: Utterance labeling

Our baseline model treats the problem as a simple utterance labeling task. We
use pre-trained text, audio, and image encoders to encode the individual
modalities and use these to train three models that can identify the emotions
in the utterances, the candidate cause utterances, and finally identify valid
emotion and cause utterance(s) pairs.

  * •

Text Encoding: For encoding the transcription of each utterance, we use pre-
trained BERT (Devlin et al., 2018) embeddings as the baseline embeddings.
Additionally, we finetune DeBERTa-Base (He et al., 2020) on the training data
for our experiments. DeBERTa makes use of a disentangled attention mechanism
and an enhanced masked encoder to improve upon BERT’s performance in a variety
of tasks. Finally, we also tried RoBERTa-Large and (Liu et al., 2019) pre-
trained EmotionRoBERTa-Base222https://huggingface.co/SamLowe/roberta-base-
go_emotions which is publicly available RoBERTa-base model finetuned on the Go
Emotions dataset (Demszky et al., 2020). For every text encoder, we perform
mean-pooling of the word embeddings to get the textual representation of the
utterance.

  * •

Video Encodings: For encoding the videos, we sampled 16 equally spaced frames
from the video and mean-pooled the embeddings for the 16 frames. For encoding
these 16 images, we used MViTv2-small (Li et al., 2022b) encoder, which
achieves state-of-the-art performance on the Kinetics video detection task
(Kay et al., 2017), which makes it an obvious choice for recognizing
activities happening in the conversations relevant for emotion/cause
detection.

  * •

Audio Encodings: We used WavLM (Chen et al., 2022) for generating audio
embeddings, which is trained on large audio data using masked speech
representation and denoising in pre-training, making it suitable for various
downstream speech tasks. We also try Wav2Vec2-Large (Baevski et al., 2020),
which is trained by masking speech input in latent space and solving a
contrastive task defined over a quantization of the latent representations
which are jointly learned.

The model architecture is a combination of three steps, each of which is
described below:

### Step 1 – Emotion Classification

First, we concatenate the text, audio and video embeddings from the respective
encoders and pass these concatenated embeddings into a dense layer, on which a
Softmax function is applied to get the probability distribution over 7 classes
(6 emotions and one neutral class). Due to a skewed distribution of the
emotion labels in the dataset, we make use of weighted Cross Entropy loss to
train the model, where the weights are taken as inverse of the frequency of
the labels in the training dataset.

### Step 2 – Candidate Cause Identification

For identifying the candidate cause, we similarly pass concatenated embeddings
through a dense layer with a Sigmoid function, which predicts the probability
of whether the utterance is a candidate cause or not. Binary Cross Entropy
Loss is used to train the model.

### Step 3 – Emotion-Cause pairing

For pairing the emotion utterances with the candidate causes, we concatenate
the represenations for the emotion utterance and the cause utterance, with a
distance embedding. This distance embedding is generated by giving positional
embedding to each utterance, sampled from a Normal Distribution. This
representation is passed through a dense layer with a Sigmoid function, which
learns to predict the probability of the emotion-cause utterance pair being a
valid emotion-cause pair or not for the given conversation, trained using
Binary Cross Entropy Loss.

###  3.2 Baseline II: BiLSTM Architecture

The BiLSTM architecture is inspired by the work in Wang et al., 2021. While
the Baseline I architecture treats the emotion and cause classification
independently for each utterance, it is dependent on the surrounding context
of the conversation too. Thus, the BiLSTM architecture models the problem as a
Sequence Labeling task. We use the best encoders in the Baseline I
architecture for generating the embeddings in this architecture.

### Step 1 – Emotion Classification

Similar to the Baseline Model, we concatenate the embeddings of the three
modalities, and pass them to a stacked BiLSTM. On top of the BiLSTM outputs,
we apply a 7-class classifier to obtain the emotion category distribution.
Similar to the Baseline I, weighted cross-entropy loss is used.

### Step 2 – Candidate Cause Identification

For Candidate Cause prediction, similarly, the concatenated embeddings are
passed through a BiLSTM on top of which a binary classifier is applied.

### Step 3 – Emotion-Cause Pairing

The Emotion-Cause pairing model remains the same in this architecture as the
Baseline I model.

In this architecture, BiLSTM provides the advantages of bidirectional and
longer contexts which should help understand the emotions present in
utterances better. This is because in a conversation, it is possible that the
emotions are not just dependent on the current utterance, but on surrounding
multimodal utterances as well.

###  3.3 Baseline III: BiLSTM-CRF Architecture

In the BiLSTM model, each classification decision was conditionally
independent. Linear-chain CRFs are models generally used to model structured
data where one output influences its neighboring outputs as it models the
various transition probabilities, and have been extensively used with BiLSTMs
for sequence labeling (Huang et al., 2015). This could be useful for emotion
predictions because the emotion of one utterance is generally influenced by
the emotions in its previous utterances. For instance, an utterance with
happiness generally tends to be followed by another happiness utterance.

### Step 1 – Emotion Classification

For this architecture, we add a CRF layer on top of the BiLSTM layers, and
make use of the CRF-loss to train the model instead of Cross-Entropy loss as
in the previous architectures. This loss models the transitions between the
labels in the architecture, modelling the task as a more complex sequence
labeling task. Thus, while the BiLSTM layer learns more about the language and
emotions expressed through the language, the CRF layer tries to learn about
the relations between the emotions.

### Step 2 – Candidate Cause Identification

For Candidate Cause prediction, the architecture remains the same as in
Baseline II. This is because the transitions between cause labels (being cause
of an emotion in an utterance or not) does not make intuitive sense, and using
BiLSTMs to capture surrounding context from other utterances is what seems
more appropriate.

### Step 3 – Emotion-Cause Pairing

The Emotion-Cause pairing model remains the same in this architecture as the
Baseline I & II models.

##  4 Experimental Setup

Model Name | Emotion | Candidate Cause | Emotion-Cause | Emotion-Cause | Leaderboard
---|---|---|---|---|---
Detection | Detection | Pairing | Pairing (Eval.)
|  P |  R |  F1 |  P |  R |  F1 |  P |  R |  F1 |  wt. F1 |  Macro F1 |  wt. F1 |  Macro F1
Baseline I |  |  |  |  |  |  |  |  |  |  |  |  |
BERT + WavLM + MViTv2 |  0.61 |  0.52 |  0.55 |  0.71 |  0.71 |  0.71 |  0.93 |  0.87 |  0.89 |  0.26 |  0.20 |  0.182 |  0.165
EmotionRoBERTa + WavLM + MViTv2 |  0.55 |  0.45 |  0.47 |  0.67 |  0.66 |  0.66 |  0.93 |  0.86 |  0.89 |  0.20 |  0.18 |  0.187 |  0.170
DeBERTa (finet.) + WavLM + MViTv2 |  0.44 |  0.36 |  0.38 |  0.60 |  0.60 |  0.60 |  0.92 |  0.85 |  0.87 |  0.10 |  0.10 |  0.094 |  0.094
RoBERTa-L + WavLM + MViTv2 |  0.59 |  0.47 |  0.49 |  0.66 |  0.65 |  0.66 |  0.93 |  0.86 |  0.88 |  0.21 |  0.19 |  0.180 |  0.165
EmotionRoBERTa + Wav2Vec2 + MViTv2 |  0.55 |  0.47 |  0.48 |  0.67 |  0.67 |  0.67 |  0.93 |  0.87 |  0.89 |  0.21 |  0.20 |  0.172 |  0.170
BiLSTM (Baseline II) |  0.55 |  0.51 |  0.52 |  0.67 |  0.67 |  0.67 |  0.93 |  0.86 |  0.89 |  0.22 |  0.21 |  0.184 |  0.179
BiLSTM + CRF(Baseline III) |  0.53 |  0.56 |  0.54 |  0.67 |  0.67 |  0.67 |  0.93 |  0.86 |  0.89 |  0.24 |  0.18 |  0.165 |  0.172
Table 1: Results for baselines on the ECAC dataset

We perform a random shuffle and use a 90-10% split for the train-validation
split. The test set was provided by the authors, but its gold labels have not
been made public.

The experiments involving Baseline II and III use EmotionRoBERTa + WavLM +
MViTv2 configuration. All the experiments involve applying a dropout of 0.3 on
the audio, visual and textual embeddings before they are passed on to the main
architectures. The BiLSTM for emotion detection consists of 4 layers while the
one for candidate cause identification contains 3 layers. The dropout between
the stacked layers of the BiLSTM is kept 0.3 as well. We use AdamW optimizer
for all the three models, and use a linear learning rate scheduler with warmup
for training the models. The Emotion Classification model is trained for 60
epochs, the Candidate Cause Identification model is trained for 40 epochs, and
the Emotion-Cause Pairing Model is trained for 40 epochs as well.

In order to train the Emotion-Cause pairing model, we create positive and
negative pairs during training. However, while the number of positive pairs is
of the order N, the number of negative pairs comes to the order of N2, and
thus we perform a random sampling of the negative pairs to keep the positive
and negative samples in the ratio 1:5. This helps us to maintain balance
between the positive and negative classes.

### Evaluation Metrics

We evaluate the 3 steps separately as well apart from evaluating the
performance for the final Emotion-Cause pairs:
Emotion Identification: We use Weighted Precision, Recall and F1-score for the
distribution between the 7 classes (6 emotions and neutral class).
Candidate Cause Identification: We again use Weighted Precision, Recall and
F1-score for evaluating the prediction between the binary classes –
is_candidate_cause and not_candidate_cause.
Emotion-Cause Pairing: For evaluating this, we generate positive and negative
pairs and use Weighted Precision, Recall and F1-score for evaluating the
classification between the positive and negative classes.
Emotion-Cause Pairs: Weighted F1-score and Macro F1-score are the official
metric used for the final evaluation for the task.

##  5 Results and Analysis

The performance of the three Baselines can be seen in Table 1. During the
Evaluation phase, our best ranked submission of Baseline II had Wt. F1 score
of 0.1836 and Macro F1 score of 0.1759, ranking 8th on the leaderboard.

### Baseline I

Among the encoders in Baseline I, BERT + WavLM + MViTv2 configuration performs
the best on the validation set, including the individual steps as well as the
final emotion-cause pair predictions. However, on the leaderboard,
EmotionRoBERTa + WavLM + MViTv2 gives the best performance, although the
difference in the leaderboard scores is marginal among the encoders. This
observation might indicate that the test data is a bit different in nature
from the training data.

Better performance of EmotionRoBERTa can be attributed to the fact that the
model’s weights have already been finetuned towards emotion-related tasks.
Further, it seems that finetuning DeBERTa on the training data caused it to
overfit, leading to worse performance than vanilla BERT/RoBERTa models.
RoBERTa-L performed slightly worse than BERT and EmotionRoBERTa.

Finally, WavLM being the newer architecture, as expected performed better than
Wav2Vec2. This is because WavLM is more robust than Wav2Vec2 and it is trained
in a combination of supervised and self-supervised learning, making its
performance much better.

### Baseline II

We use the EmotionRoBERTa + WavLM + MViTv2 configuration as encoders for the
Baseline II architecture. Contrary to expectation, the wt. F1 score on the
leaderboard decreased marginally, while the Macro F1 score increased
marginally. This is probably because of the nature of the dataset, where the
average length of a conversation is as little as 10, which causes the context
of the utterance to be rather limited. In such a situation, the additional
context from previous utterances doesn’t prove helpful to the model, and might
even prove to be noise for the model, leading to the results observed.

### Baseline III

In this, we can observe a significant fall in wt. F1 and slight fall in Macro
F1 score from the Baseline I and II architectures. This is in line with the
observation of Baseline II that due to the nature of the dataset, sequence
labeling it is not necessarily the best way to model it. Further, due to small
number of utterances in the conversations, it is likely that the transition
between labels needed for CRF doesn’t get trained that well and leads to
poorer performance.

##  6 Conclusion

In conclusion, we observe that the utterance labeling systems perform as good
as sequence labeling systems for this specific dataset. Further, we also see
that encoders which are trained on other emotion-related tasks tend to perform
better on similar emotion-related tasks.

In future, it is possible to learn joint embeddings over the 3 modalities,
which should provide better representations for each utterance (Girdhar et
al., 2023). Further, it can be experimented to utilize the speaker information
for each utterance while creating utterance representations (Liang et al.,
2023).

## References

  * Abdul-Mageed and Ungar (2017) Muhammad Abdul-Mageed and Lyle Ungar. 2017.  EmoNet: Fine-grained emotion detection with gated recurrent neural networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 718–728, Vancouver, Canada. Association for Computational Linguistics.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33:12449–12460.
  * Benke et al. (2020) Ivo Benke, Michael Thomas Knierim, and Alexander Maedche. 2020.  Chatbot-based emotion management for distributed teams: A participatory design study.  _Proceedings of the ACM on Human-Computer Interaction_ , 4(CSCW2):1–30.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008.  Iemocap: interactive emotional dyadic motion capture database.  _Language Resources and Evaluation_ , 42:335–359.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518.
  * Chen et al. (2010) Ying Chen, Sophia Yat Mei Lee, Shoushan Li, and Chu-Ren Huang. 2010.  Emotion cause detection with linguistic constructions.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_ , pages 179–187, Beijing, China. Coling 2010 Organizing Committee.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020.  GoEmotions: A dataset of fine-grained emotions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4040–4054, Online. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Ding et al. (2020) Zixiang Ding, Rui Xia, and Jianfei Yu. 2020.  ECPE-2D: Emotion-cause pair extraction based on joint two-dimensional representation, interaction and prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3161–3170, Online. Association for Computational Linguistics.
  * D’Alfonso (2020) Simon D’Alfonso. 2020.  Ai in mental health.  _Current Opinion in Psychology_ , 36:112–117.
  * Ekman et al. (1999) Paul Ekman et al. 1999.  Basic emotions.  _Handbook of cognition and emotion_ , 98(45-60):16.
  * Girdhar et al. (2023) Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023.  Imagebind: One embedding space to bind them all.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15180–15190.
  * Gui et al. (2016b) Lin Gui, Ruifeng Xu, Qin Lu, Dongyin Wu, and Yu Zhou. 2016b.  Emotion cause extraction, a challenging task with corpus construction.  In _Social Media Processing_ , pages 98–109, Singapore. Springer Singapore.
  * Gupta et al. (2017) Umang Gupta, Ankush Chatterjee, Radhakrishnan Srikanth, and Puneet Agrawal. 2017.  A sentiment-and-semantics-based approach for emotion detection in textual conversations.  _arXiv preprint arXiv:1707.06996_.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.
  * Kay et al. (2017) Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017.  The kinetics human action video dataset.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
  * Lee et al. (2010) Sophia Yat Mei Lee, Ying Chen, and Chu-Ren Huang. 2010.  A text-driven rule-based system for emotion cause detection.  In _Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text_ , pages 45–53, Los Angeles, CA. Association for Computational Linguistics.
  * Lei et al. (2014) Jingsheng Lei, Yanghui Rao, Qing Li, Xiaojun Quan, and Liu Wenyin. 2014.  Towards building a social emotion detection system for online news.  _Future Generation Computer Systems_ , 37:438–448.  Special Section: Innovative Methods and Algorithms for Advanced Data-Intensive Computing Special Section: Semantics, Intelligent processing and services for big data Special Section: Advances in Data-Intensive Modelling and Simulation Special Section: Hybrid Intelligence for Growing Internet and its Applications.
  * Li et al. (2022a) Wei Li, Yang Li, Vlad Pandelea, Mengshi Ge, Luyao Zhu, and Erik Cambria. 2022a.  Ecpec: Emotion-cause pair extraction in conversations.  _IEEE Transactions on Affective Computing_ , 14(3):1754–1765.
  * Li et al. (2022b) Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. 2022b.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4804–4814.
  * Liang et al. (2023) Xingwei Liang, You Zou, and Ruifeng Xu. 2023.  Si-lstm: Speaker hybrid long-short term memory and cross modal attention for emotion recognition in conversation.  _arXiv preprint arXiv:2305.03506_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * McKeown et al. (2012) Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder. 2012.  The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent.  _IEEE Transactions on Affective Computing_ , 3(1):5–17.
  * Pamungkas (2019) Endang Wahyu Pamungkas. 2019.  Emotionally-aware chatbots: A survey.
  * Poria et al. (2019) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019.  MELD: A multimodal multi-party dataset for emotion recognition in conversations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 527–536, Florence, Italy. Association for Computational Linguistics.
  * Poria et al. (2020) Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Deepanway Ghosal, Rishabh Bhardwaj, Samson Yu Bai Jian, Romila Ghosh, Niyati Chhaya, Alexander F. Gelbukh, and Rada Mihalcea. 2020.  Recognizing emotion cause in conversations.  _CoRR_ , abs/2012.11820.
  * Sak et al. (2014) Haşim Sak, Andrew Senior, and Françoise Beaufays. 2014.  Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition.  _arXiv preprint arXiv:1402.1128_.
  * Sawhney et al. (2021) Ramit Sawhney, Harshit Joshi, Alicia Nobles, and Rajiv Ratn Shah. 2021.  Towards emotion-and time-aware classification of tweets to assist human moderation for suicide prevention.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 15, pages 609–620.
  * Wang et al. (2021) Fanfan Wang, Zixiang Ding, Rui Xia, Zhaoyu Li, and Jianfei Yu. 2021.  Multimodal emotion-cause pair extraction in conversations.  _CoRR_ , abs/2110.08020.
  * Wang et al. (2023) Fanfan Wang, Zixiang Ding, Rui Xia, Zhaoyu Li, and Jianfei Yu. 2023.  Multimodal emotion-cause pair extraction in conversations.  _IEEE Transactions on Affective Computing_ , 14(3):1832–1844.
  * Wang et al. (2024) Fanfan Wang, Heqing Ma, Jianfei Yu, Rui Xia, and Erik Cambria. 2024.  Semeval-2024 task 3: Multimodal emotion cause analysis in conversations.  In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_.
  * Wei et al. (2020) Penghui Wei, Jiahao Zhao, and Wenji Mao. 2020.  Effective inter-clause modeling for end-to-end emotion-cause pair extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3171–3181, Online. Association for Computational Linguistics.
  * Xia and Ding (2019a) Rui Xia and Zixiang Ding. 2019a.  Emotion-cause pair extraction: A new task to emotion analysis in texts.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1003–1012.
  * Xia and Ding (2019b) Rui Xia and Zixiang Ding. 2019b.  Emotion-cause pair extraction: A new task to emotion analysis in texts.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1003–1012, Florence, Italy. Association for Computational Linguistics.
  * Yun and Park (2022) Jeewoo Yun and Jungkun Park. 2022.  The effects of chatbot service recovery with emotion words on customer satisfaction, repurchase intention, and positive word-of-mouth.  _Frontiers in psychology_ , 13:922503.


# Leveraging Google’s Publisher-specific IDs to
Detect Website Administration

Emmanouil Papadogiannakis  FORTH/University of CreteGreece ,  Panagiotis
Papadopoulos  Telefonica ResearchSpain ,  Evangelos P. Markatos
FORTH/University of CreteGreece and  Nicolas Kourtellis  Telefonica
ResearchSpain

(2022)

###### Abstract.

Digital advertising is the most popular way for content monetization on the
Internet. Publishers spawn new websites, and older ones change hands with the
sole purpose of monetizing user traffic. In this ever-evolving ecosystem, it
is challenging to effectively answer questions such as: Which entities
monetize what websites? What categories of websites does an average entity
typically monetize on and how diverse are these websites? How has this website
administration ecosystem changed across time?

In this paper, we propose a novel, graph-based methodology to detect
administration of websites on the Web, by exploiting the ad-related publisher-
specific IDs. We apply our methodology across the top 1 million websites and
study the characteristics of the created graphs of website administration. Our
findings show that approximately 90% of the websites are associated each with
a single publisher, and that small publishers tend to manage less popular
websites. We perform a historical analysis of up to 8 million websites, and
find a new, constantly rising number of (intermediary) publishers that control
and monetize traffic from hundreds of websites, seeking a share of the ad-
market pie. We also observe that over time, websites tend to move from big to
smaller administrators.

Web Administration, Advertising, Web Monetization, Identifiers

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
ACM Web Conference 2022; April 25–29, 2022; Virtual Event, Lyon,
France††booktitle: Proceedings of the ACM Web Conference 2022 (WWW ’22), April
25–29, 2022, Virtual Event, Lyon, France††price: 15.00††doi:
10.1145/3485447.3512124††isbn: 978-1-4503-9096-5/22/04††ccs: Information
systems Web applications††ccs: Information systems Online advertising††ccs:
Information systems Web mining

##  1\. Introduction

Digital advertising keeps the content we consume on the Web free of charge,
being an important stream of revenue for web publishers (Cramer-Flood, 2021).
Even during 2020, with all the adverse economic impacts of the COVID-19
pandemic, there was a reported 12.2% increase in ad-revenues (Laboratory,
2021), and $100s of billions in annual spending worldwide ($455B in 2021
(Cramer-Flood, 2021)). However, it is inherently difficult to assess the
effectiveness of digital ad spending, due to the overly complex, layered
ecosystem of digital marketing, with thousands of intermediaries brokering ads
and ad-slots between sellers and buyers. Some even consider this market
overvalued and possibly due to correction with various adverse effects (Hwang,
2020).

In an attempt to increase ad profits, advertisers, intermediaries and
publishers resort to analytics and other web tracking services for better
measuring of user audiences and their engagement with webpages. But the
increasing complexity of this ecosystem makes it hard to answer questions such
as: Who are the entities that control and monetize websites, and which
websites? How many websites does the average such entity control? Are they
from the same category, or diverse in nature? What are the characteristics of
these website administrators, and how have these changed over time?

In the last decade, journalists and academic researchers have grappled with
such questions. In fact, they made several efforts to (i) provide more
transparency to the ecosystem of web content monetization and administration
(Silverman et al., 2017), (ii) raise awareness of its impact to user’s privacy
due to online tracking and possibilities of de-anonymization (Baio, 2011;
Matic et al., 2015; Starov et al., 2018; Bashir et al., 2019; Yoon et al.,
2019), (iii) shed light on how this ecosystem drives misinformation and fake
news (Alexander, 2015; Silverman et al., 2017; Samson, 2018). For example, L.
Alexander (Alexander, 2015) used Google analytic IDs to find evidence of a
pro-Kremlin concerted web campaign, executed among different websites owned by
the same entity. C. Silverman et al. (Silverman et al., 2017) looked into
Google-related IDs and found websites being operated by the same entities,
which promoted fake news content and delivered polarizing ads during the USA
2016 presidential election. Furthermore, C.I. Samson (Samson, 2018) discussed
the issue of fake news spreading within the context of the Philippines 2016
presidential election. Such reports demonstrate the urgent need for more
transparency in the issue of website administration. In addition, academic
works (Matic et al., 2015; Starov et al., 2018; Bashir et al., 2019; Yoon et
al., 2019) have looked at the problem from the point of view of user tracking
or de-anonymization using such Google-related IDs to detect malicious websites
and their administrators. However, to date, there has been no systematic study
to reveal, at scale, the way websites are monetized and by which entities.

In this work, we try to shed light on website administration and propose a
novel, graph-based methodology to detect entities that are in charge of
websites. To that extent, we exploit the ad-related, publisher-specific IDs
that publishers embed in their websites in order to use third-party services.
We (i) apply our methodology across the top 1 million websites of Tranco list,
to detect groups of websites monetized by the same entity, (ii) study the
characteristics of the generated website administration graphs, (iii) find
_intermediary_ publishers that manage and monetize traffic from hundreds or
even thousands of websites. We perform a 2-year historical analysis of up to
top 8 million websites and explore how the small, medium and large publishers
have evolved over time.

In summary, the contributions of this work are:

  * •

We propose a novel methodology for detecting website administration and co-
ownership based on publisher-specific IDs, with applicability in different use
cases.

  * •

We conduct the first, to our knowledge, large-scale systematic study of such
publisher-specific IDs embedded in up to 8M websites. We make our
implementation (Papadogiannakis, 2021a) along with our results
(Papadogiannakis, 2021b) publicly available to support further research on
this topic.

  * •

Our findings show that approximately 90% of the websites are associated with a
single publisher and that small publishers tend to manage less popular
websites. We also conclude that there is preferential administration with an
inclination towards “News and Media” websites. Finally, we show that over
time, websites tend to move away from big administrators into smaller ones.

##  2\. Publisher-specific IDs

###  2.1. Google AdSense

_AdSense_ is a service for publishers to generate revenue by displaying ads in
their websites. For ads to be displayed, publishers need to insert the AdSense
code snippet in their website, which includes a Publisher ID: a unique
identifier for an AdSense account that follows the format pub-XXXXXXXXXXXXXXX.
The owner of the account is allowed to share the account with employees, or
even business partners, however, the account holder is always one, and
different AdSense accounts cannot be merged (Center, 2021c). An AdSense
account cannot be transferred to another individual (Center, 2021b), but two
or more AdSense accounts with different Publisher IDs can co-exist on the same
website (Center, 2021e). These other Publisher IDs can belong to a business
partner, contributing authors, or even third-parties.

###  2.2. Google Tag Manager (GTM)

_Google Tag Manager_ (GTM) is a service for web administrators to manage code
snippets (called _Tags_ \- provided by third-parties to integrate their
respective service e.g., analytics, marketing, support) in their website. GTM
provides an interface for publishers to handle such code snippets, which uses
an abstraction called container that needs to be installed in the website by
inserting its own snippet (Farney, 2016). A container is uniquely identified
by a Container ID, formatted as GTM-XXXXXX. One GTM account can create and
manage more than one containers. Usually, a GTM account represents the topmost
level of organization and, typically, an organization uses a single GTM
account (Center, 2021f). Thus, containers are not bound to a domain or a
website, and with the appropriate configuration, the same container can be
used in multiple websites (Center, 2021d).

###  2.3. Google Analytics

_Google Analytics_ is a service to track and report website traffic. The
service revolves around Properties, which contain the reports and traffic data
for one or more websites or applications. There are two types: (i) _Google
Analytics 4_ Properties that are identified by a Measurement ID, which follows
the format G-XXXXXXX, and (ii) _Universal Analytics_ (the older version of
properties (Center, 2021h)) that are uniquely identified by a Tracking ID,
formatted as UA-000000-1. When a user creates a Google Analytics account, a
unique identifier is created that acts as a prefix of Tracking IDs (i.e., the
first set of numbers). Consequently, the Tracking ID which is included in the
code snippet indicates which account data is sent to (Center, 2021g). The
suffix of a Tracking ID represents the property that data is sent to. A
website publisher that owns more than one website is able to associate a
single property with all of these websites.

##  3\. Methodology

Table 1. Detected publisher-specific IDs and their origin. | Unique | Unique | % of |  |  |
---|---|---|---|---|---|---
Description | IDs | URLs | sites | HTML | Reqs | Cookies
Publisher IDs | 71,745 | 87.273 | 10.05% | 99.4% | 77.6% | 0.00%
Tracking IDs | 485,405 | 451.498 | 52.02% | 76.9% | 94.3% | 22.5%
Measurement IDs | 47,087 | 47,606 | 5.48% | 96.0% | 91.2% | 0.36%
Container IDs | 193,693 | 179,114 | 20.64% | 99.7% | 93.0% | 0.01%

###  3.1. Crawling Methods

To detect websites operated by the same entity, we search for the identifiers
of the respective services described in Section 2. Specifically, we develop a
Puppeteer-based crawler that instruments instances of the Chromium Browser.
Using these instances, we crawl with clean state the landing page of the top
962K websites of the Tranco list, which aggregates the ranks from the lists
provided by Alexa, Umbrella, and Majestic, from 16/3/2021 to 14/4/2021(Tranco,
2021). This list is formed based on techniques which enable list stability,
facilitate reproducibility, and protect against adversarial manipulation. The
implementation of our crawler is publicly available (Papadogiannakis, 2021a).
When our crawler visits a website, it waits until the page has completely
loaded and for an additional 5-sec period to ensure that all programmatically
purchased ads (via Real Time Bidding (RTB)) have been rendered. Then, it
stores the HTML of the page, a cookie-jar and the HTTP(S) requests performed
during the website visit. We capture all requests passively in a read-only
fashion without mutating or intercepting them. This ensures that the behavior
of the website is not affected by our crawler. To collect the HTML of the
website, we utilize the Chrome DevTools Protocol (The Chromium Authors, 2014).
This way, we ensure that we capture not only the actual HTML code but also the
documents, styles or code fetched by iFrames or code snippets. Our crawler
visits 962K websites of the Tranco list (1MT crawl) from 15-27/4/2021 and
collects 415GB of data. In 93,817 websites (9.75%) the crawling process failed
due to timeouts or site inaccessibility. Overall, we detect 525,493 websites,
with at least one of the identifiers discussed in Section 2. Ethical concerns
regarding the crawling process and collected data are addressed in Appendix A.

###  3.2. Detecting Identifiers

We detect the Google Identifiers described in Section 2 by performing an
offline analysis on the collected data. Specifically, using regular
expressions111pub-[0-9]{9,}, UA-[0-9]{4,}-[0-9]+, G-[A-Z0-9]{7,} and
GTM-[A-Z0-9]{6,}, we search for these identifiers inside the page content,
HTTP(S) requests and stored cookies. Then, we remove false positives using a
combination of data-filtering techniques.

First, using the dictionary of GNU Aspell (Project and Atkinson, 2019), an
open-source spell-checking tool, we remove values which are words of the
English dictionary, but match the suffix of regular expressions (e.g.,
G-BACKPACK). Using this technique, we were able to remove $\sim$1,500 distinct
false positive values, which were found in $\sim$5,000 unique websites. Then,
we remove false positives using a list of common keywords. This list was
generated by manually inspecting over 10,000 values that satisfy the regular
expressions, and investigating whether they are actually used as identifiers.
Our keyword list contains over 1,250 values which were filtered out (e.g.,
G-APRIL2020).

As in Table 1, we find that $\sim$10% of the most popular websites monetize
their content through an AdSense account (i.e., Publisher ID), $\sim$52% use
Google Analytics to track their traffic, and $\sim$20% use the Google Tag
Manager for easier management of code snippets. Moreover, for some services,
we observe that there are more domains than publisher-specific IDs. This
suggests that some identifiers are being re-used in more than one website.
Additionally, we examine the source of information for each type of
identifier. Specifically, we investigate whether the identifiers can be found
in the HTML code of a website, its outgoing network traffic, or in the cookies
set by either the first-party or various third-parties. As shown in Table 1,
regardless of the type of identifier, the majority of them can be found in
both the HTML code of the website and the HTTP(S) requests. This result is
inline with the official guidelines for using Tags (Center, 2021a). This
indicates that the detected identifiers are not only valid, but since they are
sent to the respective Google services, they are in use. Finally, we find that
only Tracking IDs are commonly found in cookies.

Figure 1. Example of a Publisher ID bipartite graph. Blue nodes represent
websites and red nodes represent Publisher IDs. A directed edge in these
bipartite graphs indicates that the website contains the respective
identifier. Figure 2. Example of metagraph construction. Websites that share
an identifier, share an undirected edge in the resulting graph. The weight of
the edge rises proportionally with the number of common publisher-specific
IDs.

###  3.3. Bipartite graphs

Using the detected publisher-specific IDs, we construct a bipartite graph for
each of the respective types of identifiers. In these graphs, the nodes are
either websites or identifiers. Whenever a website contains an identifier, we
introduce a directed edge from the respective website node to the respective
identifier node. Tracking IDs and Measurement IDs are placed into the same
graph since they represent the same service and have similar functionality.
Thus, we create three bipartite graphs. Moreover, for Tracking IDs, we focus
only on the prefix, which refers to the account number, as discussed in
Section 2.3. Figure 1 illustrates an example of a small Publisher ID bipartite
graph.

###  3.4. Metagraph

We also form a metagraph based on the three bipartite graphs of different
publisher-specific IDs. The metagraph contains nodes only for websites and
represents the relationships between websites. Whenever two websites share an
identifier, we introduce an undirected meta-edge between the two respective
website nodes. The more identifiers two websites share, the greater the weight
of the connecting edge. Each shared identifier increases the weight of the
meta-edge by $\frac{1}{n}$, where $n$ is the total number of distinct
identifiers of this type found in more than one website. A larger edge weight
between two websites implies greater confidence that they are indeed operated
and monetized by the same entity. Figure 2 illustrates an example of how the
metagraph is constructed. The code to construct both the bipartite graphs and
the metagraph is publicly available (Papadogiannakis, 2021b).

Figure 3. Distribution of Girvan-Newman community sizes in the metagraph. Over
60% of the communities are made of just two sites.

Figure 4. Distribution of publisher-specific IDs per website. Most websites
contain only one identifier.

Figure 5. Number of websites monetized by each publisher. Most publishers
($\sim$63,000) appear in only one site.

Figure 6. Connected components sorted by size (number of nodes) for the
Publisher ID bipartite graph.

###  3.5. Metagraph Validation

We hypothesize that the metagraph constructed by following the steps above can
lead to clusters of websites operated by the same entity. Meta-edges reflect
the actual relationships between websites, thus the ones operated by the same
entity should form a strongly connected community. Since the metagraph
combines information from multiple service (i.e., bipartite graphs), it
provides us with greater confidence about the actual relationships of the
websites.

We find that there are some outlying cases where communities consist of
thousands of websites. After manual investigation, we conclude that these
communities are formed due to _intermediary publishing partners_. These
publishers provide services for content creators to monetize their content or
improve their website traffic and require that websites integrate the
partner’s identifiers. To focus our analysis on an enhanced level of
granularity, we ignore such publishers for the time being. This allows us to
study more detailed cases of website administrators and results in a metagraph
that contains $\sim$127,000 nodes and $\sim$2,885,000 edges. We discuss
intermediate publishing partners in later sections.

To find the websites operated and monetized by the same entity with high
confidence, we perform edge pruning, thus removing noise. Specifically, we
remove edges that do not belong to the top 5%, when ranked by weight. We
choose this threshold based on empirical analysis. This way, we ensure that
there are limited false positives, i.e., websites that are wrongfully added to
a community because of a typographical error in their source code, or older
identifiers. After the edge pruning, dangling nodes are also removed from the
graph as they do not provide any additional information. To further explore
this graph, we execute the Girvan–Newman community detection algorithm (Girvan
and Newman, 2002). In fact, we compare our methodology with the one described
in (Cangialosi et al., 2016), where the authors apply the Louvain method
(Blondel et al., 2008) to the connected components extracted from their
bipartite graph. Specifically, we manually examine and evaluate 40 distinct
communities, which can be found through both methodologies, consisting of 215
unique websites. We find that applying the Girvan-Newman algorithm, after
edge-pruning, results in better communities in 42.5% of the cases, in exactly
the same communities 37.5% of the cases, and in worse communities in 20% of
the cases. Consequently, the Girvan–Newman algorithm results in higher quality
communities at the expense of a higher computational cost.

Finally, we compare the communities our methodology detects with the (publicly
available) communities detected in (Cangialosi et al., 2016). We acknowledge
that the comparison is difficult to achieve because the two studies focus on
different websites, in a different time period (i.e., 5-year ago: a big
portion of the websites are no longer active and cannot be evaluated) and in a
very dynamic environment like the Web. We compare only communities that
contain websites, all of which have been crawled in our 1MT crawl. In total,
we manually evaluated and found 12 communities with results similar to ours
and 15 cases where the methodology of (Cangialosi et al., 2016) fails and
places websites which are operated by the same legal entity into different
communities. Using our methodology we detect 2,369 communities, formed by
$\sim$11,000 distinct websites. The distribution of community sizes (Figure 3)
shows that the majority of them are small (i.e., <6 websites). Indeed, 61% of
the communities are pairs of websites, indicating that the median publisher
operates just 2 websites.

Figure 7. Average popularity of publishers (based on ranking of sites) vs.
publisher size.

Figure 8. Distribution of categories for websites with Publisher ID.

Figure 9. Poisson sampling experiment for site categories

Figure 10. Shannon’s Diversity Index for website categories in detected
communities.

##  4\. Analysis of Bipartite Graphs

###  4.1. Contained publisher-specific IDs

First, we study how organizations or publishers use identifiers necessary for
Google services. For this, we measure the number of unique identifiers found
in each website of our dataset. In Figure 6, we plot the portion of websites
in our data that contain a certain number of identifiers. Around 82-83% of the
websites contain only one Container ID or Analytics ID, and about 90% of
websites have only one Publisher ID. This indicates that the majority of
organizations prefer to use the simplest and most straightforward
configuration of services in their websites, where they use a single
identifier to achieve their goal, be it monetization or traffic measuring.
Most importantly, in the case of Publisher IDs, it indicates that the majority
of websites have a single contributing author and that revenue is not shared.
This is contrasting the small portion of websites (less than 3.2% in all
cases), with 3 or more identifiers, which indicate multiple collaborating
authors, with their own Publisher IDs, contributing to a website.
Surprisingly, we see a small number of websites with an extremely large number
of identifiers. For instance, we find that _prykoly.ru_ contains 94 Tracking
IDs, while _www.pps.net_ , a website for public schools in Portland, contains
88 IDs hard-coded in the JavaScript code and the correct identifier is
selected based on the page that is visited. To further investigate this an
abnormal behavior, we lookup these websites in the _VirusTotal_(Limited,
[n.d.]) and _Sucuri_(GoDaddy Media Temple, 2021a) security services for
malicious content. Sucuri reports(GoDaddy Media Temple, 2021b) that prykoly.ru
contains a known JavaScript malware associated with a back-link purchase
service, called Sape. For pps.net, Virustotal reports(Limited, 2021) that
there are 4 detected files that communicate with this domain. In total, we
find 67 distinct websites with over 40 publisher-specific IDs of any type.
This preliminary analysis suggests that cases with numerous publisher-specific
IDs in a website might imply abnormal or even malicious behavior. This
observation, though interesting, is considered as out of scope for this work
and left for future research.

###  4.2. Publisher Size

Next, we explore the amount of websites that publishers manage and monetize.
For each publisher, we measure the number of websites in which they place
their publisher-specific IDs. Analysis from now on, is performed on distinct
domains of landing pages, and not distinct domains in the Tranco list.
Specifically, if two different domains in the Tranco list redirect to the same
domain, we measure this website only once towards the size of the respective
publisher. In Figure 6, we plot, in descending order, the number of websites
monetized by each unique Publisher ID in our data. We show that the great
majority of publishers (up to 87.8%), monetize traffic from a single site. On
the other hand, we find 340 publishers monetizing traffic from more than 10
websites each.

We observe some “mega-publishers” that can be found in hundreds or even
thousands of websites. Indeed, the top 10 publishers in our data can be found
in a total of more than 4,200 websites. We observe similar behavior in
Container IDs and Analytics ID, where we find that the top 10 identifiers can
be found in a total of 4,245 and 6,795 websites, respectively. To verify this
finding, we explore the connectivity of the three bipartite graphs (described
in Section 3.3) and generate a list of connected components in each graph. In
Figure 6 we plot, in decreasing order using a log-log scale, the number of
nodes in each connected component of the Publisher ID bipartite graph. We see
that the distribution of connected component sizes can fit a power-law with a
cutoff, an anomaly due to the intermediary publishers mentioned earlier. By
applying appropriate statistical tests (Clauset et al., 2009), we find that
the distribution is indeed heavy-tailed with power law being a better fit than
the exponential distribution (loglikelihood ratio test with $p=1.6e^{-8}$). We
find similar results for the Container IDs and Analytics IDs bipartite graphs
but exclude them for brevity. This verifies our finding that there are only a
few publishers monetizing traffic from a very large number of websites, while
the majority of publishers operate one website.

We attribute this behavior to the existence of _intermediary publishing
partners_ (Google, 2021). These are third-party services which provide
services for content creators to readily deliver their content, effortlessly
monetize it, optimize their revenue and deliver better experiences to users.
Publishers are required to integrate the service’s identifiers in their
websites so that the service can monitor traffic and user behavior, and
deliver ads. By examining requests towards third-party services, we
successfully identified multiple such “mega-publishers”, including Ezoic,
optAd360, Blogger and ProjectAgora. Specifically, Blogger provides publishers
with AdSense gadgets, which can be used to display ads in a blog, without
taking any percentage of earnings (Community, 2019). At the moment of writing,
the _PublicWWW_ service (PublicWWW, [n.d.]) reports that Blogger’s Publisher
ID can be found in more than 364,000 websites.

###  4.3. Monetizer Popularity

Next, we explore if there is an association between the popularity of website
and the size of the publishers. First, we group together websites that share
the same Publisher ID, meaning that there is a single account responsible for
their monetization. For each such account (i.e., publisher), we compute its
popularity as the average rank of the websites it operates, based on the
Tranco list. In Figure 10, we plot this average popularity of publishers, for
a given size of publishers. We show that the average website popularity
(y-axis) increases (i.e., Tranco rank decreases), as the number of monetized
websites increases (x-axis). The average popularity subsections have also been
fitted with a straight line (the negative slope is from the reversed y-axis to
indicate increased popularity) indicating a clear trend with a $R^{2}=0.28$.
We observe similar behavior when plotting the median popularity, indicating
that there is no skewness in the distribution.

As a result, independent publishers who generate revenue from a single
website, tend to monetize less popular websites. On the other hand, publishers
that manage multiple websites, usually manage the most popular ones.
Consequently, in a classic case of _rich getting richer_ , big publishers who
operate dozens of websites, not only claim a bigger share of the market and
generate a bigger revenue, but they are also able to improve their reputation
and attract more visitors. The increased popularity of some websites can also
be credited to the intermediary publishers mentioned earlier.

Figure 11. Total websites crawled and portion of websites that contain
Publisher ID and Tracking IDs.

Figure 12. Distribution of Publisher IDs per website through time. Almost 90%
of websites in all snapshots contain only one Publisher ID.

Figure 13. Transition of websites between publishers through time.

###  4.4. Website Categories

Manual inspection of communities (Section 3.5) revealed that most operators
tend to manage websites with similar content. To investigate this further, we
retrieve (if available) from _SimilarWeb_ (LTD, [n.d.]) the category of each
website with a Publisher ID in our dataset. Figure 10 illustrates the
distribution of the categories we retrieved from over 23,000 websites with a
Publisher ID. Websites with no category information are excluded from the
analysis. We see a preference towards “News and Media” websites (24.5%),
followed by websites related to “Computers, Electronics and Technology”
(18.6%), “Arts” (11%) and “Science” (8%).

Next, we investigate if there is a preference in the types of websites a
publisher monetizes, or if their portfolio is usually random. As a first step,
we perform a Poisson Sampling experiment to construct a scenario where
publishers monetize websites based on their categories’ measured popularity in
the data. For this sampling, we perform the following steps. For a given size
of publisher (i.e., number of websites they operate and monetize), we randomly
select websites from our data. For example, if the size of a publisher is 10,
we randomly select 10 websites from the 23K websites with a category. However,
this selection is biased, based on the prior probability of a website
appearing due to its category. Thus, for example, a “News and Media” website
has almost 1/4 chance to be selected in any publisher. We perform this process
for all publishers. Then, for each publisher, we compute the number of unique
website categories they have in their control (i.e., _richness_).

Figure 10 plots the richness distribution of the observed (or “actual”) data
and the Poisson-Sampling experiment data. The $y=x$ line represents the case
of a uniform distribution, where all of a publisher’s websites come from
different categories, with equal probability. We see that the average number
of website categories in our actual data is lower than a purely probabilistic
choice, for every possible publisher size. This actively demonstrates a
preferential administration of websites when it comes to their category. Thus,
publishers tend to monetize websites with similar type of content. To verify
this hypothesis, we also utilize Shannon’s diversity index (Shannon, 1948).
Shannon’s diversity index is a statistical measure, which provides information
about the composition of a community. It is defined as
${H}^{\prime}=-\sum_{i=1}^{S}p_{i}\ln p_{i}$, where $S$ is the number of
different categories in the dataset (i.e., richness) and $p_{i}$ is the
proportion of websites belonging to category $i$. The maximum value of the
diversity index is $ln(R)$, where $R$ stands for the number of distinct
website categories in a community. This represents the case where all
categories are equally common inside a cluster of websites operated by the
same publisher. As we can see in Figure 10, Shannon’s diversity index for the
websites in our data is much smaller than the maximum value and is closer to
zero. A smaller diversity index corresponds to a more unequal composition of
the community. We conclude that there is indeed a preferential administration
or monetization and that publishers tend to acquire new websites of the same
type as the ones they already manage.

##  5\. Historical Analysis

###  5.1. Historical Presence of Identifiers

We perform a historical analysis of the last two years (April 2019 to April
2021), on a trimester basis, resulting in 9 snapshots. We use the entire
dataset of HTTPArchive (Archive, 2021) and do not limit ourselves to the
websites found in the Tranco list. In Section 3.2, we show that HTTP requests
are a reliable source to find identifiers of interest. Thus, we examine
HTTP(S) requests of websites in these snapshots and detect Publisher IDs and
Tracking IDs embedded in them. Figure 13 illustrates our findings for the
total number of websites crawled, per trimester snapshot. We find that, on
average, 9.9% of websites monetize their content using Google AdSense
Publisher IDs, while around 64% use Google Analytics in order to track and
measure their traffic. These trends are stable over both the snapshots and the
sample size, with a standard deviation of only 0.7 and 1.81, respectively.
These results, computed on $8\times$ more websites than the 1MT crawl (up to
$\sim$8M websites in 2021), are inline with our earlier findings described in
Section 3, and lend credence to our analysis as being representative of the
general Web.

Figure 14. Transition linear trends between different size publishers through
time.

Figure 15. Number of websites monetized by the top 10 publishers in each
snapshot.

Figure 16. Changes in the population of different sizes of publishers.

Next, we study how many publishers contribute to the content of a website. In
Figure 13, we plot the portion of websites that contain one, two, or three or
more Publisher IDs for each time period. On the $y_{2}$ axis, we plot the
average number of distinct Publisher IDs in each website and observe that it
is almost constant across time, with a mean value of 1.11, and a standard
deviation of only 0.015. We also find that, on average, 88.75% of the
websites, have a single contributing publisher that generates revenue.
Finally, we find a very small amount of websites (less than 1% in all
snapshots) that contain identifiers of 3 or more publishers. These cases are
either due to an intermediary publishing partner, or websites running under
the partnership of various authors or authorized external authors. Overall,
these numbers match our earlier in-depth analysis using the 1MT crawl of April
2021.

###  5.2. Top Publishers Market

Next, we explore how the market of publishers has changed in the last couple
of years, and specifically, how intermediary publishing partners have grown.
First, we study how websites behave with regards to their Publisher IDs and
detect changes of these identifiers. We perform our analysis on websites which
have been crawled in all snapshots (i.e., their intersection) and contain at
least one Publisher ID. There are over 191,000 such websites. For each time
interval and for each website, we compare the detected identifiers in the
previous and the next snapshot. We ignore websites that made no change in
their Publisher IDs. For websites that do not contain exactly the same
identifiers across two consecutive snapshots, we compare the size of their
publishers. Specifically, we define the old community size of a website as the
maximum size of its publishers, detected in the first snapshot for that
website. Respectively, we retrieve the new community size from the second
snapshot. Please note that the size of a publisher for a specific snapshot is
computed across all websites in the snapshot and not only for the common
websites. If the new community size of a website is greater than the old size,
then we conclude that the website moved to a bigger publisher. If the old size
is greater, the website moved to a smaller publisher, while if the size is the
same, then the publisher made an insignificant change. This change might be
the addition or removal of a secondary contributing author, the move to a
different AdSense account, etc. The results of this analysis can be seen in
Figure 13. We observe that the majority of websites made no changes in their
Publisher IDs. Indeed, over 96% of websites have a consistent and stable
behavior and do not change their monetization scheme. In contrast, we find
that, on average, 3.35% of websites made a change in their contained Publisher
IDs.

In Figure 16, we show a linear regression model for the different cases of
Figure 13. Interestingly, we find that both the cases where a website moved to
a bigger publisher, and where a website made an insignificant change in their
Publisher IDs, have a negative slope. In contrast, the case where websites
move to a smaller publisher is the only one with a positive slope. This
suggests that there is a tendency for decentralization, meaning that websites
are inclined to move away from big intermediary publishers. To test this
hypothesis, we plot in Figure 16 the sum of websites operated by the 10 most
popular publishers in each snapshot along with the total number of websites
operated by the top 10 publishers, present in all snapshots. We can see that
there is a constant decrease in the number of websites that these “mega-
publishers” manage. Indeed, in just a 2-year span, “mega-publishers” lost
approximately 25% of their population. Interestingly, this decrease in managed
websites is observed even though the number of crawled websites has increased
over the years (as shown in Figure 13).

Next, we explore how this market of big publishers has changed over time. We
characterize as Small publishers with up to 10 websites, Medium those that
monetize from 11 to 50 websites, Large those who monetize from 51 to 100, and
as Mega, the publishers that monetize more than 100 websites. In Figure 16, we
plot the population of these classes, i.e., the number of such publishers, and
we also fit the data subsections with a straight line. As we can see, the
number of Small publishers has greatly increased over the years ($\sim$15K new
Small publishers per trimester), which is expected with the increasing ad-
revenues motivating new independent publishers to monetize their content. We
also observe that there is an increase in the number of Medium and Large
publishers ($\sim$29 new Medium and $\sim$5 new Large publishers per
trimester), while Mega publishers are the only class that shrinks over time
($\sim$2 Mega publishers lost per trimester). This is evident in the negative
slope of the fitted straight line. This behavior attests to the fact that the
market of intermediary publishing partners seems to be flourishing and that
new such services have emerged during the last couple of years. These services
provide a new platform for independent content creators to generate revenue
and lure clients away from Mega publishers. It is evident that these new
services seek their share of a competitive, but highly profitable market.

##  6\. Detecting Website Ownership

During our manually analysis, we found that there were a lot of communities
which were not only operated but also owned by the same legal entity. To
better understand the utility of the metagraph in identifying websites owned
by the same legal entity, we manually examine detected communities.Overall, we
find communities belonging to the news and media sector, music and
entertainment, as well as manufacturing and other industrial applications. As
an example, we detect a community of websites owned by Koninklijke Philips
N.V. We find 45 official websites, each with a different country code top-
level domain (ccTLD), all of which belong to the same company but serve
clients of different countries. Another community in our dataset is a cluster
of 73 news websites, all serving news content using .au as their ccTLD. In
their privacy policy, these websites mention that they are published by a
subsidiary or related body corporate of _Rural Press Pty, Ltd_ and, in their
footer, they declare that they are operated by _Australian Community Media &
Printing_. Australian Community Media is a media company operating over 160
regional publications and targets a vast audience in multiple geographic
locations.

One of the biggest detected communities is related to the music industry
(i.e., over 140 websites of popular singers or music bands). To our surprise,
these websites are owned by various companies including Atlantic Records,
Electra Records, Warner Records and Nonesuch Records. By observing the
copyright notification and the privacy notices of these websites, we find that
all of them are subsidiaries of a single multinational conglomerate, _Warner
Music Group_ (Group, 2021). This is a clear proof that our methodology is even
able to overcome the barriers of business organization and subsidiaries, and
detect ownership in the highest level of hierarchy. Finally, we find two
communities of websites related to public entertainment and information. We
detect a community of 111 radio websites, operated by _Townsquare Media, Inc_
, a US-based radio network and media company that owns hundreds of local
terrestrial radio stations (Media, 2021). We also detect a community of 76
websites, which explicitly state in their copyrights claim that they are owned
by _Gray Television, Inc._ , an American television broadcasting company.

These examples and many more not analyzed here due to space, demonstrate the
efficacy of our methodology to detect co-ownership status of websites by
organizations monetizing on them in a collective fashion. Overall we find and
report 112 distinct communities of various sizes, consisting of over 1,280
websites. For each community we report its size, the websites that compose the
community, as well as, the legal entity that owns the respective websites. We
manually visited, evaluated and verified all of these websites and make our
results publicly available (Papadogiannakis, 2021b). We report some of the
largest communities along with their size (i.e., number of websites) and some
indicative websites as examples in Appendix B.

##  7\. Related Work

The ecosystem of digital advertising and analytics has motivated a lot of
studies that aim to reverse engineer it (e.g., (Papadopoulos et al., 2017,
2018; Carrascosa et al., 2015; Englehardt and Narayanan, 2016)). In (Gill et
al., 2013), the authors studied the advertising ecosystem and services
provided by Google, including AdSense and focused on how revenues are
generated across aggregators. In (Matic et al., 2015), the authors presented
an automated tool to de-anonymize Tor hidden services using information like
Google Analytics and AdSense IDs to disclose the server’s IP. Their analysis
is limited regarding publisher-specific IDs, since they only extract 24 unique
Analytics IDs and 3 Publisher IDs. Similarly, in (Yoon et al., 2019), Yoon et
al. studied phishing threats in the DarkWeb, by trying to obtain the identity
of owners operating such websites. Using the technique of (Matic et al.,
2015), they extracted 276 Analytics IDs and 1,171 Publisher IDs. In (Starov et
al., 2018), Starov et al. analyzed identifiers of multiple analytics services
to bundle websites and discover malicious websites and campaigns. With a focus
on malicious content, they identified 7,945 Analytics IDs and 278 Container
IDs and, contrary to our work, they did not consider Publisher IDs or
Measurement IDs. In (Rogers, 2021), authors outlined how Google Analytics IDs
can be used for digital forensics investigations to unmask online actors and
lead to the entity, that operates a cluster of websites, which can be an
individual, organization, or media group.

In (Simeonovski et al., 2017), the authors associate organizations with domain
names in an attempt to create a property graph for Internet infrastructures.
To achieve this, they utilize X.509 certificates and extract the organization
to which the certificate was issued. In (Cangialosi et al., 2016) authors,
argue that relying on such certificates is not effective and propose a
methodology that revolves around the email addresses found in WHOIS records.
Similar to our work, authors build a bipartite graph and apply a community
detection algorithm to extract clusters of domains, owned by the same
organization. Limitations of the proposed methodology include that many WHOIS
records contain the email address of the registrar or the hosting provider
instead (i.e., WHOIS privacy service). Finally, our methodology provides an
additional advantage in the cases where websites are purchased by a new legal
entity, since new website owners or administrators have an incentive to update
the publisher-specific IDs in their new websites in order to gain revenue or
insight. This does not apply to WHOIS records.

In (Bashir et al., 2019), Bashir et al. performed a study of the specification
and adoption of ads.txt files during a 15-month period and clustered
publishers serving identical ads.txt files. Similar to our work, they found
that there is a big number of smaller clusters (i.e., less than 5 websites)
but only a few big clusters with over 50 websites. Finally, authors manually
investigated the top clusters in their dataset and found that such clusters
exist due to (i) shared media properties with a common owner, (ii) independent
publishers, (iii) the use of the same platform to deliver their content, or
(iv) the use of consolidated SSP services.

##  8\. Discussion

###  8.1. Summary

In this work, we shed light on website administration by using bipartite
graphs and exploiting the publisher-specific IDs that publishers embed in
their websites to use third-party ad-related services. We studied various
properties induced by these graphs, reflecting important characteristics of
administration, such as portfolio size, popularity, etc., and we identified
power-law patterns of website administration, as well as indications of
preferential monetization in the type of controlled websites. We studied the
use of such publisher-specific IDs across time and showed how the market of
intermediary publishing partners has boomed in the last few years. We showed
that our methodology can be used to detect ownership in the Web and even
overcome the company organization barriers (i.e., subsidiary companies).

###  8.2. Limitations

Our methodology is based on detecting publisher-specific IDs using regular
expressions. However, there are cases where alphanumeric values might match
with these regular expressions without being actual identifiers. While we
perform various techniques to limit these false positives (Section 3.2), we
acknowledge that there might be cases that we miss. Additionally, our study
focuses on publisher-specific IDs related to services offered by Google, one
of the biggest players in the advertising and analytics ecosystem. Even though
the analysis of Google services provides a good coverage of the real world,
there are several other ad networks and analytics services that can be
studied. Finally, we acknowledge that our analysis of website categories
(Section 4.4) relies on SimilarWeb, which might be prone to errors or
subjective bias.

###  8.3. Implications

We believe our graph methodology and analysis is a powerful tool for web and
privacy measurements aiming to understand the context, nature and activity of
websites, as well as the possible leverage or political agendas behind their
administration. In fact, our proposed technique can help researchers,
journalists, and even individual users to better understand popular websites
and the entities that control and monetize them. Furthermore, our preliminary
analysis shows that outlier websites in the bipartite graphs yielded by our
method may reveal anomalous or even malicious behavior, suggesting that our
methodology can be used to discover malicious actors without even examining
their published content. Also, ad networks can make use of our technique to
detect fraudulent or fake news-related website administrators that may violate
their ad campaign policies. Altogether, we believe that our method can help
improve the safety and health of the Web ecosystem at large.

## Acknowledgements

This project received funding from the EU H2020 Research and Innovation
programme under grant agreements No 830927 (Concordia), No 830929
(CyberSec4Europe), No 871370 (Pimcity) and No 871793 (Accordion). These
results reflect only the authors’ view and the Commission is not responsible
for any use that may be made of the information it contains.

## References

  * (1)
  * Alexander (2015) Lawrence Alexander. 2015\.  Open-Source Information Reveals Pro-Kremlin Web Campaign.  https://globalvoices.org/2015/07/13/open-source-information-reveals-pro-kremlin-web-campaign/.
  * Archive (2021) Internet Archive. 2021\.  HTTPArchive.  https://httparchive.org/.
  * Baio (2011) Andy Baio. 2011\.  Think You Can Hide, Anonymous Blogger? Two Words: Google Analytics.  https://www.wired.com/2011/11/goog-analytics-anony-bloggers/.
  * Bashir et al. (2019) Muhammad Ahmad Bashir, Sajjad Arshad, Engin Kirda, William Robertson, and Christo Wilson. 2019\.  A Longitudinal Analysis of the Ads.Txt Standard. In _Proceedings of the Internet Measurement Conference_ (Amsterdam, Netherlands) _(IMC ’19)_. Association for Computing Machinery, New York, NY, USA, 294–307.  https://doi.org/10.1145/3355369.3355603
  * Blondel et al. (2008) Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008.  Fast unfolding of communities in large networks.  _Journal of statistical mechanics: theory and experiment_ 2008, 10 (2008), P10008.
  * Cangialosi et al. (2016) Frank Cangialosi, Taejoong Chung, David Choffnes, Dave Levin, Bruce M. Maggs, Alan Mislove, and Christo Wilson. 2016.  Measurement and Analysis of Private Key Sharing in the HTTPS Ecosystem. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ (Vienna, Austria) _(CCS ’16)_. Association for Computing Machinery, New York, NY, USA, 628–640.  https://doi.org/10.1145/2976749.2978301
  * Carrascosa et al. (2015) Juan Miguel Carrascosa, Jakub Mikians, Ruben Cuevas, Vijay Erramilli, and Nikolaos Laoutaris. 2015.  I Always Feel like Somebody’s Watching Me: Measuring Online Behavioural Advertising. In _Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies_ (Heidelberg, Germany) _(CoNEXT ’15)_. Association for Computing Machinery, New York, NY, USA, Article 13, 13 pages.  https://doi.org/10.1145/2716281.2836098
  * Center (2021a) Google Help Center. 2021a.  About Tags.  https://support.google.com/tagmanager/answer/3281060.
  * Center (2021b) Google Help Center. 2021b.  Ad placement policies.  https://support.google.com/adsense/answer/2659106.
  * Center (2021c) Google Help Center. 2021c.  Manage user access to your account.  https://support.google.com/adsense/answer/2646544.
  * Center (2021d) Google Help Center. 2021d.  Organize your containers.  https://support.google.com/tagmanager/answer/6261285.
  * Center (2021e) Google Help Center. 2021e.  Revenue share.  https://support.google.com/adsense/answer/1346295.
  * Center (2021f) Google Help Center. 2021f.  Setup and install Tag Manager.  https://support.google.com/tagmanager/answer/6103696.
  * Center (2021g) Google Help Center. 2021g.  Tracking ID and property number.  https://support.google.com/analytics/answer/7372977.
  * Center (2021h) Google Help Center. 2021h.  Universal Analytics property.  https://support.google.com/analytics/answer/10220206.
  * Clauset et al. (2009) Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. 2009.  Power-law distributions in empirical data.  _SIAM review_ 51, 4 (2009), 661–703.
  * Community (2019) AdSense Help Community. 2019\.  What is the adsenseHostId on blogger?  https://support.google.com/adsense/thread/18637422/what-is-the-adsensehostid-on-blogger-why-there-is-another-pub-id.
  * Cramer-Flood (2021) Ethan Cramer-Flood. 2021\.  Worldwide Digital Ad Spending 2021.  https://www.emarketer.com/content/worldwide-digital-ad-spending-2021.
  * Dittrich and Kenneally (2012) David Dittrich and Erin Kenneally. 2012.  _The Menlo Report: Ethical Principles Guiding Information and Communication Technology Research_.  Technical Report.
  * Englehardt and Narayanan (2016) Steven Englehardt and Arvind Narayanan. 2016.  Online Tracking: A 1-Million-Site Measurement and Analysis. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ (Vienna, Austria) _(CCS ’16)_. Association for Computing Machinery, New York, NY, USA, 1388–1401.  https://doi.org/10.1145/2976749.2978313
  * Farney (2016) Tabatha Farney. 2016\.  _Google Analytics and Google Tag Manager_.  ALA TechSource, Chicago.
  * Gill et al. (2013) Phillipa Gill, Vijay Erramilli, Augustin Chaintreau, Balachander Krishnamurthy, Konstantina Papagiannaki, and Pablo Rodriguez. 2013\.  Best Paper – Follow the Money: Understanding Economics of Online Aggregation and Advertising. In _Proceedings of the 2013 Conference on Internet Measurement Conference_ (Barcelona, Spain) _(IMC ’13)_. Association for Computing Machinery, New York, NY, USA, 141–148.  https://doi.org/10.1145/2504730.2504768
  * Girvan and Newman (2002) Michelle Girvan and Mark EJ Newman. 2002.  Community structure in social and biological networks.  _Proceedings of the national academy of sciences_ 99, 12 (2002), 7821–7826.
  * GoDaddy Media Temple (2021a) Inc. GoDaddy Media Temple. 2021a.  Sucuri - Free website security check and malware scanner.  https://sitecheck.sucuri.net/.
  * GoDaddy Media Temple (2021b) Inc. GoDaddy Media Temple. 2021b.  Sucuri Report - prykoly.ru.  https://sitecheck.sucuri.net/results/prykoly.ru.
  * Google (2021) Google. 2021.  Certified Publishing Partner.  https://www.google.com/ads/publisher/partners/.
  * Group (2021) Warner Music Group. 2021\.  Services - Recorded Music.  https://www.wmg.com/services.
  * Hwang (2020) Tim Hwang. 2020\.  _Subprime Attention Crisis: Advertising and the Time Bomb at the Heart of the Internet_.  FSG Originals x Logic, Farrar, Straus and Giroux, New York.
  * Laboratory (2021) IAB Technology Laboratory. 2021\.  IAB Releases Internet Advertising Revenue Report for 2020\.  https://www.iab.com/news/iab-internet-advertising-revenue/.
  * Limited ([n.d.]) Chronicle Security Ireland Limited. [n.d.].  VirusTotal - Analyze suspicious files and URLs to detect types of malware, automatically share them with the security community.  https://www.virustotal.com/.
  * Limited (2021) Chronicle Security Ireland Limited. 2021\.  VirusTotal Report - pps.net.  https://www.virustotal.com/gui/domain/www.pps.net.
  * LTD ([n.d.]) Similarweb LTD. [n.d.].  Website Traffic - Check and Analyze Any Website.  https://www.similarweb.com/.
  * Matic et al. (2015) Srdjan Matic, Platon Kotzias, and Juan Caballero. 2015\.  CARONTE: Detecting Location Leaks for Deanonymizing Tor Hidden Services. In _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_ (Denver, Colorado, USA) _(CCS ’15)_. Association for Computing Machinery, New York, NY, USA, 1455–1466.  https://doi.org/10.1145/2810103.2813667
  * Media (2021) Townsquare Media. 2021\.  Digital Media and Radio Advertising Company.  https://www.townsquaremedia.com/.
  * Papadogiannakis (2021a) Emmanouil Papadogiannakis. 2021a.  Scrape Titan.  https://gitlab.com/papamano/scrape-titan.
  * Papadogiannakis (2021b) Emmanouil Papadogiannakis. 2021b.  Website Administration Graphs.  https://gitlab.com/papamano/website-administration-graphs.
  * Papadopoulos et al. (2018) Panagiotis Papadopoulos, Nicolas Kourtellis, and Evangelos P. Markatos. 2018.  The Cost of Digital Advertisement: Comparing User and Advertiser Views. In _Proceedings of the 2018 World Wide Web Conference_ (Lyon, France) _(WWW ’18)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1479–1489.  https://doi.org/10.1145/3178876.3186060
  * Papadopoulos et al. (2017) Panagiotis Papadopoulos, Nicolas Kourtellis, Pablo Rodriguez Rodriguez, and Nikolaos Laoutaris. 2017\.  If You Are Not Paying for It, You Are the Product: How Much Do Advertisers Pay to Reach You?. In _Proceedings of the 2017 Internet Measurement Conference_ (London, United Kingdom) _(IMC ’17)_. Association for Computing Machinery, New York, NY, USA, 142–156.  https://doi.org/10.1145/3131365.3131397
  * Project and Atkinson (2019) GNU Project and Kevin Atkinson. 2019.  GNU Aspell - Free and Open Source spell checker.  http://aspell.net/.
  * PublicWWW ([n.d.]) PublicWWW. [n.d.].  Source Code Search Engine.  https://publicwww.com/.
  * Rivers and Lewis (2014) Caitlin M. Rivers and Bryan L. Lewis. 2014.  Ethical research standards in a world of big data.  _F1000Research_ 3 (2014), 38.  https://doi.org/10.12688/f1000research.3-38.v2
  * Rogers (2021) Richard Rogers. 2021\.  _Digital Forensics: Repurposing Google Analytics IDs_.  Amsterdam University Press, Amsterdam,The Netherlands, 241–245.  http://www.jstor.org/stable/j.ctv1qr6smr.36
  * Samson (2018) Celine Isabelle Samson. 2018\.  VERA FILES FACT CHECK YEARENDER: Ads reveal links between websites producing fake news.  https://www.verafiles.org/articles/vera-files-fact-check-yearender-ads-reveal-links-between-web.
  * Shannon (1948) Claude Elwood Shannon. 1948\.  A mathematical theory of communication.  _The Bell system technical journal_ 27, 3 (1948), 379–423.
  * Silverman et al. (2017) Craig Silverman, Jane Lytvynenko, Lam Thuy Vo, and Jeremy Singer-Vine. 2017.  Inside The Partisan Fight For Your News Feed.  https://www.buzzfeednews.com/article/craigsilverman/inside-the-partisan-fight-for-your-news-feed.
  * Simeonovski et al. (2017) Milivoj Simeonovski, Giancarlo Pellegrino, Christian Rossow, and Michael Backes. 2017.  Who Controls the Internet? Analyzing Global Threats Using Property Graph Traversals. In _Proceedings of the 26th International Conference on World Wide Web_ (Perth, Australia) _(WWW ’17)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 647–656.  https://doi.org/10.1145/3038912.3052587
  * Starov et al. (2018) Oleksii Starov, Yuchen Zhou, Xiao Zhang, Najmeh Miramirkhani, and Nick Nikiforakis. 2018.  Betrayed by Your Dashboard: Discovering Malicious Campaigns via Web Analytics. In _Proceedings of the 2018 World Wide Web Conference_ (Lyon, France) _(WWW ’18)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 227–236.  https://doi.org/10.1145/3178876.3186089
  * The Chromium Authors (2014) The Chromium Authors. 2014\.  Chrome DevTools Protocol.  https://chromedevtools.github.io/devtools-protocol/.
  * Tranco (2021) Tranco. 2021.  Tranco list with the 1M top sites generated on 14 April 2021.  https://tranco-list.eu/list/7JVX/full.
  * Yoon et al. (2019) Changhoon Yoon, Kwanwoo Kim, Yongdae Kim, Seungwon Shin, and Sooel Son. 2019.  DoppelgäNgers on the Dark Web: A Large-Scale Assessment on Phishing Hidden Web Services. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 2225–2235.  https://doi.org/10.1145/3308558.3313551

##  Appendix A Ethical Considerations

The execution of this work has followed the principles and guidelines of how
to perform ethical information research and the use of shared measurement data
(Dittrich and Kenneally, 2012; Rivers and Lewis, 2014). In particular, this
study paid attention to the following dimensions.

We keep our crawling to a minimum to ensure that we do not slow down or
deteriorate the performance of any web service in any way. Therefore, we crawl
only the landing page of each website and visit it only once. We do not
interact with any component in the visited website, and only passively observe
network traffic. In addition to this, our crawler has been implemented to wait
for both the website to fully load and an extra period of time before visiting
another website. Consequently, we emulate the behavior of a normal user that
stumbled upon a website.

In accordance to the GDPR and ePrivacy regulations, we did not engage in
collection of data from real users. Also, we do not share with any other
entity any data collected by our crawler. Our analysis is, to a large extent,
based on public historical data (e.g., HTTPArchive Project). Moreover, we
ensure that the privacy of publishers and administrators is not invaded. We do
not collect any of their information (e.g., email addresses) and only discuss
publishers, who explicitly and voluntarily disclose their identity in their
websites, as we did in Section 6. Last but not least, we intentionally do not
make our 1MT crawl dataset public, to ensure that there is no infringement of
copyrighted material.

##  Appendix B Detected Communities

Table 2. Communities detected using the proposed methodology. Through manual investigation we determine the legal entity behind these websites. Description | Size | Websites
---|---|---
MinuteMedia | 172 | showsnob.com, sodomojo.com,
|  | sportdfw.com, thejetpress.com,
|  | reignoftroy.com, 90min.de, …
Warner Music Group | 142 | brunomars.com, blakeshelton.com,
|  | greenday.com, vancejoy.com,
|  | paramore.net, disturbed1.com, …
Townsquare Media, Inc | 111 | wkdq.com, wgrd.com, wbkr.com,
|  | wbckfm.com, mix108.com, b985.fm,
|  | keyw.com, 929nin.com, …
Gray Media Group, Inc | 76 | wrdw.com, witn.com, whsv.com,
|  | wcax.com, wbay.com, nbc29.com,
|  | kwch.com, kktv.com, abc12.com,…
Australian Community Media | 73 | thesenior.com.au, nvi.com.au,
|  | theflindersnews.com.au,
|  | mailtimes.com.au, portnews.com.au, …
Postmedia Network Canada Corp | 58 | lfpress.com, nationalpost.com,
|  | windsorstar.com, winnipegsun.com,
|  | intelligencer.ca, coldlakesun.com, …
Philips | 45 | usa.philips.com, philips.com.br,
|  | philips.com.mx, philips.com.pk,
|  | philips.cz, philips.ru, philips.pl …

We manually examined communities and tried to determine the legal entity
operating or even owning all websites in each community. We report some of the
largest of these communities in Table 2 along with their size (i.e., number of
websites) and some indicative websites as examples.