# Detecting and Analyzing Collusive Entities on YouTube

Hridoy Sankar Dutta∗ IIIT-DelhiOkhla Industrial Estate, Phase IIINew
DelhiDelhiIndia110020 ,  Mayank Jobanputra∗ IIIT-DelhiOkhla Industrial Estate,
Phase IIINew DelhiDelhiIndia110020 ,  Himani Negi† BVCOE, New DelhiNew
DelhiIndia and  Tanmoy Chakraborty  IIIT-DelhiOkhla Industrial Estate, Phase
IIINew DelhiDelhi110020India

(2021)

###### Abstract.

YouTube sells advertisements on the posted videos, which in turn enables the
content creators to monetize their videos. As an unintended consequence, this
has proliferated various illegal activities such as artificially boosting of
views, likes, comments, and subscriptions. We refer to such videos (gaining
likes and comments artificially) and channels (gaining subscriptions
artificially) as “collusive entities”. Detecting such collusive entities is an
important yet challenging task. Existing solutions mostly deal with the
problem of spotting fake views, spam comments, fake content, etc., and
oftentimes ignore how such fake activities emerge via collusion. Here, we
collect a large dataset consisting of two types of collusive entities on
YouTube – videos submitted to gain collusive likes and comment requests, and
channels submitted to gain collusive subscriptions.

We begin by providing an in-depth analysis of collusive entities on YouTube
fostered by various blackmarket services. Following this, we propose models to
detect three types of collusive YouTube entities – videos seeking collusive
likes, channels seeking collusive subscriptions, and videos seeking collusive
comments. The third type of entity is associated with temporal information. To
detect videos and channels for collusive likes and subscriptions respectively,
we utilize one-class classifiers trained on our curated collusive entities and
a set of novel features. The SVM-based model shows significant performance
with a true positive rate of $0.911$ and $0.910$ for detecting collusive
videos and collusive channels respectively. To detect videos seeking collusive
comments, we propose CollATe, a novel end-to-end neural architecture that
leverages time-series information of posted comments along with static
metadata of videos. CollATe is composed of three components – metadata feature
extractor (which derives metadata-based features from videos), anomaly feature
extractor (which utilizes the time-series data to detect sudden changes in the
commenting activity), and comment feature extractor (which utilizes the text
of the comments posted during collusion and computes a similarity score
between the comments). Extensive experiments show the effectiveness of CollATe
(with a true positive rate of $0.905$) over the baselines.

YouTube, collusion, blackmarket, artificial boosting, OSNs

∗ Both authors contributed equally to the paper

† The work was done when Himani was an intern at IIIT Delhi.

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††journal: TIST††journalvolume: 37††journalnumber:
4††article: 111††publicationmonth: 8††ccs: Information systems Social
networks††ccs: Security and privacy

##  1\. Introduction

Online media is increasingly becoming the most effective medium for sharing
ideas, thoughts, and information. The primary reasons behind the large user
engagement are the ease of accessibility and ever-evolving attractive sharing
facility. The content shared in online media usually includes personal data,
documents, photos, and videos. Generally, videos are used to convey meaningful
information in a shorter duration, providing unparalleled advantages to
content consumers. YouTube, an online video-sharing platform, allows users to
upload, share, or live-stream videos on the Internet. Free service, measurable
analytics, availability of multiple genres, and access to broad audiences have
made YouTube the most popular platform for both content creators and content
consumers. The popularity of a YouTube video is generally determined by the
number of likes or comments it receives over a period of time. Similarly, the
popularity of a YouTube channel is measured by the number of subscribers. The
natural way to gain traffic to a channel/video and make it noticeable is a
time-consuming process for content creators. Apart from creating attractive
content, oftentimes, the content creators become desperate to figure out
effective shortcuts to gain quick popularity of their content and channels.
This may lead them to choose unethical and inorganic ways with the aid of
blackmarket services to gain popularity within a short duration. The primary
objective of an artificial boosting mechanism is to convince the YouTube
ranking algorithm to prefer a video/channel over its competitors.

In recent years, YouTube has started a YouTube Partner
Programme111https://support.google.com/youtube/answer/72851?hl=en where
content creators can make money from the advertisements and other revenue
streams. However, the channel has to meet minimum two requirements – at least
$1,000$ subscribers and at least $4,000$ hours of watch time within the past
12 months. Furthermore, content creators who are new to YouTube and have
managed to attract only a small audience do not get significant engagement
despite having fantastic content. It leads them to invest to the blackmarket
services to reach out to more audiences and increase their revenues in a
faster and effective way. The blackmarket-driven artificial engagement is
strictly against the YouTube Terms of Service and may lead to a permanent ban
on the user accounts. To counter this, blackmarket services provide their
facilities in such a way that collusive entities (channels and videos) are
evaded from being detected by the in-house fake detection algorithms deployed
by YouTube. Interestingly, in our dataset, we found $7$ such collusive
channels, which are marked as verified by YouTube! This clearly shows that
YouTube is unable to detect such fraudulent entities, thereby creating an
inadequate social space for the entire YouTube population.

Throughout the paper, we use the following nomenclature. A YouTube video is
said to be collusive if likes or comments of the video are artificially
inflated with the help of blackmarket services. Similarly, a collusive YouTube
channel is the one which receives artificial subscriptions from blackmarket
services. We use collusive entities to refer to both collusive videos and
channels.

The current work provides a large-scale investigation and detection of
collusive entities on YouTube. We create a unique dataset of collusive
entities collected from YouLikeHits (a credit-based freemium blackmarket
service). This, to our knowledge, is the first labeled dataset of YouTube
collusive entities. We start our analysis by examining videos submitted to
blackmarket services for collusive appraisals (likes and comments) based on
two perspectives – propagation dynamics and video metadata. We then analyze
the collusive channels based on location, channel metadata, and network
properties. In the collusive channel network, the structural properties of the
giant component show that it is a small-world. Further, we are interested in
detecting whether a new entity is collusive or not. Since we collected the
collusive data directly from the blackmarket websites, we are sure that the
collected data does not have any noisy labels. In such cases, instead of
collecting a large dataset representing the entire YouTube population, we
rather focus on designing sophisticated methods that can leverage the
characteristics of entities in one class (collusive in our case) and learn
their representations for the prediction task. Note that we were unable to
extract any temporal information related to like and subscription activities
for videos and channels respectively, due to the restrictions and limitations
of the YouTube API.

We propose three models to detect three types of collusive entities. The first
and the second models utilize one-class classifiers trained only on the
collusive entities using video metadata and channel features to detect
collusive videos and channels respectively. The third model attempts to detect
videos seeking collusive comments. In addition to the video metadata, it uses
textual and temporal information of each comment. The temporal information
indicates the aggressive patterns of blackmarket users (as these users
aggressively post comments on collusive YouTube videos in order to gain
credits). In addition to this, the semantic representation of each comment
enables us to learn the similarity pattern of these users in posting comments
and also to evade the existing fake detection strategies. During the
collection of this dataset, we realized that many videos have a limited number
of comments despite being posted for collusive comments to the blackmarket
services. We discarded such videos as the comments were possibly deleted,
which might result in noisy information in the labeled data. Finally, we end
up collecting a relatively smaller dataset of videos seeking collusive
comments compared to those videos/channels seeking collusive
likes/subscriptions. This further raises another challenge of learning
generalized and robust representation. Thus, by incorporating these properties
of blackmarket users and taking into account the issues mentioned above, we
propose CollATe, a denoising autoencoder model to detect videos submitted in
blackmarket services for collusive comments. CollATe consists of three
components – metadata feature extractor, anomaly feature extractor, and
comment feature extractor to learn feature representation of videos.

The first and second models achieve significant accuracy with a true positive
rate of $0.911$ and $0.910$ respectively. The third model, CollATe achieves a
true positive rate of $0.905$, outperforming seven baselines. Altogether this
work sheds light on how collusion happens on YouTube and focuses on the
detection of collusive entities to make YouTube an adequate video sharing
platform for the content creators and content consumers. We believe this could
be used to curb the adverse effects of collusion in gaining artificial social
growth. We summarize the contributions of this work as follows:

  * •

To the best of our knowledge, we are the first to investigate YouTube videos
and channels submitted to blackmarket services for collusive appraisals.

  * •

We prepare four unique datasets of collusive entities on YouTube: (i) videos
submitted to YouTube for collusive likes, (ii) videos submitted to YouTube for
collusive comments, (iii) channels submitted to YouTube for collusive
subscriptions, and (iv) the network of collusive YouTube channels. The dataset
is attained using YouTube API and custom-designed scrapers that can be easily
extended and used to collect large YouTube data in an effective way. We
believe that these four datasets would help researchers analyze blackmarket-
driven collusive activities happening on YouTube and develop tools to detect
them.

  * •

We analyze the YouTube videos for collusive likes and comments from two
perspectives – propagation dynamics and video metadata. We also analyze
collusive YouTube channels based on their location, channel metadata, and
network properties. The giant component of the collusive channel network turns
out to be a small-world.

  * •

We utilize one-class classification models to detect videos and channels
submitted to blackmarkets for collusive likes and subscriptions. We also
propose CollATe, an end-to-end neural framework to detect YouTube videos
seeking collusive comments.

Reproducibility: Codes and datasets are available at the following link:
https://github.com/LCS2-IIITD/CollATe.

Organization of the paper: The remainder of the paper is organized as follows.
Section 2 discusses related work. Section 3 presents a detailed study of the
blackmarket services and motivates the research problem. Section 4 discusses
the data collection strategy. Section 5 analyzes collusive YouTube videos and
channels. Section 6 introduces the proposed frameworks to detect collusive
entities. The experimental results are presented in Section 7. Section 8 shows
the important implications of the collusive entities detected using our
models. Section 9 concludes the paper with future directions.

##  2\. Related Work

We divide the relevant related work into two parts: (i) fraud/spam detection
in online media, and (ii) studies on blackmarket services.

###  2.1. Fraud/spam detection in online media

Plenty of studies focused on the detection of fraudulent activities in a wide
range of platforms. In recent years, fraudulent activities are most common in
major online media sites such as Facebook (De Cristofaro et al., 2014; Badri
Satya et al., 2016; Ikram et al., 2017), Instagram (Sen et al., 2018; Zhang
and Sun, 2017) and Twitter (Egele et al., 2015; Gupta et al., 2013; Stringhini
et al., 2015; Ghosh et al., 2012; Thomas et al., 2011; Wang, 2010; Nilizadeh
et al., 2017; Egele et al., 2013). Kumar and Shah (2018) presented a review on
existing fake and fraud detection strategies in online media platforms. A
number of fake comment detection strategies (Alsaleh et al., 2015; Uysal,
2018) on YouTube were also proposed in recent years. Most of these studies
rely solely on the textual data of the comment. Nevertheless, there is no
prior work that considers the temporal properties of comments, which in the
case of collusion, is the most important factor, the reason being collusive
users perform appraisal operations aggressively in order to gain credits
rapidly. Li et al. (2016) proposed LEAS, an algorithm to detect fake
engagements in video sharing platforms using a temporal engagement graph
between users and video objects. Marciel et al. (2016) proposed a set of tools
to detect view fraud in online video portals. Chen et al. (2015) investigated
the problem of fake views caused by robots in video sharing platforms.

In the field of fraud detection in online advertising, Metwally et al. (2007)
proposed an advertising network model to discover coalitions between pairs of
fraudsters in e-commerce platforms. Dave et al. (2012) designed an automated
approach for ad networks to detect click-spam attacks. Hussain et al. (2018)
analyzed disinformation and crowd manipulation tactics on YouTube by analyzing
video metadata. Alassad et al. (2019) examined intensive groups among YouTube
commenter networks by constructing a two-level optimization problem for
maximizing local degree centrality and global modularity measures. Faddoul et
al. (2020) conducted a longitudinal analysis of the promotion of conspiracy
videos on YouTube. Yang et al. (2017) studied the problem of injection attacks
on the recommendation systems (fake co-visitations) of YouTube. A number of
studies have been conducted on detecting spam on video streaming platforms.
Alberto et al. (2015) proposed Tubespam, a novel classification model for
comment spam filtering on YouTube. Uysal (2018) studied the performance of
five state-of-the-art text feature selection methods for spam filtering on
YouTube using Naive Bayes and Decision Tree. Yusof and Sadoon (2017) detected
video spammers on YouTube based on the EdgeRank algorithm to decide which
post/stories should appear in each user’s news feed. Chowdury et al. (2013)
proposed a spam detection system for YouTube using a set of spam-related
attributes from videos. Sureka (2011) detected forum spammers on YouTube based
on the mining comment activity log of a user and extracting patterns
indicating spam behavior. Aiyar and Shetty (2018) detected spam comments on
YouTube by showing the effectiveness of using character n-grams instead of
word n-grams to improve the accuracy of the classification model.

More recently, YouTube has become the most important tool for live streamers.
In our dataset, we found around $30\%$ of the collusive channels involved in
streaming live videos. Zhang et al. (2018b); Zhang et al. (2018a) discussed
the detection of copyright infringement on YouTube live videos. Zhang et al.
(2018b) developed a crowd-sourced based copyright infringement detection
(CCID) scheme from live chat messages on YouTube and Twitch to identify
original copyright content from the owner. Zhang et al. (2018a) presented an
end-to-end supervised detection framework to combat copyright infringement in
live video streams using the live chat messages from the audiences. However,
these methods tend to combat the problem of fake, fraud, and spam detection in
video-sharing platforms but are not applicable for collusive entity detection.

###  2.2. Studies on blackmarket services

Despite the fact that a plethora of studies exist on detecting fraud/spam
activities in online media, there has been relatively less work on
investigating blackmarket services providing collusive appraisals. Acker
(2018) focused on how manipulators create disinformation by fake engagement
activities on YouTube. Keller (2018) provided a broad overview of the
blackmarket services providing fake YouTube views. The authors reported that
one of the blackmarket services, named Devumi had earned more than $\$1.2$
million in around $3$ years of service by selling $196$ million views.

Studies that are more related to the current work include Shah et al. (2017),
Dutta et al. (2018), and Chetan et al. (2019), which investigated the problem
of blackmarket-based collusive activities in Twitter. Shah et al. (2017)
studied multiple types of blackmarket link fraud behaviors in Twitter by
analyzing the connectivity patterns of fake followers via the egonet and
boomerang networks. Dutta et al. (2018) proposed SCoRe, a supervised method to
detect collusive retweeters affiliated to blackmarket services in Twitter. The
authors also showed the differences between fake and collusive activities
based on the synchronicity of retweet behaviors. Dutta and Chakraborty (2019)
extended their previous work (Dutta et al., 2018) to show the differences
between the working principles of premium and freemium blackmarket services.
Chetan et al. (2019) proposed CoReRank, an unsupervised method to detect
collusive users and suspicious tweets by leveraging the user’s retweeting and
quoting patterns. Dutta et al. (2020) proposed HawkesEye, a framework to
detect fake retweeters using Hawkes process and topic modeling on tweets.
Arora et al. (2019) proposed a multi-task learning approach to detect tweets
submitted to freemium blackmarket services. Dhawan et al. (2019) proposed
DeFrauder, an unsupervised method to spot online fraudulent collusive groups
in review websites. Wang et al. (2011) proposed a review graph model to detect
spammers in online review stores. Zhu et al. (2016) proposed an automated
approach to detect collusion behavior in online question-answering systems.
Other studies identified fake followers on Twitter (Castellini et al., 2017;
Cresci et al., 2015; Aggarwal and Kumaraguru, 2014; Stringhini et al., 2013;
Dutta et al., 2021). Mehrotra et al. (2016) and Jiang et al. (2014) are some
of those who used network-centric properties to detect fake followers. Fake
Follower Check222https://tinyurl.com/y29t3uuu is one such tool to detect fake
followers based on profile-centric and behavioral features of Twitter users.
We encourage the reader to go through (Dutta and Chakraborty, 2020) for a
detailed survey of collusion in online media platforms.

To the best of our knowledge, there has been no major attempt to detect
collusive entities on video sharing platforms such as YouTube. Our current
effort provides a deeper understanding of the collusive activities on YouTube
and focuses on designing automated approaches to detect these activities.

##  3\. Background and Motivation

###  3.1. Blackmarket services

Blackmarket services help online media users in gaining appraisals
inorganically for their content. They offer services related to online social
networks (Facebook: like, follow, share; Instagram: like, follow; Twitter:
retweet, like, follow), recruitment platforms (LinkedIn: endorsement,
recommendation, connection), video sharing platforms (YouTube: video views,
video comments, channel subscription; Vimeo: video views, video comments),
etc. The inorganic appraisals help in artificial boosting of online media
content, thereby creating an inadequate social space. Online media entities
such as big companies, advertising firms seeking active participation in their
promotional campaigns target these websites to expedite their reach to their
target audiences.

The blackmarket services are divided into two types based on the mode of
service (Malhotra et al., 2015):

  * •

Premium services: These services charge customers for the facilities they
provide. Customers have to register themselves and opt for one of their plans
to gain appraisals.

  * •

Freemium services: These services are free of cost and work like a barter
system. The primary goal of freemium services is to let their customers
familiarize with free services and convince them to subscribe to the premium
plans. Most of the freemium services are credit-based, where each customer
receives virtual credits by appraising the content of other customers.

In this work, we focus our analysis on
YouLikeHits333https://www.youlikehits.com/, a credit-based freemium service
which helps content creators on YouTube to boost their subscribers, views,
comments and likes.

###  3.2. Collusion on YouTube platform

YouTube is a video sharing platform where users can upload videos by creating
YouTube channels. The platform is free of cost and is operated by two types of
users:
(i) Content creators: Users who upload videos to their channel.
(ii) Content consumers: Users who watch videos, interact with videos in the
form of likes/comments, or subscribe to channels.

When a content creator uploads a video, content consumer can perform the
following actions –like the video, dislike the video, comment on the video,
share the video, save the video to wishlist, and subscribe the content
creator’s channel. The popularity of a YouTube video is measured by the number
of appraisals it receives from the content consumers. Thus the content
creators need to ensure that their content receives high appraisals from the
consumers. Moreover, with the advent of the concept of Monetization on
YouTube444https://www.youtube.com/account_monetization?nv=1, content creators
have begun to attract audiences by uploading videos aimed towards specific
genres such as teaching, entertainment, business, etc. This may further
motivate them to choose artificial ways to gain quick popularity in their
content. Currently, the earning potential of a channel/video is solely driven
by the number of subscribers/views. When a video is posted on YouTube, it is
shared with the YouTube community of similar channels as recommendations. With
millions of videos posted every second, a majority of the videos go unnoticed
to the target audience. The organic way of gaining appraisals is a tedious
task, which leads content creators to opt for an alternative way by means of
blackmarket services.

##### How collusion happens on YouTube?

Collusion on YouTube happens when a video or a channel is posted in
blackmarket services for appraisals. In this work, we refer to a user who
submits the content to the blackmarket service as a collusive user. In a
freemium blackmarket service, collusive users receive credit points upon
performing appraisals on the content of other collusive users. In the case of
YouTube, the majority of the blackmarket services request the collusive users
to submit only the video or channel URL. Collusive users can contribute to the
artificial boosting of YouTube videos and channels in several ways: (i)
viewing other videos, (ii) posting a comment on videos555Note that blackmarket
services may ban a collusive user upon identifying a spammy comment posted by
the user., (iii) posting likes on a video, and (iv) subscribing to a channel.

Figure 1. An example of anomalous pattern in videos detected by CollATe for
collusive (in red) and other videos (in green). The x-axis displays the time
span (in days) starting from when the first comment was posted and y-axis
determines an anomaly score calculated by CollATe. Here, the anomaly score is
the Mahalanobis distance computed using Equation 2 as mentioned in Section
6.3.1. The peak width (horizontal black line) corresponding to every peak
indicates the duration of the peak.

Fig. 1 shows the peaks observed in collusive videos (in red) for collusive
comment appraisals detected using our proposed approach, CollATe. However, we
do not see any such peaks in other random videos (more details can be found in
Section 6.3.3). The reason is that the collusive users tend to perform
collusive appraisals aggressively to gain credit points, which they can use
later to add new content. This aggressive nature results in peaks in the
inter-arrival time of appraisals of the collusive entities; however, normal
users do not exhibit such behavior.

##### Why only YouTube?

In this article, our focus is to detect collusive entities only on YouTube.
The primary reason of conducting this study in one platform is due to the
challenge in collecting a large dataset of videos and channels hosted on other
platforms and submitted to blackmarket services for collusive appraisals.
While conducting this study, we checked a few freemium blackmarket services
which provide collusive appraisals to video-sharing platforms such as Twitch
and Vimeo. During the collection of information from the blackmarket services,
we observed that a very less data is available for these platforms.
Additionally, unlike YouTube, the APIs of these platforms have certain
limitations and do not provide enough public information which can be used for
detailed analysis of the collected entities.

##### Challenges in collusive entity detection.

Detecting collusive entities is a difficult task (Dutta et al., 2018; Dutta
and Chakraborty, 2019). There exist blackmarket services which have created
intelligent mechanisms to produce collusive appraisals. We list down the
unique challenges of the collusive entity detection task below:

  1. (1)

Collusive accounts are not full-time bots, spam-accounts or fake accounts used
only for appraisal. Thus, existing studies focusing on bot, spam and fake
account detection are not able to capture their behavior as shown in (Dutta et
al., 2018; Dutta and Chakraborty, 2019).

  2. (2)

Collusive accounts do not show a completely genuine behavior. Some of their
appraised content would be endorsed not because of their interest in the
content, but merely to comply with the barter system in blackmarket services.
This implies that collusive users show an amalgamation of both organic and
inorganic behavior – being genuine users, they organically appraise the
content of their interest; being a blackmarket members, they also
inorganically appraise the content posted in blackmarket services.

  3. (3)

Limited contextual information is available on short texts present in
collusive entities such as comments, replies, etc., which makes it difficult
for deep neural networks to generate appropriate representations.

##  4\. Dataset description

###  4.1. Data collection

The major challenge is to collect a large set of YouTube videos and channels
submitted to the blackmarket services for collusive appraisals and a
contrasting set of videos. We started our data collection by designing
multiple web scrapers for the following purposes – (i) scraping data from
blackmarket website (YouLikeHits) and (ii) scraping video related data (i.e.,
description, comments) of YouTube videos. Both the scrapers performed their
operation independently. We used
Joblib666https://joblib.readthedocs.io/en/latest/ library to utilize multiple
cores in the deployed server. We also extensively used YouTube
API777https://developers.google.com/youtube/v3/ for the following purposes –
(i) extracting subscribers of YouTube channels, and (ii) extracting the exact
time at which the comments are posted.

We collected the information of collusive videos from YouLikeHits, a
blackmarket service that provides three types of collusive appraisals – (i)
likes to YouTube videos, (ii) comments to YouTube videos, and (iii)
subscriptions to YouTube channels. We queried multiple search engines with
keywords such as ‘free YouTube likes’, ‘free YouTube comments’. Interestingly,
apart from links to the websites providing artificial YouTube views and
subscriptions, we found a large number of blackmarket websites pointing to
other online media platforms such as Twitter, Facebook, Instagram. It
indicates the popularity of blackmarket websites to achieve artificial social
status in a much rapid way among online media users. We developed a scraper to
retrieve YouTube entities (videos/channels) involved in collusive appraisals
on YouLikeHits. Interestingly, we observe that some of the collusive videos
are propagated by verified YouTube channels. We found $342$ (resp. $86$)
videos submitted to YouLikeHits for collusive likes (resp. comments). We also
found $7$ verified YouTube channels, which are registered in YouLikeHits for
collusive subscriptions. While scraping, we found that only $25\%$ of videos
for collusive likes and $4.05\%$ of videos for collusive comments are deleted
by YouTube’s current fraud detection system. All the above observations show
that YouTube is unable to detect these entities effectively using its in-house
fraud detection mechanism. These insights further motivated us to design
efficient methods to detect YouTube entities that are involved in gaining
collusive appraisals with the help of blackmarket services.

We extracted video metadata and comments of all the videos using our custom-
designed scrapers and YouTube API. Finally, we divided the dataset into three
unique sets – $\boldsymbol{V_{l}}$ (videos submitted to YouLikeHits for
collusive likes), $\boldsymbol{V_{c}}$ (videos submitted to YouLikeHits for
collusive comments) and $\boldsymbol{V_{s}}$ (channels submitted to
YouLikeHits for collusive subscriptions). The entire data statistics is showed
in Table 1 and Fig. 2. Note that we did not find any videos which are common
across $\boldsymbol{V_{l}}$ and $\boldsymbol{V_{c}}$.

Table 1. Summary of the dataset. Here, the column # unique CC refers to the
number of unique content creators of videos submitted for collusive likes and
comments. Entities refers to videos or channels; actions refers to
like/comment for videos and subscription for channels.

Type | # entities | # deleted | # verified | # unique CC | Max actions | Min actions | Avg. actions
---|---|---|---|---|---|---|---
$\boldsymbol{V_{l}}$ | $45572$ | $15662$ | $342$ | $28702$ | $3151770$ | $0$ | $1333$
$\boldsymbol{V_{c}}$ | $25106$ | $1060$ | $86$ | $11752$ | $1008428$ | $0$ | $120$
$\boldsymbol{V_{s}}$ | $7847$ | $0$ | $7$ | $-$ | $12935205$ | $0$ | $5378$

(a)

(b)

(c)

Figure 2. Distribution of (a) likes of YouTube videos, (b) comments of YouTube
videos, and (c) subscribers of YouTube channels in our dataset.

###  4.2. Data privacy

To our knowledge, this work is the first effort that aims to analyze YouTube
videos and channels submitted to blackmarket services for collusive
appraisals. We emphasize that we will not release the sensitive information
(i.e., video ids and uploader details) when we make the dataset public. The
entire data collection process was performed after taking proper Institutional
Review Board (IRB) approval from our institute.

##  5\. Analyzing collusive YouTube entities

We analyze the collusive videos from two aspects - (i) propagation dynamics,
and (ii) video metadata. In the first part of this section, we will present
the propagation dynamics of collusive videos using two metrics – initial burst
and lifetimes. In the second part, we will show the analysis of the collusive
YouTube channels based on location, channel metadata and network structure.

###  5.1. Videos submitted for collusive comments/likes

For videos submitted for collusive comments, we extract the video metadata and
video comments (full text of each comment and timestamp at which the comment
was posted). Note that due to the restrictions of YouTube API, we are unable
to provide detailed insights of the videos submitted to blackmarket websites
for collusive likes. The API only allows retrieving the total count of likes
and dislikes.

(a)

(b)

Figure 3. Distribution of temporal properties characterizing propagation
dynamics – (a) initial delays and (b) lifetimes of Youtube collusive videos.

####  5.1.1. Propagation dynamics of artificial boosting

Here we focus on the temporal properties of collusive videos based on two
features – initial burst and lifetimes. As the timestamps of occurrence of
like activities are not available with us due to API restrictions, we perform
the analysis only on videos submitted to blackmarkets for collusive comments.
(i) Initial burst: We consider the initial burst as the first time when there
is a peak in the arrival rate of comments in a video. Section 6.3.1 outlines
the peak detection technique. Through this analysis, we characterize how
rapidly a video receives collusive appraisals. Fig. 3(a) shows that around
$30\%$ of the videos have an initial burst of artificial boosting (first peak)
within 7 days, and around $50\%$ of the videos have an initial burst within
the first one month from the date of posting.
(ii) Lifetimes: Here we consider the lifetime of the collusive activity over a
video. We calculate the delay between the burst of the first peak and the fall
of the last peak. Fig. 3(b) shows that about $80\%$ of the videos have
lifetimes within $7$ days. This illustrates how these videos gain rapid
attention through blackmarket services.

####  5.1.2. Metadata of collusive videos

We now analyze the metadata of collusive videos. We study the genres of the
videos and the titles of videos submitted to blackmarket for collusive
appraisals.
(i) Video genres: We observe the distribution of genre of videos submitted to
blackmarket services. The common genres on YouTube are – Gaming (GA),
Entertainment (EN), Travel & Events (TE), Film & Animation (FA), Music (MU),
People & Blogs (PB), Autos & Vehicles (AV), Education (ED), Comedy (CO), News
& Politics (NP), Howto & Style (HS), Science & Technology (ST), Sports (SP),
Pets & Animals (PA) and Nonprofits & Activism (NA). We plot the genre-wise
distribution of likes, dislikes and comments of collusive videos in Fig. 4.
The bars for likes and dislikes are drawn from the video metadata for
collusive likes, and bars for comments are drawn from the video metadata for
collusive comments. As expected, we observe ‘Music’ to be the most popular
genre for videos submitted for collusive likes ($63.05\%$) and comments
($69.81\%$).

Figure 4. Genre-wise distribution of likes, dislikes and comments of collusive
videos. Full forms of the labels in the x-axis are mentioned in Section 5.1.2.

(ii) Wordcloud of video title: We show the wordcloud generated from the title
of the videos submitted to blackmarkets for collusive likes and comments in
Fig. 5. For clarity, we remove the two-letter words and common stopwords. Here
the font size corresponds to the frequency of the text. We clearly observe the
presence of similar keywords such as promotional keywords like ‘free’, ‘best’,
‘top’, etc. in both the cases. With the presence of these keywords, it is
evident that videos for collusive like/comment appraisals focus on target-
specific keywords for quick promotion.

(iii) Uploader authenticity: We also study the authenticity of the uploader of
videos for collusive comments and likes. Surprisingly, we observe that
verified users (marked by YouTube) are also involved in gaining collusive
appraisals via blackmarket services (see Table 1).

(a)

(b)

Figure 5. Wordcloud of titles of videos submitted for (a) collusive comments
and (b) collusive likes.

###  5.2. Channels submitted for collusive subscription requests

In our analysis thus far, we focused only on YouTube video submitted to
blackmarket services for collusive likes and comments. In this section, we
examine YouTube channels submitted to blackmarket services for collusive
subscriptions.
(i) Country-wise distribution: Fig. 6(a) shows the world map plot for the
country-wise distribution of collusive YouTube channels (in red color),
indicating the count of channels submitted to blackmarket services for
collusive subscriptions. Out of $7,847$ collusive channels, we find $3,804$
channels with no countries mentioned in the profile. In the remaining set, USA
tops the list with $20.11\%$ of collusive YouTube channels followed by
Indonesia with $19.58\%$.
(ii) Metadata of collusive YouTube channels: We show the video, view and
subscriber count of collusive YouTube channels. For better visualization of
the distributions, we create custom range of the counts – Low ($1<count<100$),
Medium ($100<=count<=1000$) and High ($>1000$). Fig. 6(b) shows the
distribution of video, subscriber and view count for YouTube channels
submitted for collusive subscriptions. We observe that video count and
subscriber count are comparable across all ranges; however, there are too many
channels with high view count registered in blackmarkets for collusive
subscriptions. Fig. 6(c) shows the wordcloud aggregated over channel titles.
We eliminate common stop words and two-letter word. Here also, the font size
corresponds to the frequency of the text. We observe that collusive channels
have keywords such as ‘game’, ‘tutorial’, ‘music’, etc., corresponding to
personal interests, which help them improve the outreach by connecting with
the viewers and subscribers within the same domains.

(a)

(b)

(c)

Figure 6. (a) Country-wise distribution of collusive YouTube channels, (b)
distribution of YouTube channels based on video, subscriber and views, and (c)
wordcloud of channel titles.

(iii) Network observations: We study the network structure of collusive
channels using social network analysis tools. First, we create an undirected
network – nodes are the collusive YouTube channels, and an edge represents
common subscribers among two channels. We use the YouTube API to collect the
subscribers of each channel. Out of $7,847$ channels, we observe only $2,721$
channels with public subscriptions (i.e., end-users can see what channels they
are subscribed to). This forms the node-set in the network. We obtain $2,650$
edges among these nodes. We obtain $168$ different connected components from
this graph. We consider the final graph to be the maximum connected component
(giant component) with $1,320$ nodes and $1,396$ edges for our analysis. We
show various network properties (see Table 2) to better understand the
structure of the giant component. We notice that a large fraction of nodes
($48.51\%$) belong to the largest connected component, which perhaps indicates
the barter system in freemium services. The average shortest path length of
the network is $9.037$, which is of the order of the comparable random network
of the same size and average degree. Note that the comparable random network
is created using the same number of nodes and edges as that of the collusive
YouTube channel graph but with random edge connections. The average clustering
coefficient of the giant component in the collusive channel network is
$0.0023$, which is an order of magnitude higher than that of the comparable
random network whose clustering coefficient is $0.0016$. These two structural
properties of the giant component indicate that the network of collusive
YouTube channels is a small-world (Bagler, 2008).

Table 2. Statistics of collusive YouTube channel network.

# nodes | # edges | AD* | Diameter | APL** | Density
---|---|---|---|---|---
$1320$ | $1396$ | $2.115$ | $19$ | $9.037$ | $0.0016$

  * •

* - Average degree. ** - Average path length.

Table 3. Notations and denotations.

Notation | Denotation
---|---
$TS$ | Time sequence vector of comments
$c^{(i)}$ | Cumulative comment count at each timestamp
$e$ | Error vector
$\mathcal{N}(\mu,\eta)$ | Gaussian distribution using the error vectors
$a(c)$ | Anomaly score calculated using Mahalanobis distance
$\eta$ | Comment similarity score
$P$ | Number of peaks
W | Number of windows
$q$ | Query comment
$C$ | Window comments
$g(q)$ | Embedding for the query comments
$g(C)$ | Embedding for the window comments
$x\in X$ | Input data point $x$ in input space $X$
$V$ | Intermediate decoded space
$Z$ | Intermediate encoding
$y\in Y$ | Output label $y$ in labels space $Y$
$\hat{x}$ | Noisy or corrupted input data point
$\tau$ | Encoder present in the autoencoder
$\psi$ | Decoder present in the autoencoder

Table 4. Abbreviations used throughout the paper.

Abbreviation | Description
---|---
$MFE$ | Metadata Feature Extractor
$AFE$ | Anomaly Feature Extractor
$CFE$ | Comment Feature Extractor
$DAC$ | Denoising Autoencoder Classifier
$ARIMA$ | Auto Regressive Integrated Moving Average
$WMD$ | Word Mover Distance

##  6\. Detecting collusive entities

In the previous section, we have discussed how collusive videos and channels
have an adverse effect on YouTube social space. Therefore, the question we
would like to answer is – can we automatically detect if an entity on YouTube
is submitted for collusive appraisals?

In this section, we discuss the methodology for the detection of videos and
channels submitted to blackmarket services for collusive likes, comments, and
subscriptions. First, we will show how we detect videos for collusive likes,
and channels for collusive subscriptions. We pose this problem as a one-class
classification problem as the set of genuine videos is unknown to us888One can
argue that YouTube verified channels/videos might constitute the genuine set.
We did not consider them because of the following reasons. (i) In our dataset,
we found 7 verified channels which were submitted to the blackmarkets.
Therefore, we were unsure whether all verified entities are really genuine or
not. (ii) Verified channels/videos are usually very popular, receiving huge
appraisals from the viewers. Therefore, they may not look like a normal,
random YouTube entity and may lead to bias in the dataset.. We utilize several
one-class classifiers with the proposed sets of static features for the
detection. Secondly, we develop a novel end-to-end framework, named CollATe
that leverages video metadata, anomalous activities and comment similarity for
detecting videos for collusive comments. Table 4 summarizes the notations with
their denotations and Table 4 summarizes the abbreviation and description used
in the model.

##### Reason behind proposing three collusive entity detection tasks.

YouTube has three types of appraisals: likes, comments, subscriptions for two
types of entities: videos and channels. Likes and comments are for videos, and
subscriptions are for channels. In case of collusion, the blackmarket services
provide appraisals in such a way that one kind of appraisal is not related to
other e.g., if a customer requests for likes on a YouTube video, the video
will only receive likes and not any other appraisal. This is also seen in case
of Twitter (Dutta and Chakraborty, 2019) where there are two types of
appraisals, followers and retweets, for two types of entities, users and
tweets, respectively. As each appraisal is independent of other appraisals, we
proposed three detection tasks and did not combine them to just one task.

##### Reason behind proposing two one-class models and one binary
classification model.

The reason behind not framing the first two models as a binary classification
task is the unavailability of textual and temporal information for these two
models. In literature, it has been seen that collusive accounts show a hybrid
behavior – they sometimes behave like a genuine accounts and exhibit organic
activities; at the same time, they inorganically appraise the content of other
accounts to gain credits from blackmarket services. This made us to
hypothesize that collusive accounts tend to have very diverse topics of
interest as they appraise content without their genuine interest (textual
information) and show sudden changes in the commenting activity (temporal
information). In our case, the appraisal present in the third task (i.e.,
collusive comments for YouTube videos) is an interactive-based appraisal where
a user posts a textual comment on a specific time for a video. However,
appraisals present in the first two tasks, i.e., collusive likes for YouTube
videos and collusive subscriptions for YouTube channels, are tap-based
appraisals where a user simply taps on the like/subscribe button. The only way
to collect temporal information for these appraisals is using YouTube Live
Streaming API which has a quota limitation and makes it infeasible in a real-
time scenario where data comes in streaming fashion. Moreover, tap-based
appraisals do not contain any textual information. Thus, we proposed the first
two models as one-class classification models and the last model as a binary
classification model.

###  6.1. Features for detecting videos submitted for collusive likes

In this section, we present novel features to characterize videos submitted to
blackmarket services for collusive like requests. In Section 7.2, we show how
these features affect the performance of different one-class classification
models.

$\bullet$ Activeness ($\alpha$): We observe that collusive videos generally
receive more likes with a relatively small number of views. We compute the
Activeness as the ratio of the total number of likes to the total number of
views gained by the video. It shows the high engagement of content consumer on
the basis of likes and views. Fig. 7(a) shows the distribution of activeness
for collusive videos.

$\bullet$ Favorability ($\beta$): We notice that collusive videos receive less
dislikes even for a large number of views999Note that the number of dislikes
is not complementary of the number of likes, as users who view a video may not
make any action (like/dislike). Therefore, it is not necessary that the sum of
likes and dislikes of a video is the number of views.. We compute the
Favorability as the ratio of dislikes to the sum of likes and dislikes gained
by the video. It shows the likelihood of likes gained by the video. Fig. 7(b)
shows the distribution of favorability for collusive videos.

$\bullet$ View-rate ($\gamma$): The collusive users may focus on increasing
the likes of the videos. However, with the growth in likes, view count will
automatically increase (the reverse is not true though). This unintended
growth in views also helps the content creator to gain complementary collusive
views. We compute the view-rate as the ratio of the total number of views
gained by the video to the total number of days since its submission to
YouTubey. Fig. 7(c) shows the distribution of view-rate for collusive videos.

$\bullet$ Video duration ($\delta$): We observe that YouTube videos for
collusive comment requests are fairly short length videos with an average
duration of $6$ minutes. We consider the length of the video (in seconds) as
one of the features for the collusive entity detection. Fig. 7(d) shows the
distribution of video duration for collusive videos.

The final metadata feature extractor $v_{e}$ has the following form:

(1) $v_{e}=(\alpha,\beta,\gamma,\delta)$

(a)

(b)

(c)

(d)

Figure 7. Exploratory analysis of features used to characterise videos
submitted for collusive likes.

###  6.2. Features for detecting channels submitted for collusive
subscriptions

In this section, we present the novel features to characterize channels
submitted to blackmarket services for collusive subscriptions. Note that due
to YouTube API restrictions, we are unable to collect granular details about
the YouTube channels.

$\bullet$ Hidden subscriber count: It is the number of subscribers the channel
receives which are hidden from its profile.

$\bullet$ Video count: It is the count of the total number of videos present
in the channel.

$\bullet$ Subscriber count: It is the count of the total number of subscribers
the channel has received.

$\bullet$ View count: It is the total number of views the channel has
received.

$\bullet$ Comment count: It is the total number of comment the channel has
received.

###  6.3. Detecting videos submitted for collusive comments

In this section, we present novel features to characterize videos submitted to
blackmarket services for collusive comments. In Section 7.2, we will show the
importance of these features. It is known that not all the comments posted on
a video are collusive in nature. This makes it hard to identify collusive
comments even for human experts. Our dataset contains rich social information
about YouTube videos such as uploader details, video metadata, channel
details, raw comments text, and comment timestamp. The goal of our proposed
model is to transform the data into useful features and identify the collusive
videos.

CollATe comprises three components: (i) metadata feature extractor ($v_{m}$),
(ii) anomaly feature extractor ($v_{a}$), and (iii) comment feature extractor
($v_{c}$) as shown in Fig. 8. Once $v_{m}$, $v_{a}$ and $v_{c}$ are derived,
they are concatenated to create the final video representation. The collusive
video detector takes the learned representations as input to predict if a
video is collusive. The details of the metadata feature extractor is mentioned
in Section 6.1 (we use the same metadata feature extractor except view-rate
($\gamma$)). The reason behind not choosing view-rate is that a popular
YouTube video is likely to have similar view-rate to the collusive one. Two
other components are mentioned below.

Figure 8. The architecture of CollATe. The green colored network is the
metadata feature extractor, the orange colored network represents the anomaly
feature extractor, and comment feature extractor is blue colored.

####  6.3.1. Anomaly feature extractor

The goal of the anomaly feature extractor is to detect sudden changes in the
commenting activity over a video. Such activities generally continue for a
shorter time duration and then stop suddenly. We would like to detect such
activities for a given video to derive useful features. The anomaly feature
extractor takes sequential time-series data of comments as input. To extract
the anomalies (peaks) from the time-series data, we employ Gated Recurrent
Units (GRU) as the core module of the anomaly feature extractor. We adapt the
idea of stacking recurrent layers for anomaly detection from Malhotra et. al
(Malhotra et al., 2015).

Overview: Consider a time sequence vector $TS=\\{c^{(1)},c^{(2)}\allowbreak
c^{(3)},\dots c^{(n)}\\}$ where every point $c^{(t)}\in R^{m}$ in the time
sequence is a $m$-dimensional vector $\\{c^{(t)}_{1},c^{(t)}_{2},\allowbreak
c^{(t)}_{3},\dots c^{(t)}_{m}\\}$, denoting the cumulative comment count at
each timestamp for a given video. Using the time sequence vector $TS$, we
train a stacked GRU network. This model learns to predict the next $l$ values
for $d$ of the input variables such that $1\leq d\leq m$. We take $m$ units in
the input layer and $d\times l$ units in the output layer. The hidden layer
GRU units are fully connected through recurrent connections. We stack the GRU
layers in a manner that every unit in the lower GRU hidden layer is fully
connected with every unit in the hidden layer above it through feed-forward
connections.

We utilize the predicted values for computing the prediction error
distribution using which we detect the unusual comment activities (peaks). The
error vector $e^{(t)}$ is defined as $e^{(t)}=[e^{(t)}_{11},\dots,\allowbreak
e^{(t)}_{1l},\dots,e^{(t)}_{d1},\dots,e^{(t)}_{dl}]$, where $e^{(t)}_{ij}$ is
computed by taking the difference between the actual value of $c^{(t)}_{i}$
and its predicted value at time $t-j$. We fit a multivariate Gaussian
distribution $\mathcal{N}(\mu,\Sigma)$ using the error vectors, where $\mu$
and $\Sigma$ are estimated using Maximum Likelihood Estimation.

To label the observation as either collusive or non-collusive, we calculate
the anomaly score $a(c)$, which is defined as the Mahalanobis distance between
the computed error vector $e^{(t)}$ and the distribution $\mathcal{N}$.

(2)
$a(c)=(c-\mu)^{T}\Sigma^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(c-\mu)$

Finally, we concatenate all the observations for a given video time sequence
and feed the concatenated sequence to the find_peaks module of
Scipy101010https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.find_peaks.html
to retrieve the peak width and peak height. Using the same data, we get two
useful features that we pass into the collusive video detector.

(i) Peak count ($\phi$): It is the count of the of peaks detected for a video.

(ii) Average peak area ($\omega$): The average peak area is calculated as the
average of the overall area covered by each peak. After exploratory data
analysis, we observe a very clear distinction between the distribution of peak
width and height of collusive and other random videos. The width and height of
each peak is calculated using the peak_width module of
Scipy111111https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.peak_widths.html.

Figs. 9(a) shows the peak count and average peak area for collusive videos.

The final anomaly feature extractor $v_{a}$ has the following form:

(3) $v_{a}=(\phi,\omega)$

##### Training details:

As the idea here is to detect unusual commenting behaviour, we train GRU on
$\boldsymbol{V_{o}}$ (more details on $\boldsymbol{V_{o}}$ can be found in
Section 6.3.3) which is a set of random videos. Once trained, the model will
not be able to generate patterns similar to collusive videos resulting in
higher valued error vector or in simple terms – an anomalous activity. In our
experiments, we noticed that LSTM cell tends to generate multiple peaks for
single activity while GRU was able to mitigate this issue well. For the same,
we choose to employ GRU cells in place of LSTM cells (Malhotra et al., 2015).
Moreover, we could also reduce the model training time substantially using the
GRU cells in place of the LSTM cells.

(a)

(b)

Figure 9. (a) Exploratory analysis of the features used for anomaly detector
in case of collusive videos. (b) Linear pattern between subscribers and views
of YouTube channels i.e., channels with more subscriptions tend to lead to
more views.

####  6.3.2. Comment feature extractor

YouTube users generally comment on videos if either they like/dislike the
video or the domain of interest is the same. On the other hand, the collusive
users are majorly interested in uplifting their credits just by posting a
large number of comments. We observe that they tend to borrow the content from
the recently posted comments, make marginal changes and post the same. We
hypothesize that the comments posted during artificial boosting should have
high textual similarity between the collusive comments.

To this end, we measure the similarity of closely related comments posted
during the boosting timeline. We extract full comment texts of all the videos
using the YouTube API. The data collection strategy is detailed in Section
4.1.

Window selection strategy: Computing a similarity score by utilizing all the
previously posted comments for a given query comment might seem intuitive but
is a computationally heavy task. For the same, we first retrieve the comments
posted during peak time. Moreover, we propose to use a fixed-size ($w$) moving
window and roll it over the set of retrieved comments. We define the last
comment in each set as query comment ($q_{i}$) and the other comments as the
window comments ($C_{i}$). The best performing window size ($w$) from the
experiments was found to be of size $10$.

Comment similarity score ($\eta$): To calculate comment similarity score, we
first encode both the query comment $q_{i}$ and the window comments $C_{i}$
using Universal Sentence Encoder (USE) (Cer et al., 2018). As we know that
collusive users can belong to different geographical places and may not use
the same language, we chose to work with the multilingual USE
model121212https://tfhub.dev/google/universal-sentence-encoder-multilingual/1
to retrieve comment embeddings. In the literature, USE based model has been
able to achieve state-of-the-art for SemEval-2017 Task (Cer et al., 2017) on
Semantic Textual Similarity Multilingual, and Cross-lingual Focused
Evaluation. Currently, the multilingual model supports 16 different languages
and has shown strong performance in cross-lingual text retrieval. The input to
the model can be variable length text in any of the supported languages and
the output is a 512-dimensional vector. We transform each query comment into a
fix-length embedding vector $g(q_{i})$ and the window comments into set of
vectors $g(C_{i})$ using USE.

It has been observed that the same video was posted multiple times for the
collusive comments. It may also happen that the collusive user ran out of
credit points due to which the collusive video is no longer shown on the
blackmarket website until the user again starts earning some credit points. In
such cases, our peak detection strategy is likely to detect multiple peaks
($p$) for the videos ($V$). Moreover, a peak can have a large number of
comments. We define the comment similarity score ($\eta$) for a given video as
follows:

(4) $\eta=\frac{\sum_{i=1}^{P}P_{s(i)}}{P}$ (5)
$P_{s(i)}=\frac{\sum_{j=1}^{W}\max_{j:w_{j}\in P}(g(q_{j})\cdot
g(C_{j})^{T})}{W}$

where $P_{s(i)}$ denotes comment similarity score for $i$th peak, $P$ denotes
the number of peaks, $W$ denotes the number of windows, and $g(q_{j})$ and
$g(C_{j})$ denote the $j$th query embedding and $j$th window embedding
respectively. We choose the maximum of the dot product of query embedding and
the sentence embedding under the assumption that one comment is derived from
only one other comment.

We use the derived score $\eta$ along with the total comment count ($t_{c}$)
to create $v_{c}$. The final comment feature extractor $v_{c}$ has the
following form:

(6) $v_{c}=(\eta,t_{c})$

We then concatenate the learned representations from the metadata feature
extractor $v_{e}$, anomaly detector $v_{a}$ and comment feature extractor
$v_{c}$ to form the video feature representation denoted as $v=v_{e}\oplus
v_{a}\oplus v_{c}$.

####  6.3.3. Collusive video detector

Here we introduce the collusive video detector, as shown in Fig. 10. The
primary purpose of the detector is to learn the distribution from the
collusive data and identify similar ones. Although this task looks very
similar to the tasks mentioned in Sections 6.1 and 6.2, it actually uses a
different kind of features by incorporating the temporal representation of the
comments. To investigate this, we manually collected a small set of YouTube
channels that are not posted on YouLikeHits. We denote this set of videos by
$\boldsymbol{V_{o}}$. Here we only considered collecting the other set of
videos ($\boldsymbol{V_{o}}$) from the top $5000$ channels because YouTube
channels with more subscriptions tend to lead to more views, thus resembling
the collusive channels. Although it is not possible to find out view-timestamp
from the YouTube API for any given video, we can still expect the overall
behavior of $\boldsymbol{V_{o}}$ to be similar to Figure 9(b) as a user-driven
blackmarket service is analogous to a channel in a few ways such as both
benefit from more subscribers, the subscribers are loyal and tend to
watch/like/comment on the videos posted. We selected a playlist from each
channel at random. From each playlist, we randomly selected k (where, k=$3$ in
our case) videos. As the overall data was biased towards some specific genres,
we removed some videos at random to maintain generality in data. The presence
of such similar patterns in both the classes increases the difficulty of the
classification task even further but will make the classifier more relevant
for real-world collusion detection. Here $\boldsymbol{V_{o}}$ contains videos
having similar comment growth patterns and thus can be considered a
noisy/adversarial set in terms of comment growth patterns. Note that it does
not represent the entire YouTube population. For the same, we report the true
positive rate (TPR) of the model with respect to the set of collusive videos.

Training a fully connected network that directly optimizes only the supervised
objective by gradient descent also does not work very well in such cases. What
works better is to initially use a local unsupervised criterion to pre-train
each layer, with the goal of learning to produce a useful higher-level
representation from the lower-level representation output by the previous
layer. From this initial point, gradient descent on the supervised objective
leads to better solutions in terms of generalization performance (Vincent et
al., 2010; Zhang et al., 2018c).

We propose to use Denoising Autoencoder Classifier (DAC), which uses
autoencoders, a deep unsupervised learning method that improves the
generalization of supervised learning on limited labeled data. In particular,
we investigate a case where the input signal is noisy and we explore the
multi-task learning with two tasks: a classification output, coupled with a
reconstruction output of a Denoising Autoencoder. To establish the
effectiveness of this approach, we also present a comparison of this approach
against different baselines, including a multi-layer perceptron (MLP) model as
a replacement of the Denoising Autoencoder (more details in Table 6).

Figure 10. The architecture of Denoising Autoencoder Classifier.

Throughout this section, we define the input space as X, input data point x
$\in$ X, label space as Y, output label y $\in$ Y, intermediate decoded space
V, intermediate encoding z $\in$ Z, noisy or corrupted input data point
$\hat{x}$ $\in$ X.

At a high level, DAC consists of two structural components: the denoising
autoencoder, and the dense layer as the classifier. Denoising autoencoder has
a very similar structure to the autoencoders, except rather than reducing only
the reconstruction loss over the input data, it maps noisy inputs to original
clean inputs. Autoencoders consist of two components: an encoder
$(\tau:X\implies Z)$ and a decoder $(\psi:Z\implies X)$ where $\tau$ and
$\psi$ are inversely related. They first map an input $x\in
X\in\mathbb{R}^{d}$ to an intermediate representation $z\in
Z\in\mathbb{R}^{d^{\prime}}$ of reduced dimensionality $(d^{\prime}<d)$ via
$\tau$. This intermediate representation is then reconstructed back to a point
$x^{\prime}$ in the input space, and the difference between $x$ and
$x^{\prime}$ is measured using a loss function $L$.

On the contrary, in case of denoising autoencoder, a noisy representation of
$x$, $\hat{x}$, is fed to the network yielding $\hat{x}^{\prime}$, and then
$\hat{x}^{\prime}$ is compared to the original clean input data point $x$. The
classifier utilizes the intermediate representation $z$ of the denoising
autoencoder. After encoding noisy input, the classification task is added that
contributes to the encoder weights independently from the decoder. This task
consists of a fully-connected layer that predicts the label $y^{\prime}\in Y$.
Finally, both the classification loss and reconstruction loss propagate
backward and contribute to encoder weights.

Training details: We consider a fully-connected autoencoder with two hidden
layers containing 128 units each for all the experiments. We use mean squared
error as the loss function difference between $x$ and $\hat{x}^{\prime}$. To
generate corruption in training data, we randomly corrupt the input data by
$\pm 10\%$ of its original value. We use the entire unlabeled training set,
i.e., $1384$ videos – $756$ collusive and $628$ random videos, to train the
denoising autoencoder for 25 epochs and minimize mean squared error (MSE) from
the reconstruction output. During this stage, we do not update the weights for
edges between the encoded layer and classification output. The denoising
autoencoder classifier is then trained to minimize the categorical cross-
entropy loss on the labeled training samples for 150 epochs. Note that we
choose the optimal value of each parameter based on the hyperparameter search.
We implement our model using Python 3.8.3 and TensorFlow 2.3.1. The model has
a total of $18k$ trainable parameters and is less than $100KB$ in size.

##  7\. Experimental results

###  7.1. Baselines for detecting videos submitted for collusive comments

We validate the performance and robustness of CollATe by comparing with seven
baselines. Since there is no existing research dealing with the same problem
present in this paper, we develop a few of our own baselines and adopt some of
the existing studies to our problem setting. We report the performance of all
the competing methods along with the detailed analysis of the performance and
robustness.

  * •

Extractor models: CollATe uses the combination of metadata, anomaly and
comment feature extractors. Thus, we propose two baselines: (i) considering
only one extractor at a time ($B_{1}$), and (ii) considering two extractors at
a time ($B_{2}$). This also provides the ablation study of our method.

  * •

Variant of proposed model ($B_{3}$): As shown in Fig. 8, comment feature
extractor and anomaly feature extractor can be modified to obtain a variant of
CollATe. We modify both the components to obtain different variants of the
proposed model as described below:

LSTM-based Anomaly Extractor: We replace LSTM units in place of GRU for
detecting the anomalous comments. We observe that this variant tends to
generate multiple peaks for the videos with only one-time collusive comments.
This change not only adds the noisy comments but also deteriorates the overall
performance significantly as the comment feature extractor is dependent on
this stage.

WMD-based Comment Similarity Score: Word Mover’s Distance (WMD) (Kusner et
al., 2015) is a popular metric for calculating semantic similarity at a
document level. We utilize standard pre-trained Word2Vec embeddings from
Gensim library131313We tried with glove-twitter-200 and word2vec-google-
news-300 embeddings and found the latter to perform better. to compute the
Word Mover’s Distance $d_{ij}$. We transform the distance $d_{ij}$ into a
similarity score $s_{ij}$ as below:

(7) $s_{ij}=\frac{1}{1+d_{ij}}$

where $i$ denotes the comment index, and $j$ denotes the window index. Rest of
the details for computing $\eta$ remains the same as mentioned in Section 6.3.

There are two major differences between this model and CollATe – (i) word-
level vs. sentence-level embeddings, and (ii) monolingual vs. multilingual
support. We discuss the impact of these differences through our experiment
results in the next section.

  * •

DetectPV ($B_{4}$): We use the method proposed by Bulakh et al.(Bulakh et al.,
2014) as our fourth baseline. It uses a supervised learning approach to
identify fraudulently promoted videos by extracting features from the video
metadata. We wanted to check if video metadata-based fraudulent video
detection techniques would be useful to detect collusive videos on YouTube.

  * •

ARIMA ($B_{5}$): Since CollATe uses temporal information of comments posted on
videos to detect peaks and their associated properties, one may argue that a
time series based anomaly detection method may be able to detect collusive
videos. To this end, we consider the method proposed by (Box and Pierce, 1970)
as another baseline. ARIMA is an auto-regressive integrated moving average
model that uses the combination of auto-regression and moving average to
detect anomalies on time-series data.

  * •

CNN ($B_{6}$): Feature interaction-based prediction models such as
convolutional neural network (CNN) models have been used extensively for
various image processing tasks (Sindagi and Patel, 2018; Tu et al., 2019). In
our case, we use 1-D CNN layer, followed by dense layers to form a fully-
connected neural network. The model learns to extract and map features from
the data to the output labels.

  * •

WND ($B_{7}$): We use the Wide & Deep Factorization (WND) method proposed in
Guo et al.(Guo et al., 2017) as another baseline. The model consists of a wide
part (to memorize the past behavior) and deep part (to embed into lower
dimension) and utilizes deep neural network and factorization machines to
address the interactions among the features.

###  7.2. Prediction results

Tasks 1 and 2: Identify the videos (channels) submitted to blackmarket
services for collusive likes (subscriptions).
Due to the restriction of the YouTube Data API, we are unable to access the
timestamp of like/dislike activity. This resists us to create any time-centric
features for collusive like appraisals. Similarly, we are unable to get
detailed information about YouTube channels from the API. For our prediction
task, we use one-class classification models with the features mentioned in
Sections 6.1 and 6.2. The models are trained on one class, which in our case
is the collusive class. Note that we only report the true positive rate (TPR)
for our prediction task as we are only interested in the proportion of actual
positives that are correctly identified by the model. We perform the
prediction task with the following state-of-the-art one-class classifiers:
one-class SVM (Schölkopf et al., 2001), isolation forests (Liu et al., 2008),
minimum covariance determinant (Hubert et al., 2018), and local outlier factor
(Breunig et al., 2000). All the scores are reported after 5-fold cross
validation. Note that the test set remains same across all the competing
methods.

Table 5 reports the performance of the one-class classification models for
detecting videos submitted for collusive likes (Task 1) and channels submitted
for collusive subscriptions (Task 2). In the former task, we observe the best
accuracy of the model (with true positive rate of $0.911$) with one-class SVM.
To analyze the influence of each feature, we perform experiments, taking
$(n-1)$ features at a time, i.e., dropping each feature in isolation. The most
important feature turns out to be view rate based on quantified relative
importance. In the latter task, we once again observe that one-class SVM
performs the best (with true positive rate of $0.910$). View count turns out
to be the best feature for this task. Fig. 11 shows the feature importance for
Task 1 and Task 2 with the relative TPR drop percentage when we drop each
feature in isolation.

(a)

(b)

Figure 11. Feature importance considering (n-1) features at a time, i.e., dropping each feature in isolation (a) for Task 1 and (b) Task 2. Table 5. Performance of one-class classification models on two tasks. Task 1: videos submitted for collusive likes; Task 2: channels submitted for collusive subscriptions. Model | Task | True positive rate
---|---|---
One-class SVM | Task 1 | 0.911
| Task 2 | 0.910
Isolation forests | Task 1 | $0.902$
| Task 2 | $0.906$
Minimum covariance determinant | Task 1 | $0.900$
| Task 2 | $0.901$
Local outlier factor | Task 1 | $0.899$
| Task 2 | $0.894$

Task 3: Identify the videos submitted to blackmarket services for collusive
comments.
Our third task is to determine whether a given video was submitted to
blackmarket services for collusive comments. We use the approach explained in
Section 6.3 to detect if a video is collusive. Table 6 shows that DAC
consistently achieves better accuracy compared to its baselines. In Table 6,
we mark the best performing extractor for each of the baselines, $B_{1}$,
$B_{2}$ and $B_{3}$ in bold.

Here, CFE-USE denotes the usage of Universal Sentence Encoder while CFE-WMD
denotes the usage of Word Mover’s distance for calculating sentence
similarity. CFE-USE (comment similarity and comment count) with DAC is
individually the best performing extractor for baseline $B_{1}$. Although we
expect AFE to perform better than at least MFE, due to the nature of the non-
collusive video data, it does not. These results support our argument about
the higher similarity in the distribution for both collusive and non-collusive
datasets. For the same, we take into account multiple extractors that not only
improvise the performance but also increase the robustness. $B_{2}$ (MFE +
CFE-USE) with DAC outperforms other permutations as it captures the most
useful representation for classification.

$B_{3}$ (CFE-WMD) seems to perform better than a few baselines but is unable
to outperform CollATe. This is expected because WMD-based similarity score
calculation depends on word-level embeddings rather than sentence-level
embeddings. Moreover, we also find that the feature importance is evenly
distributed in CollATe compared to $B_{3}$. Kołcz and Teo (2009) also
suggested that the weight vector of a robust classifier should be distributed
as evenly as possible. Finally, we report the baseline performance of $B_{4}$
(DetectPV), $B_{5}$ (ARIMA), $B_{6}$ (CNN) and $B_{7}$ (WND). We observe that
the overall performance of CollATe is better than the baselines.

Table 6. Performance comparison of CollATe with baselines for detecting videos submitted for collusive comments. Method | TPR | FPR | Accuracy | AUC
---|---|---|---|---
$B_{1}$ (AFE) | $0.715$ | 0.455 | 0.638 | 0.629
$B_{1}$ (CFE-USE) (MLP) | $0.789$ | 0.410 | 0.718 | 0.689
$B_{1}$ (MFE) | $0.825$ | 0.444 | 0.702 | 0.690
$B_{1}$ (CFE-USE) (DAC) | 0.866 | 0.157 | 0.845 | 0.834
$B_{2}$ (MFE $+$ CFE-USE) (MLP) | $0.792$ | 0.395 | 0.704 | 0.698
$B_{2}$ (MFE $+$ AFE) | $0.795$ | 0.412 | 0.701 | 0.692
$B_{2}$ (AFE $+$ CFE-USE) (DAC) | 0.875 | 0.177 | 0.839 | 0.848
$B_{2}$ (MFE $+$ CFE-USE) (DAC) | $0.881$ | 0.179 | 0.853 | 0.851
$B_{3}$ (AFE + MFE + CFE-WMD) (DAC) | 0.866 | 0.182 | 0.844 | 0.841
$B_{4}$ (DetectPV) | $0.829$ | $0.232$ | $0.802$ | 0.798
$B_{5}$ (ARIMA) | $0.768$ | $0.429$ | 0.677 | 0.669
$B_{6}$ (CNN) | $0.679$ | $0.491$ | 0.609 | 0.594
$B_{7}$ (WND) | $0.890$ | 0.250 | 0.825 | 0.820
CollATe | 0.905 | 0.194 | 0.860 | 0.855

##  8\. Interesting Observations

In this section, we detail the important implications of collusive entity
detection task. To this end, we consider only the top $10\%$ of the collusive
entities detected (true positive) using our models to present our analysis.
For tasks 1 and 2, the top $10\%$ is generated using the scoring
function141414https://tinyurl.com/ve5yvr47 of our best-performing one-
classification model. For task 3, the top $10\%$ is generated using the
softmax values present in the last layer of CollATe. We use the term highly
collusive to refer to the collusive entities present in the top $10\%$ in each
of the cases.

######  Observation 1 (Highly collusive videos have high video ratings):

We define rating as the ratio of likes to the sum of likes and dislikes gained
by the video – the video will have rating $1$ if there are no dislikes. We
observe that highly collusive videos have very high ratings with an average
rating of $0.931$. This shows how blackmarket services have been able to gain
collusive appraisals in an effective way.

######  Observation 2 (Highly collusive videos have very short video
duration):

We note that highly collusive videos have a very short duration, with average
video length of only ~$4$ minutes. As confirmed by
YouTube151515https://youtube-creators.googleblog.com/2012/08/youtube-now-why-
we-focus-on-watch-time.html, it itself promotes videos that keep people on
YouTube for a long period of time. Thus, videos with short duration do not get
the proper audience naturally, thereby making the authors choose blackmarket
services to gain artificial appraisals in a quicker way.

######  Observation 3 (Gaining collusive likes does not guarantee to gain
collusive comments):

We observe that the videos submitted for collusive likes do not have many
comments in them, with the average number of comments being only $12.4$. This
also corroborates with the fact that we do not have any intersection between
the sets $V_{l}$ and $V_{c}$. We can ensure that collusive like requests and
collusive comment requests are two completely independent activities on the
blackmarket platforms.

######  Observation 4 (Highly collusive channels are popular channels with a
large number of videos):

We note that highly collusive channels are popular YouTube channels with
average subscribers count of $39,477$, average view count of $8,762,188$, and
average videos count of $111$. The reason is that getting new subscribers for
those channels is an extremely difficult task, which makes them choose
artificial ways of gaining new subscribers by means of the blackmarket
services.

######  Observation 5 (Highly collusive videos have a moderate inter-arrival
rate of comments):

We study the inter-arrival rate of comments in the highly collusive videos.
Surprisingly, we observe a mean inter-arrival rate of ~$5$ hours for each
comment. The possible reason behind such high value of inter-arrival rate is
due to the expiration of credits for the collusive comments in the blackmarket
services.

######  Observation 6 (Highly collusive videos have a moderate inter-arrival
rate of comments):

We study the inter-arrival rate of comments in the highly collusive videos.
Surprisingly, we observe a mean inter-arrival rate of ~$5$ hours for each
comment. The possible reason behind such high value of inter-arrival rate is
due to the expiration of credits for the collusive comments in the blackmarket
services.

###### Observation 7:

(‘People & Blogs’ is the most popular genre for highly collusive channels) We
observe ‘People & Blogs’ to be the most popular genre ($33.53\%$) for highly
collusive channels. This genre usually contains YouTube bloggers who upload
original content, and share it all with friends, family, and the world on
YouTube. This is due to the reason that blogging in YouTube is currently
considered as the most profitable way to earn
money161616https://medium.com/@KeywordsHeaven/blogging-vs-
youtube-72803bb3dacf. Thus, to gain quick popularity and recognition in the
community, these users join blackmarket services. The second most popular
genre for highly collusive channels is ‘Music’ ($22.60\%$).

###### Observation 8:

(Highly collusive videos are family-friendly and non-paid unlisted videos) We
observe that highly collusive videos are family-friendly and non-paid videos.
The possible reason behind not submitting violent and mature YouTube videos to
blackmarket services may be due to the reason that YouTube does not allow age-
restricted videos to be monetized. The possible reason behind not submitting
paid videos to blackmarket services is that the monetization of YouTube videos
is determined by the level of engagement (likes/comments) a video generates.

(a)

(b)

(c)

Figure 12. Example of (a) collusive video (for likes), (b) collusive channel
(for subscriptions), and (c) collusive video (for comments) detected by our
models. Sensitive information are blurred.

Fig. 12 shows the snapshot of some collusive entities detected using our
models. Fig. 12(a) shows a collusive YouTube video where the number of views
is much lesser than the number of likes. This clearly gives us the indication
that the YouTuber is buying artificial likes from the blackmarket websites.
Fig. 12(b) shows a collusive YouTube channel where the description clearly
states that the author is looking for quick popularity. The YouTube is
maintaining a timeline of the increase in the subscriber count of the channel.
Also, the channel description contains promotional keywords such as ‘gaining
more subscribers’. Fig. 12(c) shows a collusive YouTube video where the video
has more number of comments than the views and likes. This also gives us the
indication that the YouTuber is buying artificial comments from the
blackmarket websites.

##  9\. Conclusion and Future Work

With the increase in the involvement of content creators and content consumers
on YouTube, social growth has become the most important metric for popularity.
To achieve rapid social growth, a large number of content creators go against
the stream by registering their videos/channels to blackmarket services for
collusive appraisals. Although many studies have been carried out to detect
fraud and fake activities on multiple online media platforms, identification
of collusive entities remains a relatively important unexplored area of
research. The major contributions of this work are manifold. (i) We collected
a large dataset of YouTube videos submitted to blackmarkets for collusive
likes, comments and subscriptions. To the best of our knowledge, this is the
first dataset of this kind. (ii) We analyzed the collusive videos from two
perspectives: propagation dynamics and video metadata. The YouTube channels
are analyzed based on their location, video metadata and network properties.
(iii) In order to detect videos submitted in blackmarket services for
collusive likes and channels submitted for collusive subscriptions, we utilize
one-class classification models trained only on the collusive data. The SVM-
based model achieves $0.911$ TPR using video metadata features and $0.910$ TPR
using channel features to detect videos and channels submitted to blackmarket
services for collusive likes and subscriptions respectively. (iv) We then
proposed CollATe, a system that combines three feature extractors (metadata,
anomaly and comment) to learn representations of a video. CollATe makes use of
extractors effectively for identifying whether a video is registered in
blackmarket services for collusive comment appraisals. Extensive experiments
on our dataset show that CollATe is effective in detecting collusive entities
with $0.905$ TPR. (v) As a final contribution, we show the important
implications of the collusive entity detection task. We expect this research
to push further studies in online media outlets to explore the dynamics of
collusive behavior.

We believe our proposed methodologies can be helpful for a variety of tasks in
similar research. For the first two models, i.e., identifying videos
(channels) submitted to blackmarket services for collusive likes
(subscriptions), it can only be adopted to video-sharing platforms as the
features used in the models are generated from video (channel) metadata
present on YouTube. In case of CollATe, the anomaly feature extractor and
comment feature extractor can be adopted by any video or non-video sharing
platforms where a user posts texts on some entities of that platform. E.g., in
case of Twitter to detect users who submit tweets to blackmarket services for
collusive retweet appraisals, the anomaly feature extractor can utilize the
retweet time-series data to detect sudden changes in the retweeting activity;
comment feature extractor can utilize the text of the comments posted during
collusion and compute a similarity score between the comments. However, we
have not performed this experiment in the current paper since the major aim of
this paper is to provide a detection framework for collusive YouTube entities,
not developing a framework for detecting collusive entities on Twitter which
we explored in our previous studies (Dutta et al., 2018; Dutta and
Chakraborty, 2019; Chetan et al., 2019; Arora et al., 2019; Arora et al.,
2020).

Despite encouraging results, collusive entities detection still remains a
challenging problem with many open research questions. In the future, we are
interested in exploring the following avenues. First, we plan to take into
account the sentiment of the comments by considering the average comment
sentiment during collusion for detecting the collusive activity. Second, we
wish to study the interdependency of the collusive videos and collusive users
that can help in identifying the core users of the blackmarket services.
Third, we intend to detect collusive entities at an inter-platform level as
well. Fourth, our final goal will be to design a web-based scalable collusive
entity detection system for online video sharing platforms.

## Acknowledgement

The authors would like to thank the support of ECR/2017/001691 (SERB),
Ramanujan Fellowship, and ihub-Anubhuti-iiitd Foundation (set up under the NM-
ICPS scheme of the Department of Science and Technology, India).

## References

  * (1)
  * Acker (2018) Amelia Acker. 2018\.  Data Craft: the Manipulation of Social Media Metadata.  Data & Society Research Institute (2018).
  * Aggarwal and Kumaraguru (2014) Anupama Aggarwal and Ponnurangam Kumaraguru. 2014.  Followers or phantoms? An anatomy of purchased Twitter followers.  arXiv preprint arXiv:1408.1534 (2014).
  * Aiyar and Shetty (2018) Shreyas Aiyar and Nisha P Shetty. 2018.  N-gram assisted Youtube spam comment detection.  Procedia computer science 132 (2018), 174–182.
  * Alassad et al. (2019) Mustafa Alassad, Nitin Agarwal, and Muhammad Nihal Hussain. 2019\.  Examining intensive groups in YouTube commenter networks. In International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation. Springer, 224–233.
  * Alberto et al. (2015) Túlio C Alberto, Johannes V Lochter, and Tiago A Almeida. 2015.  Tubespam: Comment spam filtering on youtube. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). IEEE, 138–143.
  * Alsaleh et al. (2015) Mansour Alsaleh, Abdulrahman Alarifi, Fatima Al-Quayed, and AbdulMalik Al-Salman. 2015\.  Combating comment spam with machine learning approaches. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). IEEE, 295–300.
  * Arora et al. (2020) Udit Arora, Hridoy Sankar Dutta, Brihi Joshi, Aditya Chetan, and Tanmoy Chakraborty. 2020.  Analyzing and Detecting Collusive Users Involved in Blackmarket Retweeting Activities.  ACM Transactions on Intelligent Systems and Technology (TIST) 11, 3 (2020), 1–24.
  * Arora et al. (2019) Udit Arora, William Scott Paka, and Tanmoy Chakraborty. 2019\.  Multitask learning for blackmarket tweet detection. In Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. 127–130.
  * Badri Satya et al. (2016) Prudhvi Ratna Badri Satya, Kyumin Lee, Dongwon Lee, Thanh Tran, and Jason Jiasheng Zhang. 2016\.  Uncovering fake likers in online social networks. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, 2365–2370.
  * Bagler (2008) Ganesh Bagler. 2008\.  Analysis of the airport network of India as a complex weighted network.  Physica A: Statistical Mechanics and its Applications 387, 12 (2008), 2972–2980.
  * Box and Pierce (1970) George EP Box and David A Pierce. 1970.  Distribution of residual autocorrelations in autoregressive-integrated moving average time series models.  Journal of the American statistical Association 65, 332 (1970), 1509–1526.
  * Breunig et al. (2000) Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. 2000.  LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data. 93–104.
  * Bulakh et al. (2014) Vlad Bulakh, Christopher W Dunn, and Minaxi Gupta. 2014\.  Identifying fraudulently promoted online videos. In Proceedings of the 23rd International Conference on World Wide Web. 1111–1116.
  * Castellini et al. (2017) Jacopo Castellini, Valentina Poggioni, and Giulia Sorbi. 2017\.  Fake twitter followers detection by denoising autoencoder. In Proceedings of the International Conference on Web Intelligence. ACM, 195–202.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, 1–14.  https://doi.org/10.18653/v1/S17-2001
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018\.  Universal sentence encoder.  arXiv preprint arXiv:1803.11175 (2018).
  * Chen et al. (2015) Liang Chen, Yipeng Zhou, and Dah Ming Chiu. 2015.  Analysis and detection of fake views in online video services.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 11, 2s (2015), 44.
  * Chetan et al. (2019) Aditya Chetan, Brihi Joshi, Hridoy Sankar Dutta, and Tanmoy Chakraborty. 2019.  CoReRank: Ranking to Detect Users Involved in Blackmarket-Based Collusive Retweeting Activities. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. ACM, 330–338.
  * Chowdury et al. (2013) Rashid Chowdury, Md Nuruddin Monsur Adnan, GAN Mahmud, and Rashedur M Rahman. 2013.  A data mining based spam detection system for youtube. In Eighth International Conference on Digital Information Management (ICDIM 2013). IEEE, 373–378.
  * Cresci et al. (2015) Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2015\.  Fame for sale: Efficient detection of fake Twitter followers.  Decision Support Systems 80 (2015), 56–71.
  * Dave et al. (2012) Vacha Dave, Saikat Guha, and Yin Zhang. 2012.  Measuring and fingerprinting click-spam in ad networks. In Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication. ACM, 175–186.
  * De Cristofaro et al. (2014) Emiliano De Cristofaro, Arik Friedman, Guillaume Jourjon, Mohamed Ali Kaafar, and M Zubair Shafiq. 2014.  Paying for likes?: Understanding facebook like fraud using honeypots. In Proceedings of the 2014 Conference on Internet Measurement Conference. ACM, 129–136.
  * Dhawan et al. (2019) Sarthika Dhawan, Siva Charan Reddy Gangireddy, Shiv Kumar, and Tanmoy Chakraborty. 2019.  Spotting Collusive Behaviour of Online Fraud Groups in Customer Reviews.  arXiv preprint arXiv:1905.13649 (2019).
  * Dutta et al. (2021) Hridoy Sankar Dutta, Kartik Aggarwal, and Tanmoy Chakraborty. 2021.  DECIFE: Detecting Collusive Users Involved in Blackmarket Following Services on Twitter.  arXiv preprint arXiv:2107.11697 (2021).
  * Dutta and Chakraborty (2019) Hridoy Sankar Dutta and Tanmoy Chakraborty. 2019.  Blackmarket-driven Collusion among Retweeters–Analysis, Detection and Characterization.  IEEE Transactions on Information Forensics and Security (2019).
  * Dutta and Chakraborty (2020) Hridoy Sankar Dutta and Tanmoy Chakraborty. 2020.  Blackmarket-driven collusion on online media: a survey.  arXiv preprint arXiv:2008.13102 (2020).
  * Dutta et al. (2018) Hridoy Sankar Dutta, Aditya Chetan, Brihi Joshi, and Tanmoy Chakraborty. 2018.  Retweet us, we will retweet you: Spotting collusive retweeters involved in blackmarket services. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 242–249.
  * Dutta et al. (2020) H. S. Dutta, V. R. Dutta, A. Adhikary, and T. Chakraborty. 2020.  HawkesEye: Detecting Fake Retweeters Using Hawkes Process and Topic Modeling.  IEEE Transactions on Information Forensics and Security 15 (2020), 2667–2678.
  * Egele et al. (2013) Manuel Egele, Gianluca Stringhini, Christopher Kruegel, and Giovanni Vigna. 2013.  Compa: Detecting compromised accounts on social networks.. In NDSS. 1–17.
  * Egele et al. (2015) Manuel Egele, Gianluca Stringhini, Christopher Kruegel, and Giovanni Vigna. 2015.  Towards detecting compromised accounts on social networks.  IEEE Transactions on Dependable and Secure Computing 14, 4 (2015), 447–460.
  * Faddoul et al. (2020) Marc Faddoul, Guillaume Chaslot, and Hany Farid. 2020\.  A Longitudinal Analysis of YouTube’s Promotion of Conspiracy Videos.  arXiv preprint arXiv:2003.03318 (2020).
  * Ghosh et al. (2012) Saptarshi Ghosh, Bimal Viswanath, Farshad Kooti, Naveen Kumar Sharma, Gautam Korlam, Fabricio Benevenuto, Niloy Ganguly, and Krishna Phani Gummadi. 2012.  Understanding and combating link farming in the twitter social network. In Proceedings of the 21st international conference on World Wide Web. ACM, 61–70.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction.  arXiv preprint arXiv:1703.04247 (2017).
  * Gupta et al. (2013) Aditi Gupta, Hemank Lamba, Ponnurangam Kumaraguru, and Anupam Joshi. 2013.  Faking sandy: characterizing and identifying fake images on twitter during hurricane sandy. In Proceedings of the 22nd international conference on World Wide Web. ACM, 729–736.
  * Hubert et al. (2018) Mia Hubert, Michiel Debruyne, and Peter J Rousseeuw. 2018\.  Minimum covariance determinant and extensions.  Wiley Interdisciplinary Reviews: Computational Statistics 10, 3 (2018), e1421.
  * Hussain et al. (2018) Muhammad Nihal Hussain, Serpil Tokdemir, Nitin Agarwal, and Samer Al-Khateeb. 2018.  Analyzing disinformation and crowd manipulation tactics on YouTube. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 1092–1095.
  * Ikram et al. (2017) Muhammad Ikram, Lucky Onwuzurike, Shehroze Farooqi, Emiliano De Cristofaro, Arik Friedman, Guillaume Jourjon, Mohammed Ali Kaafar, and M Zubair Shafiq. 2017.  Measuring, characterizing, and detecting facebook like farms.  ACM Transactions on Privacy and Security (TOPS) 20, 4 (2017), 13\.
  * Jiang et al. (2014) Meng Jiang, Peng Cui, Alex Beutel, Christos Faloutsos, and Shiqiang Yang. 2014.  Catchsync: catching synchronized behavior in large directed graphs. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 941–950.
  * Keller (2018) Michael H Keller. 2018\.  The flourishing business of fake YouTube views.  The New York Times (2018).
  * Kołcz and Teo (2009) Aleksander Kołcz and Choon Hui Teo. 2009.  Feature weighting for improved classifier robustness. In CEAS’09: sixth conference on email and anti-spam. 1–10.
  * Kumar and Shah (2018) Srijan Kumar and Neil Shah. 2018.  False information on web and social media: A survey.  arXiv preprint arXiv:1804.08559 (2018).
  * Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015.  From word embeddings to document distances. In International conference on machine learning. 957–966.
  * Li et al. (2016) Yixuan Li, Oscar Martinez, Xing Chen, Yi Li, and John E Hopcroft. 2016.  In a world that counts: Clustering and detecting fake social engagement at scale. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 111–120.
  * Liu et al. (2008) Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008\.  Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining. IEEE, 413–422.
  * Malhotra et al. (2015) Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, and Puneet Agarwal. 2015.  Long short term memory networks for anomaly detection in time series. In Proceedings. Presses universitaires de Louvain, 89.
  * Marciel et al. (2016) Miriam Marciel, Rubén Cuevas, Albert Banchs, Roberto González, Stefano Traverso, Mohamed Ahmed, and Arturo Azcorra. 2016\.  Understanding the detection of view fraud in video content portals. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 357–368.
  * Mehrotra et al. (2016) Ashish Mehrotra, Mallidi Sarreddy, and Sanjay Singh. 2016\.  Detection of fake twitter followers using graph centrality measures. In 2016 2nd International Conference on Contemporary Computing and Informatics (IC3I). IEEE, 499–504.
  * Metwally et al. (2007) Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi. 2007\.  Detectives: detecting coalition hit inflation attacks in advertising networks streams. In Proceedings of the 16th international conference on World Wide Web. ACM, 241–250.
  * Nilizadeh et al. (2017) Shirin Nilizadeh, François Labrèche, Alireza Sedighian, Ali Zand, José Fernandez, Christopher Kruegel, Gianluca Stringhini, and Giovanni Vigna. 2017.  Poised: Spotting twitter spam off the beaten paths. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 1159–1174.
  * Schölkopf et al. (2001) Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. 2001\.  Estimating the support of a high-dimensional distribution.  Neural computation 13, 7 (2001), 1443–1471.
  * Sen et al. (2018) Indira Sen, Anupama Aggarwal, Shiven Mian, Siddharth Singh, Ponnurangam Kumaraguru, and Anwitaman Datta. 2018.  Worth its weight in likes: towards detecting fake likes on Instagram. In Proceedings of the 10th ACM Conference on Web Science. ACM, 205–209.
  * Shah et al. (2017) Neil Shah, Hemank Lamba, Alex Beutel, and Christos Faloutsos. 2017\.  The many faces of link fraud. In 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 1069–1074.
  * Sindagi and Patel (2018) Vishwanath A Sindagi and Vishal M Patel. 2018.  A survey of recent advances in cnn-based single image crowd counting and density estimation.  Pattern Recognition Letters 107 (2018), 3–16.
  * Stringhini et al. (2015) Gianluca Stringhini, Pierre Mourlanne, Gregoire Jacob, Manuel Egele, Christopher Kruegel, and Giovanni Vigna. 2015.  $\\{$EVILCOHORT$\\}$: Detecting Communities of Malicious Accounts on Online Services. In 24th $\\{$USENIX$\\}$ Security Symposium. 563–578.
  * Stringhini et al. (2013) Gianluca Stringhini, Gang Wang, Manuel Egele, Christopher Kruegel, Giovanni Vigna, Haitao Zheng, and Ben Y Zhao. 2013.  Follow the green: growth and dynamics in twitter follower markets. In Proceedings of the 2013 conference on Internet measurement conference. ACM, 163–176.
  * Sureka (2011) Ashish Sureka. 2011\.  Mining user comment activity for detecting forum spammers in youtube.  arXiv preprint arXiv:1103.5044 (2011).
  * Thomas et al. (2011) Kurt Thomas, Chris Grier, Dawn Song, and Vern Paxson. 2011\.  Suspended accounts in retrospect: an analysis of twitter spam. In Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference. ACM, 243–258.
  * Tu et al. (2019) Zhigang Tu, Wei Xie, Dejun Zhang, Ronald Poppe, Remco C Veltkamp, Baoxin Li, and Junsong Yuan. 2019.  A survey of variational and CNN-based optical flow techniques.  Signal Processing: Image Communication 72 (2019), 9–24.
  * Uysal (2018) Alper Kürşat Uysal. 2018\.  Feature selection for comment spam filtering on YouTube.  Data Science and Applications 1, 1 (2018), 4–8.
  * Vincent et al. (2010) Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010\.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  Journal of machine learning research 11, Dec (2010), 3371–3408.
  * Wang (2010) Alex Hai Wang. 2010\.  Don’t follow me: Spam detection in twitter. In 2010 international conference on security and cryptography (SECRYPT). IEEE, 1–10.
  * Wang et al. (2011) Guan Wang, Sihong Xie, Bing Liu, and S Yu Philip. 2011\.  Review graph based online store review spammer detection. In 2011 IEEE 11th International Conference on Data Mining. IEEE, 1242–1247.
  * Yang et al. (2017) Guolei Yang, Neil Zhenqiang Gong, and Ying Cai. 2017\.  Fake Co-visitation Injection Attacks to Recommender Systems.. In NDSS. 1–17.
  * Yusof and Sadoon (2017) Yuhanis Yusof and Omar Hadeb Sadoon. 2017.  Detecting video spammers in youtube social media. In Proceedings of International Conference on Computing and Informatics. 228–234.
  * Zhang et al. (2018a) Daniel Yue Zhang, Jose Badilla, Herman Tong, and Dong Wang. 2018a.  An end-to-end scalable copyright detection system for online video sharing platforms. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 626–629.
  * Zhang et al. (2018b) Daniel Yue Zhang, Qi Li, Herman Tong, Jose Badilla, Yang Zhang, and Dong Wang. 2018b.  Crowdsourcing-based copyright infringement detection in live video streams. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 367–374.
  * Zhang et al. (2018c) Jinsong Zhang, Yi Zhang, Lianfa Bai, and Jing Han. 2018c.  Lossless-constraint Denoising based Auto-encoders.  Signal Processing: Image Communication 63 (2018), 92 – 99.  https://doi.org/10.1016/j.image.2018.02.002
  * Zhang and Sun (2017) Wuxain Zhang and Hung-Min Sun. 2017.  Instagram spam detection. In 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing (PRDC). IEEE, 227–228.
  * Zhu et al. (2016) Zhen-hui Zhu, Yang Zhi, and Ya-fei Dai. 2016.  A New Approach to Detect User Collusion Behavior in Online QA System. In International Conference on Computer Networks and Communication Technology (CNCT 2016). Atlantis Press, 836–842.