# A Simple-but-effective Baseline for Training-free Class-Agnostic Counting

Yuhao Lin, Haiming Xu, Lingqiao Liu, Javen Qinfeng Shi
Australian Institute for Machine Learning
University of Adelaide
{yuhao.lin01, hai-ming.xu, lingqiao.liu<EMAIL_ADDRESS>
###### Abstract

Class-Agnostic Counting (CAC) seeks to accurately count objects in a given
image with only a few reference examples. While previous methods achieving
this relied on additional training, recent efforts have shown that it’s
possible to accomplish this without training by utilizing pre-existing
foundation models, particularly the Segment Anything Model (SAM), for counting
via instance-level segmentation. Although promising, current training-free
methods still lag behind their training-based counterparts in terms of
performance. In this research, we present a straightforward training-free
solution that effectively bridges this performance gap, serving as a strong
baseline. The primary contribution of our work lies in the discovery of four
key technologies that can enhance performance. Specifically, we suggest
employing a superpixel algorithm to generate more precise initial point
prompts, utilizing an image encoder with richer semantic knowledge to replace
the SAM encoder for representing candidate objects, and adopting a multiscale
mechanism and a transductive prototype scheme to update the representation of
reference examples. By combining these four technologies, our approach
achieves significant improvements over existing training-free methods and
delivers performance on par with training-based ones.

##  1 Introduction

Figure 1:  Visualizing the performance, our method notably narrows the gap in
counting accuracy between training-free and training-based approaches.
Impressively, it even surpasses the top-performing training-based methods on
the CARPK dataset, underscoring the potential of training-free approaches.

Class-Agnostic Counting (CAC) [14] is increasingly gaining recognition in the
field of computer vision. In contrast to traditional class-specific object
counting [19, 22], which relies on training models for specific predefined
categories, CAC adopts a more flexible approach. It expects the model to adapt
to a diverse array of objects across various classes using only a minimal
number of examples [14]. This adaptability makes CAC a compelling area of
research, offering potential for broad application in scenarios where rapid or
dynamic object recognition is required.

To attain the ability for Class-Agnostic Counting, a common approach is to
utilize a training dataset containing various objects along with their
corresponding density maps. This training data is used to train a model that
compares visual features between reference examples and the query image [14,
22, 6]. This methodology enables class-independent adaptation, which is
essential for scenarios requiring dynamic counting. However, a significant
drawback of these methods is their reliance on large-scale annotated training
data, which presents challenges in terms of labor intensity and scalability
across diverse visual categories [16].

Recently, the Segment Anything Model (SAM) [9] has garnered substantial
attention across various research domains due to its remarkable ability to
generate high-quality object masks for any given image. This versatility has
led to its application in numerous research areas, as evidenced by multiple
studies [27, 15, 25, 21]. In the context of CAC, researchers have started
adopting SAM for producing instance-level segmentation masks corresponding to
referenced objects. By counting the number of instance masks, they can
determine the final count of objects. Thanks to SAM’s robust instance
segmentation capabilities, this counting-via-instance-segmentation approach
can function effectively even when utilizing the off-the-shelf SAM model
without any additional training. Training-free methods hold appeal for CAC
since they eliminate the need for a dedicated training dataset and alleviate
the challenges associated with collecting extra training data. Nevertheless,
despite their promise, current training-free CAC methods still lag behind
their training-based counterparts in terms of performance, e.g., as
demonstrated in Figure 1, the current top training-free method falls short of
the best training-based approaches by 84.89% and 105.82% on the FSC-147 [19]
and CARPK [8] datasets, respectively.

In our research, we introduce an improved approach to achieve training-free
Class-Agnostic Counting using SAM. Our method provides a straightforward yet
robust means to significantly narrow the performance gap often observed in
training-free CAC methods. This positions our approach as a valuable benchmark
for future research in this domain. Our approach to CAC relies on four pivotal
technological components, each playing a crucial role: First, we emphasize the
paramount importance of precise point prompt placement within SAM to achieve
accurate object masking. To this end, we advocate the utilization of
superpixels, which enables the creation of initial point prompts, resulting in
high object recall without imposing significant computational overhead.
Moreover, our thorough investigations have exposed limitations in SAM’s
current image encoder concerning its ability to effectively capture semantic-
level similarity. In response, we propose the adoption of image encoders
equipped to capture richer semantic information. This proposition holds the
potential to enhance the representation of objects during the counting
process. Additionally, we recognize the significance of implementing a
multiscale mechanism, particularly for the precise counting of small objects.
By incorporating this mechanism, our approach accommodates variations in
object size across diverse instances, thus yielding more accurate counting
results. Finally, we introduce a transductive prototype update strategy for
reference examples. Through the augmentation of reference examples with
candidate objects that are likely to share the same object category, we
substantially enhance the reference-candidate similarity metric. This
enhancement empowers us to more effectively determine whether an object
proposal qualifies as the target of interest, ultimately contributing to the
overall accuracy of our Class-Agnostic Counting methodology. Collectively,
these four technological advancements serve as the cornerstone of our
approach.

Through our rigorous experimental evaluation, we have demonstrated the
remarkable performance of our proposed methods and the effectiveness of the
four key components. Significantly, we have shown, for the first time, that a
Class-Agnostic Counting (CAC) model, even without any training, can achieve
performance on par with that of training-based CAC approaches.

##  2 Related Work

###  2.1 Class-Agnostic Counting Methods

Lu et al. [14] first address CAC, employed convolutional neural networks to
extract and concatenate features from both query images and exemplars to
estimate object counts. However, this method faced challenges related to
overfitting when directly regressing from concatenated features. Addressing
this, CFOCNet [29] introduced improvements by explicitly modeling similarity,
using a Siamese network to track to improve localization and count accuracy.

Further enhancements in similarity modeling were proposed by Shi et al. [22]
and You et al. [29], who integrated advanced techniques such as self-attention
and learnable similarity metrics to reduce appearance variability and guide
feature fusion, respectively. Moreover, the incorporation of a vision
transformer for feature extraction by Liu et al. [12] marked a significant
step towards leveraging the strengths of transformer models in the field,
demonstrating the use of cross-attention to merge image and exemplar features
effectively. Most recently, LOCA [6] introduces an object prototype extraction
module that iteratively merges exemplar shape and appearance with image
features, enabling adaptation from low-shot to zero-shot object counting
scenarios.

However, these methods all need numerous training data, and the exemplar
annotation is labor-consuming. In specific, the most commonly used dataset in
CAC task, FSC-147 [19], includes 3659 training images with more than 10,000
exemplar annotations needed.

Ma et al. [16] first address CAC with Segment Anything Model (SAM) [9] in a
training-free manner. After that Shi et al. [23] proposed a novel training-
free object counter that introduces prior-guided mask generation to improve
accuracy, along with a two-stage text-based object counting method, yielding
competitive results on standard benchmarks. However, the performance of these
training-free methods heavily lags behind the training-based CAC methods. In
this work, we propose a simple but effective training-free approach to
significantly reduce the performance gap.

Figure 2:  Top row illustrates the creation process of object-prior point
prompts. Bottom row depicts a pipeline overview of our proposed training-free
object counting approach. Details of the transductive update module are
provided in Eq. 4. Reference objects are marked with yellow boxes in the input
image (zoom in for clarity).

###  2.2 Superpixel Methods

Superpixel is an important concept in computer vision, primarily used to group
pixels into perceptually meaningful regions, which can significantly reduce
the complexity of image processing tasks. As a way to effectively reduce the
number of image primitives for subsequent processing, superpixel algorithms
have been widely adopted in vision problems such as saliency detection [26],
object detection [7], and semantic segmentation [24].

Superpixel was popularized by Ren and Malik [20] through their pioneering work
on the Normalized Cuts algorithm, and has been significantly advanced by the
introduction of the Simple Linear Iterative Clustering (SLIC) [2] algorithm.
SLIC uses K-means clustering in a five-dimensional space of color and pixel
coordinates to create compact and uniform superpixels, which has made it a
benchmark in the field. Several variants(LSC [11], Manifold SLIC [13], SNIC
[1]) have been developed upon SLIC.

###  2.3 Vision Foundation Models

For pure vision area, DINO (self-DIstillation with NO labels) [3] and its more
advanced iteration, DINOv2 [17], represent significant strides in self-
supervised learning within computer vision. Through a teacher-student
distillation process that does not rely on labeled data, DINO masters
semantic-rich visual representation. DINOv2 builds on this by refining the
architecture and training mechanisms, further improving the model’s efficiency
and the accuracy of learned representations.

Most recently, SAM [9] specializes in image segmentation, adeptly generating
accurate object masks from prompts such as points and boxes. SAM’s proficiency
is evidenced by its remarkable performance on diverse segmentation benchmarks
and its zero-shot transfer abilities, which extend across a range of datasets
[4, 5].

These foundation models collectively redefine the capabilities of computer
vision systems. They provide a versatile framework for tackling new tasks and
adapting to unfamiliar data distributions without the necessity of bespoke
training [27, 15, 25].

##  3 Method

In this section, we introduce four innovative modules that enhance the
effectiveness of employing SAM in training-free CAC, i.e., object-prior point
prompt (Sec. 3.1) and semantic-rich feature (Sec. 3.2) are matters for the
training-free model to deliver satisfactory counting outcomes. Additionally,
the multi-scale strategy (Sec. 3.3) and transductive prototype updating (Sec.
3.4) help substantially elevate counting accuracy. The comprehensive framework
of our method is depicted in Figure 2.

###  3.1 Object-prior Point Prompt Matters

In the use of SAM for zero-shot object proposal generation [9], a regular grid
of point prompts is utilized to automatically generate class-agnostic masks.
In the scenario of object counting, counted objects are usually tiny or
crowded. In order to increase the recall of the object proposals, fine-grained
grid point prompts became indispensable. As presented in Figure 2, naively
increasing the number of points grid from 1600 to 3600 increases recall by
20%. However, an increase in the number of point grids leads to a doubling of
computational time overhead. In order to ensure the recall rate without
increasing time complexity too much, we propose to utilize the superpixel
segmentation algorithm [20] to firstly perceptual group pixels into clusters
and then take cluster centers as object-prior point prompts to feed into SAM
for object proposal generation.

Figure 3: Effectiveness demonstration of the use of superpixel on the quality
of mask proposals generated by SAM. Left: an image with reference exemplars
(hot-air balloon) in black boxes. Right: a bubble chart illustrates the trade-
off between the recall rate of the interested object and the time cost.
Without increasing the number of point prompts, SAM with
superpixel222Performing superpixel on this example image takes only 10ms with
AMD Ryzen 9 3900X CPU, a negligible computational expense compared to the SAM
computation cost.can significantly improve the recall rate of hot-air balloons
in the output mask proposals, thus avoiding the potential computational burden
caused by the demand for denser point grids. Please find visualizations of the
generated mask proposal in the appendix.

In this study, we use the classic superpixel algorithm named Simple Linear
Iterative Clustering (SLIC) which can generate compact superpixels with a low
computational overhead [2]. Given the initial cluster number $K$, SLIC can
directly output superpixel centers $\mathcal{C}=\\{(x_{i},y_{i})\\}_{i=1}^{K}$
which can then be fed into SAM together with the provided reference boxes (or
points) $\mathcal{R}$ batch by batch for mask proposal generation

$\mathcal{M}_{\text{ref}}=\mathbb{D}(\mathbb{E}(I),\mathcal{R}),\quad\mathcal{M}_{\text{proposal}}=\mathbb{D}(\mathbb{E}(I),\mathcal{C})$
(1)

where $\mathbb{E}$ and $\mathbb{D}$ are encoder and decoder of SAM
respectively. $I$ is the input image. $\mathcal{M}_{\text{ref}}$ denotes the
generated masks for corresponding referenced prompts and
$\mathcal{M}_{\text{proposal}}$ means all potential mask proposals 333The
biggest mask for background segmentation and masks for referenced objects are
filtered from $\mathcal{M}_{\text{proposal}}$ for simplicity, i.e.,
$\mathcal{M}_{\text{proposal}}\cap\mathcal{M}_{\text{ref}}=\emptyset$..

Figure 4:  Visualization of the similarity mappings of image features to
reference object features (marked in black boxes) using SAM and DINOv2. It
distinctly shows that SAM’s similarity mapping erroneously highlights numerous
areas unrelated to the target object, whereas DINOv2’s mapping accurately
encompasses the objects of interest. This demonstrates that DINOv2’s features
possess more semantically relevant knowledge.

###  3.2 Semantic-rich Feature Matters

When object masks are ready, the next step is to recognize all object masks
similar to the given reference objects from $\mathcal{M}_{\text{proposal}}$.
One straightforward way is to compare the similarity of the region-of-mask
features (RoMF) of referenced objects to that of the RoMF of others

$\displaystyle\mathcal{S}^{i}=\text{sim}(\mathsf{P}_{\text{ref}},\
F^{i}_{\text{proposal}}),\quad\text{where}$ (2)
$\displaystyle\mathsf{P}_{\text{ref}}=\frac{1}{n_{\text{ref}}}\sum_{j=1}^{n_{\text{ref}}}\text{AvgPool}\left(\mathbb{E}(I),\
\mathcal{M}^{j}_{\text{ref}}\right)$ $\displaystyle
F^{i}_{\text{proposal}}=\text{AvgPool}\left(\mathbb{E}(I),\
\mathcal{M}^{i}_{\text{proposal}}\right)$

where AvgPool is the average pooling operation. $n_{\text{ref}}$ is the number
of given reference objects and is set to 3 by default in the literature [19].
$\mathsf{P}_{\text{ref}}$ denotes the average of referenced object features,
a.k.a prototypes. $\text{sim}(\cdot,\cdot)$ is the similarity function and the
cosine function is used in this study. $\mathcal{S}^{i}$ is the similarity
score for each object proposal $i$.

Next, an overall segmentation for interested objects can be obtained by
selecting mask proposals whose similarity score is higher than a predefined
threshold $\theta$. However, searching for an optimal $\theta$ can be hard due
to the feature representations of SAM lacking clear semantic-related knowledge
[16, 10]. As empirical analysis presented in Figure 4, many objects of
different categories from the reference object may have a quite close
similarity score to that of objects in the same category, thus it is
inevitable to output poor counting results with noisy segment generation. For
instance, the SAM feature struggles to differentiate between apples and other
messy objects in row 1, as well as children’s hair and feet in row 2 of Figure
4.

Figure 5:  Effectiveness demonstration of our multi-scale mechanism in
scenarios involving extremely tiny counting objects. Merely increasing the
quantity of point prompts555Multiple colors are used for the point grid in the
figures of columns 2-4 for better visualization.in SAM fails to yield accurate
instance-level mask proposals. However, integrating SAM with our multi-scale
mechanism (using 32*32 points for aerial photography of sheep in this
demonstration) effectively achieves better mask quality.

In order to obtain a plausible counting accuracy, using semantic-rich features
for mask proposal selection becomes indispensable. Recently, many vision
foundation models with rich semantic knowledge have been made accessible to
the public, such as DINO [3, 17]. Combining with the semantic-rich feature
representation, the final counting results can be obtained

$\displaystyle\text{count}=n_{\text{ref}}+\sum_{i=1}^{N}\mathbbm{1}(\mathcal{S}^{i}>\theta),\quad\text{where}$
(3) $\displaystyle\mathcal{S}^{i}=\text{sim}(\mathsf{P}_{\text{ref}},\
F^{i}_{\text{proposal}}),$
$\displaystyle\mathsf{P}_{\text{ref}}=\frac{1}{n_{\text{ref}}}\sum_{j=1}^{n_{\text{ref}}}\text{AvgPool}\left(\mathbb{E}_{\text{sem}}(I),\
h(\mathcal{M}^{j}_{\text{ref}})\right)$ $\displaystyle
F^{i}_{\text{proposal}}=\text{AvgPool}\left(\mathbb{E}_{\text{sem}}(I),\
h(\mathcal{M}^{i}_{\text{proposal}})\right)$

where $\mathbb{E}_{\text{sem}}$ is the encoder with rich semantic knowledge
and $h(\cdot)$ is the interpolation function to accommodate the resolution of
SAM generated mask to that of the encoder output. $N$ is the number of object
proposals. $\theta$ is the similarity threshold and is empirically set to 0.4
in this study.

###  3.3 Multi-scale Segmentation Helps

In the task of object counting, there exist some scenes where objects that
need to be counted are extremely tiny and crowded, e.g., aerial photography of
sheep. It becomes challenging for SAM to generate reasonable instance-level
masks. As Figure 5 shows, naive increasing the number of points from 6400 to
even 14400 still fails to make SAM generate precise mask proposals for the
sheep instance. In order to tackle this problem, we propose a simple multi-
scale mechanism. Specifically, we first cut the input image into
$n_{\text{p}}\times n_{\text{p}}$ patches of the same size666Empirically,
$n_{\text{p}}=2$ is found to perform well in this study.. Then we resize these
patches to the original size of the input image and feed them together with
the original input image into SAM for mask proposal generation. Please see
Figure 2 for details. As the bottom left picture in Figure 5 shows, using this
simple multi-scale mechanism can successfully obtain precise instance-level
mask proposals.

| Training-free |  Reference Format | FSC-147 | CARPK
---|---|---|---|---
| MAE $\downarrow$ | RMSE $\downarrow$ | MAE $\downarrow$ | RMSE $\downarrow$
GMN [14] [ACCV 19’] | No | Box | 26.52 | 124.57 | 9.90 | -
FamNet [19] [CVPR 21’] | No | Box | 22.08 | 99.54 | 18.19 | 33.66
CFOCNet+ [28] [WACV 21’] | No | Box | 22.10 | 112.71 | - | -
BMNet+ [22] [CVPR 22’] | No | Box | 14.62 | 91.83 | 5.76 | 7.83
SAFECount [29] [WACV 23’] | No | Box | 14.32 | 85.54 | 5.33 | 7.04
LOCA [6] [ICCV 23’] | No | Box | 10.79 | 56.97 | 9.97 | 12.51
SAM Baseline [23] | Yes | N.A. | 42.48 | 137.50 | 16.97 | 20.57
Count-Anything [16] | Yes | Box | 27.97 | 131.24 | - | -
TFOC [23] [WACV 24’] | Yes | Box | 19.95 | 132.16 | 10.97 | 14.24
Ours | Yes | Box | 12.26 | 56.33 | 4.39 | 5.70
TFOC [23][WACV 24’] | Yes | Point | 20.10 | 132.83 | 11.01 | 14.34
Ours | Yes | Point | 12.47 | 49.97 | 4.39 | 5.70
Table 1:  Quantitative comparisons between our methods and others using point
and box reference formats on the FSC-147 and CARPK benchmarks are presented.
Our approach surpasses other training-free methods and shows competitive
results against training-based ones. The best counting outcomes among
training-free methods are emphasized in bold font.

###  3.4 Transductive Prototype Updating Helps

In Sec. 3.2, the prototype of the interested object $\mathsf{P}_{\text{ref}}$
is simply the average of given few-shot reference object features. Since
objects in the same class may have various appearances in the same image, this
simple prototype may not be representative enough to match all interested
objects. We argue that potentially interested-object features from mask
proposals can be helpful for matching generalization and propose a novel
transductive prototype updating strategy to improve the quality of the
prototype

$\mathsf{P}_{\text{ref}}^{\text{new}}=\frac{n_{\text{ref}}\cdot\mathsf{P}_{\text{ref}}+\sum_{i}^{N}\mathbbm{1}(\mathcal{S}^{i}>\delta)\cdot
F^{i}_{\text{proposal}}}{n_{\text{ref}}+\sum_{i}^{N}\mathbbm{1}(\mathcal{S}^{i}>\delta)}$
(4)

where $\delta$ is a similarity threshold and is set to 0.5 in this study for
selecting plausible reference candidates.

##  4 Experiment

Figure 6:  Qualitative results on the FSC-147 dataset are displayed. Exemplars
are marked with black (white) boxes, and counting values are noted in the
bottom-right corner. Error predictions are indicated by red boxes. Our method
accurately predicts masks in both densely and sparsely populated scenes. Best
viewed by zooming in.

In this section, we first provide details of our experimental setup. Next, we
assess our proposed method on two widely-used CAC benchmarks, comparing it
against both training-based and training-free CAC methods. Lastly, we conduct
comprehensive ablation studies to further explore the effectiveness of our
approach.

###  4.1 Experimental Setup

Dataset: In our study, we assess our method using two popular datasets for
counting tasks: FSC-147 [19] and CARPK [8]. FSC-147, the first extensive CAC
dataset, includes 6,135 images across 147 categories, divided into 3,659
training, 1,286 validation, and 1,190 test images. Each image is annotated
with bounding boxes and central points of three exemplars. Since our approach
doesn’t require training, we focus solely on the test set, which features
1,190 images from 29 categories. The CARPK dataset comprises 1,448 images of
car parks captured from drones. Following the protocol in [6, 22, 23], we
select twelve exemplars from the training images of CARPK for use in all test
images. Similar to FSC-147, our evaluation is confined to the test set,
containing 459 images.

Evaluation Metrics: Follow [6, 22], we report the Mean Absolute Error (MAE),
Root Mean Square Error (RMSE). In particular, MAE =
$\frac{1}{n}\sum^{n}_{i=1}|y_{i}-\hat{y_{i}}|$, RMSE =
$\sqrt{\frac{1}{n}\sum^{n}_{i=1}(y_{i}-\hat{y_{i}})^{2}}$, where n is the
number of test images, $y$ is the ground truth and $\hat{y}$ is the
prediction.

Comparing Methods: Our approach is benchmarked against the latest state-of-
the-art methods in both trained and training-free scenarios. For trained
scenarios, we compare with several CAC techniques: GMN (General Matching
Network [14]), FamNet (Few-shot adaptation and matching Network [19]), CFONet+
(Class-agnostic Fewshot Object Counting Network [28]), the advanced BMNet+
(Bilinear Matching Network [22]), SAFECount (similarity-aware feature
enhancement [29]), and LOCA (Low-shot Object Counting network with iterative
prototype Adaptation [6]). In the training-free setting, our models are
evaluated alongside two existing methods: Count-Anything [16] and TFOC
(training-free object counting [23]).

All experiments in this work are conducted on one NVIDIA GeForce RTX 3090 GPU
and all other experimental settings are identical to the TFOC [23] approach
for a fair comparison.

###  4.2 Comparison with State-of-the-Arts

Quantitative Results: Table 1 reports the comparison results on two popular
CAC benchmarks. First, our approach achieves consistent performance
improvements over the compared training-free methods on various evaluation
metrics. Specifically, On the FSC-147 dataset, our approach lowers the Mean
Absolute Error (MAE) to 12.26, outperforming the naive SAM Baseline, which has
an MAE of 42.48, and the Count-Anything method with an MAE of 27.97. Compared
to the state-of-the-art training-free method TFOC [23], the performance gain
of our approach reaches +7.69 MAE on the FSC-147 dataset and +6.58 MAE on
CARPK dataset. Meanwhile, the RMSE metric of our approach is also
significantly lower than the TFOC method which demonstrates that our method
can perform better in scenarios with more crowded objects.

Second, our approach greatly reduces the performance gap between the training-
based and training-free methods. For example, on FSC-147, the gap has
decreased from 9.16 to 1.47 on MAE and even outperformed 0.64 on RMSE. On the
CARPK dataset, our approach excels in all the compared algorithms and creates
a state-of-the-art performance.

Finally, we also evaluate the proposed method with point references and the
consistent superior performance demonstrates the robustness of our approach.

Qualitative Analysis: Figure 6 displayed the visualization results of various
counting methods in different scenarios and our approach can always more
accurately identify objects of interest in various scenes. Specifically, for
scenes in the first three rows, both of the compared training-based and
training-free methods mistakenly identified areas that do not belong to the
interested object, such as the teacup handle and the spoon in row 1, the
crowded leaves in row 2 and the table legs and the top edge of the chair back
in row 3. While our approach can successfully and accurately segment out the
interested object and produce a more accurate counting result. This will be
attributed to the utilization of semantic-rich feature representations for
SAM’s mask proposal selection in our approach. Furthermore, for more
challenging scenes in the last two rows, the density maps generated by the
training-based BMNet+ method are vague and exhibit substantial deviations in
counting outcomes. The TFOC method, on the other hand, either erroneously
includes extensive background areas in its object count or overlooks numerous
objects in crowded scenes. Yet, our method yields accurate segmentation masks
for individual objects due to its multi-scale mechanism, resulting in output
counting numbers that more closely align with the ground truth.

##  5 Ablation study

Components | FSC-147
---|---
SP | DINOv2 | TPU | MS | MAE $\downarrow$ | RMSE $\downarrow$
- | - | - | - | 30.91 | 140.10
✓ | - | - | - | 30.89 | 140.06
- | ✓ | - | - | 18.89 | 134.01
✓ | ✓ | - | - | 17.32 | 132.64
✓ | ✓ | ✓ | - | 16.04 | 132.40
✓ | ✓ | - | ✓ | 13.77 | 87.40
✓ | ✓ | ✓ | ✓ | 12.26 | 56.33
Table 2: Performance with different components. SP: superpixel, TPU: transductive prototype updating, MS: multi-scale. | SAM Backbone | MAE $\downarrow$ | RMSE $\downarrow$
---|---|---|---
TFOC [23] | ViT-B | 19.95 | 132.16
ViT-H | 30.41 | 138.28
Ours | ViT-B | 14.43 | 87.54
ViT-H | 12.26 | 56.33
Table 3: Performance with different SAM backbone.

In this section, we are interested in ablating our approach from the following
perspective views on FSC-147 dataset:

Effectiveness of different components: Table 2 showcases the impact of each
component in our method. Specifically, although SAM with superpixel can
achieve higher interested-object recall (shown in Figure 2), due to the lack
of semantic differentiation ability in SAM features (shown in Figure 4),
simply replacing default point grid with superpixel does not necessarily bring
performance gains, as the results shown in the first two rows of Table 2. The
next two rows show that using DINOv2 feature representation for SAM’s mask
proposal selection significantly reduces the final counting error and the
combing of superpixel can further bring performance gains. The subsequent two
rows highlight the individual benefits of transductive prototype updating and
the multi-scale mechanism on our model’s effectiveness. When these four
components are combined, they lead to the most impressive counting results.

Backbone size of SAM: Table 3 compares TFOC and our method across different
ViT backbones, revealing that our approach improves as the backbone scales
from base to huge. In contrast, TFOC’s performance declines with ViT-H,
possibly due to an increase in mask proposals leading to poorer selection. Our
method, leveraging semantically rich features, benefits from the higher number
of generated object proposals.

Various semantic-rich vision models: Table 4 displays our method’s performance
across different vision models, indicating that superior vision models yield
improved counting accuracy.

Vision Model | Patch Grid | MAE $\downarrow$ | RMSE $\downarrow$
---|---|---|---
CLIP [18] | 16*16 | 17.83 | 53.81
DINO [3] | 28*28 | 14.28 | 59.33
DINO v2 [17] | 37*37 | 12.26 | 56.33
Table 4: Performance with different semantic-rich model. Update Round | MAE $\downarrow$ | RMSE $\downarrow$
---|---|---
0 | 13.77 | 87.40
1 | 12.26 | 56.33
2 | 12.30 | 57.50
Table 5: Performance with various rounds of TPU.

Figure 7: Performance with various threshold $\theta$ and $\delta$.

Multi-round transductive prototype updating: Our study typically employs one
round of transductive prototype updating (TPU). Table 5 demonstrates that
additional rounds do not enhance performance.

Threshold sensitivity: Our method involves two hyperparameters: a similarity
threshold $\theta$ for proposal selection and $\delta$ for transductive
prototype updating. Figure 7 illustrates the robustness of our approach to
variations in these hyperparameters.

##  6 Conclusion

This paper presents a novel, training-free approach for Class-Agnostic
Counting (CAC), significantly closing the performance gap with trained models.
Key contributions include the use of superpixels, a richer semantic image
encoder, a multiscale mechanism, and a transductive prototype updating
strategy. Through rigorous experimental evaluation, the effectiveness of these
four key components are demonstrated. The research shows that a CAC model
without any training can achieve performance on par with training-based CAC
approaches. This is a significant advancement in the field, indicating the
potential of training-free approaches and setting a new benchmark for future
research in CAC.

Supplementary Material

In this supplementary material, we provide more details to complement the
manuscript, including visualization of generated mask proposal in Sec. 7, CAC
with text references in Sec. 8 and qualitative results on CARPK dataset in
Sec. 9.

##  7 Visualization of Generated Mask Proposal

Figure 8: Visualization of mask proposals generated by SAM with various point
prompts. In this example, the hot-air balloon is the user-interested object
and the bottom row presents the recall of this object. The red boxes display
mask generation for a very fine object (the street lamp).

In the main paper, we have presented the effectiveness of the use of
superpixel on the quality of mask proposals generated by SAM in Figure 3. In
this section, we further illustrate the visualization of generated mask
proposals in Figure 8. It clearly shows that SAM with superpixel can achieve a
high recall rate for the interested object without demanding of denser grid of
points. Meanwhile, we find that SAM with superpixel can also generate mask
proposals for very thin objects thanks to the object-prior prompt, such as the
street lamp marked in the red box. Nonetheless, employing SAM with a standard
grid of points might not consistently achieve segmentation for such objects.

##  8 CAC with Text References

In the main paper, the efficiency of our method using box and point references
has been confirmed. In this section, we extend our examination to include the
use of text references in testing our approach. For a fair comparison, we
adopt the method from TFOC [23] and employ the CLIP-Surgery model to derive
coarse bounding boxes for the target object using text references. Then our
approach can use these coarse bounding boxes for object counting. As presented
in the following Table 6, our approach is significantly superior to the
compared methods.

| Training-free | FSC-147
---|---|---
|  MAE $\downarrow$ |  RMSE $\downarrow$
SAM Baseline [23] | Yes | 42.48 | 137.50
TFOC [23] (vanilla) | Yes | 32.86 | 142.89
TFOC [23] | Yes | 24.79 | 137.15
Ours | Yes | 14.88 | 56.91

Table 6: Effect of our approach with text prompt on FSC-147.

##  9 Qualitative Results on CARPK

In this section, we further provide some qualitative visualization for the
CARPK dataset. As shown in Figure 9, our method can generate precise masks for
each car instance and achieve accurate counting results.

Figure 9:  Qualitative results on the CARPK dataset are displayed. Counting
values are noted in the bottom-left corner. Best viewed by zooming in.

## References

  * Achanta and Susstrunk [2017] Radhakrishna Achanta and Sabine Susstrunk.  Superpixels and polygons using simple non-iterative clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4651–4660, 2017.
  * Achanta et al. [2012] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk.  Slic superpixels compared to state-of-the-art superpixel methods.  _IEEE transactions on pattern analysis and machine intelligence_ , 34(11):2274–2282, 2012.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chen et al. [2023] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.  Anydoor: Zero-shot object-level image customization.  _arXiv preprint arXiv:2307.09481_ , 2023.
  * Deng et al. [2023] Ruining Deng, Can Cui, Quan Liu, Tianyuan Yao, Lucas W Remedios, Shunxing Bao, Bennett A Landman, Lee E Wheless, Lori A Coburn, Keith T Wilson, et al.  Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging.  _arXiv preprint arXiv:2304.04155_ , 2023.
  * Djukic et al. [2022] Nikola Djukic, Alan Lukezic, Vitjan Zavrtanik, and Matej Kristan.  A low-shot object counting network with iterative prototype adaptation.  _arXiv preprint arXiv:2211.08217_ , 2022.
  * Gould et al. [2008] Stephen Gould, Jim Rodgers, David Cohen, Gal Elidan, and Daphne Koller.  Multi-class segmentation with relative location prior.  _International journal of computer vision_ , 80:300–316, 2008.
  * Hsieh et al. [2017] Meng-Ru Hsieh, Yen-Liang Lin, and Winston H Hsu.  Drone-based object counting by spatially regularized regional proposal network.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4145–4153, 2017.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * Li et al. [2023] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao.  Semantic-sam: Segment and recognize anything at any granularity.  _arXiv preprint arXiv:2307.04767_ , 2023.
  * Li and Chen [2015] Zhengqin Li and Jiansheng Chen.  Superpixel segmentation using linear spectral clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1356–1363, 2015.
  * Liu et al. [2022] Chang Liu, Yujie Zhong, Andrew Zisserman, and Weidi Xie.  Countr: Transformer-based generalised visual counting.  _arXiv preprint arXiv:2208.13721_ , 2022.
  * Liu et al. [2016] Yong-Jin Liu, Cheng-Chi Yu, Min-Jing Yu, and Ying He.  Manifold slic: A fast method to compute content-sensitive superpixels.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 651–659, 2016.
  * Lu et al. [2019] Erika Lu, Weidi Xie, and Andrew Zisserman.  Class-agnostic counting.  In _Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part III 14_ , pages 669–684. Springer, 2019.
  * Ma and Wang [2023] Jun Ma and Bo Wang.  Segment anything in medical images.  _arXiv preprint arXiv:2304.12306_ , 2023.
  * Ma et al. [2023] Zhiheng Ma, Xiaopeng Hong, and Qinnan Shangguan.  Can sam count anything? an empirical study on sam counting.  _arXiv preprint arXiv:2304.10817_ , 2023.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ranjan et al. [2021] Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai.  Learning to count everything.  In _CVPR_ , pages 3394–3403, 2021.
  * Ren and Malik [2003] Ren and Malik.  Learning a classification model for segmentation.  In _Proceedings ninth IEEE international conference on computer vision_ , pages 10–17. IEEE, 2003.
  * Shen et al. [2023] Qiuhong Shen, Xingyi Yang, and Xinchao Wang.  Anything-3d: Towards single-view anything reconstruction in the wild.  _arXiv preprint arXiv:2304.10261_ , 2023.
  * Shi et al. [2022] Min Shi, Hao Lu, Chen Feng, Chengxin Liu, and Zhiguo Cao.  Represent, compare, and learn: A similarity-aware framework for class-agnostic counting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9529–9538, 2022.
  * Shi et al. [2023] Zenglin Shi, Ying Sun, and Mengmi Zhang.  Training-free object counting with prompts.  _arXiv preprint arXiv:2307.00038_ , 2023.
  * Shu et al. [2013] Guang Shu, Afshin Dehghan, and Mubarak Shah.  Improving an object detector and extracting regions using superpixels.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3721–3727, 2013.
  * Wang et al. [2023] Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al.  Caption anything: Interactive image description with diverse multimodal controls.  _arXiv preprint arXiv:2305.02677_ , 2023.
  * Yang et al. [2013] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang.  Saliency detection via graph-based manifold ranking.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3166–3173, 2013.
  * Yang et al. [2023] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng.  Track anything: Segment anything meets videos.  _arXiv preprint arXiv:2304.11968_ , 2023.
  * Yang et al. [2021] Shuo-Diao Yang, Hung-Ting Su, Winston H Hsu, and Wen-Chin Chen.  Class-agnostic few-shot object counting.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 870–878, 2021.
  * You et al. [2023] Zhiyuan You, Kai Yang, Wenhan Luo, Xin Lu, Lei Cui, and Xinyi Le.  Few-shot object counting with similarity-aware feature enhancement.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 6315–6324, 2023.


# The influence of coordinated behavior on toxicity

Edoardo Loru  Sapienza University of Rome  Matteo Cinelli  Department of
Computer Science, Sapienza University of Rome  Maurizio Tesconi  Institute of
Informatics and Telematics, National Research Council  Walter Quattrociocchi
Department of Computer Science, Sapienza University of Rome

###### Abstract

In the intricate landscape of social media genuine content dissemination may
be altered by a number of threats. Coordinated Behavior (CB), defined as
orchestrated efforts by entities to deceive or mislead users about their
identity and intentions, emerges as a tactic to exploit or manipulate online
discourse. This study delves into the relationship between CB and toxic
conversation on Twitter. Using a dataset of 11 million tweets from 1 million
users preceding the 2019 UK General Elections, we show that users displaying
CB typically disseminate less harmful content, irrespective of political
affiliation. However, distinct toxicity patterns emerge among different CB
cohorts. Compared to their non-CB counterparts, CB participants show
marginally elevated toxicity levels only when considering their original
posts. We further show the effects of CB-driven toxic content on non-CB users,
gauging its impact based on political leanings. Our findings suggest a nuanced
but statistically significant influence of CB on digital discourse.

##  1 Introduction

Social media are nowadays one of the main arenas for public debate where users
get their information and interact with other peers under the potential
influence of feed algorithms used to prioritize their engagement with like-
minded content [1, 2, 3]. According to recent studies, such systems can
challenge democracy in various ways [4, 5, 6]. Problems include the fast
spread of false information [7, 8, 9, 10, 11], more division among groups [12,
13, 14, 15, 16], and harmful behaviors online [17, 18, 19]. Despite efforts to
fix these issues, solutions are hard to find [5, 6].

Further complicating this ecosystem is the phenomenon of Coordinated Behavior
(CB), which can be defined as an unexpected, suspicious, or exceptional
similarity among users of a group [20]. Social media campaigns, such as online
activism, protests, and disinformation campaigns [21, 22, 23], generally
involve participants coordinating their actions to disseminate content widely.
Initially, scientific research focused on the benefits of coordination for
social movements. However, it has become evident that benign actors, such as
activists, are using similar techniques and malicious actors are engaged in
political astroturfing [24] and the dissemination of inappropriate content.
Coordinated behavior on social media can have negative consequences, including
distorting public opinion and contributing to the polarization of society.
Recognizing these problems, researchers and practitioners are working on
strategies to identify, characterize, and mitigate coordinated behavior [25,
26, 27]. In particular, the characterization of coordinated groups is a
crucial aspect discussed in the existing literature [25]. This can be done at
different levels of depth but remains essential due to the absence of ground-
truth data for detection tasks. To assess the harm of coordinated behavior,
established methods in the literature can be used, primarily the analysis of
content shared by coordinated users, which includes the identification of fake
news [28, 29], hate speech and toxicity detection [18, 30].

In the context of the rapidly evolving digital landscape, the 2019 UK General
Elections provide a pertinent setting to explore the dynamics of online
behavior. Our study aims to disentangle the complex relationship between CB
and the prevalence of toxic content on the Twitter platform. For this purpose,
we analyze a dataset encompassing 11 million tweets from a diverse pool of 1
million users. From our analysis, a salient observation emerges: users
demonstrating high coordination in online activities tend to disseminate less
toxic content. This propensity holds regardless of their political leaning,
suggesting that coordination might not necessarily be synonymous with
malicious intent or negative discourse. However, when we delve deeper into the
subsets of coordinated users, nuanced toxicity patterns begin to materialize,
indicating a varied landscape of content sharing even within these coordinated
groups. In more detail, we observe that the extensive retweeting activity of
coordinated users plays a role in evaluating their toxicity levels. Both
coordinated and non-coordinated users tend to post original content with a
higher toxicity level compared to that they disseminate through retweets, with
the former category being even more toxic than the latter. Beyond merely
observing these trends, our study delves into the consequential effects of
content stemming from CB efforts. Specifically, we explore how interacting
with toxic content, particularly when associated with CB, affects the
behaviors of non-coordinated users. A key aspect of this examination is
understanding the potential modulation of reactions by political orientations.
Does a user’s political inclination amplify or attenuate their reaction to CB-
driven toxicity? After carefully considering these dynamics, our results point
towards a subtle yet undeniably present influence of CB on online
conversations and narratives. While the impact might not always be overtly
visible in direct content comparisons, the undercurrents shaping digital
discourse reveal a significant role in coordinated behaviors.

##  2 Materials and Methods

###  2.1 Data

In this work, we use a publicly
available111https://doi.org/10.5281/zenodo.4647893, large collection of tweets
gathered in the run-up to the 2019 UK general election [20], spanning from 12
November 2019 to 12 December 2019 (election day). Within this time frame, all
tweets that featured at least one of the predefined election-related hashtags
in Table 1 were collected; some of the hashtags have a clear political
alignment (Labour or Conservative), while the rest only refer to the election
itself and can be considered neutral. Additionally, the dataset includes all
tweets published by the official accounts of the two parties and their leaders
and all interactions they received (i.e., retweets and replies), as summarized
in Table 2. The final dataset combines these two collection processes,
resulting in a set of 11,264,280 tweets posted by 1,179,659 distinct users.

#### Toxicity

To estimate the toxicity conveyed by a message, we leveraged the publicly
available Perspective API [31], which is an established standard for toxic
speech detection. We will thus follow the definition of “toxic content” it is
based upon, which is “a rude, disrespectful, or unreasonable comment that is
likely to make people leave a discussion”. To achieve more robust results, we
standardized all tweets in our dataset by enforcing UTF-8 encoding, converting
them to lower-case, and stripping them of all hashtags, URLs, and emojis.
Therefore, we input them to Perspective API and assign to each the toxicity
score output by the model, a value in the range $[0,1]$ that represents “the
likelihood that someone will perceive the text as toxic” [31].

Figure 1: Empirical Cumulative Distribution Function (ECDF) of the number of
tweets posted by superspreaders compared to the rest of the users, taking into
account original tweets (left panel) and retweets (right panel).

#### Coordinated communities

By applying the state-of-the-art method proposed in [20], we subsequently
identified the coordinated users within our dataset, assigning each user a
score between 0 and 1 that measures the extent of coordination exhibited by
that user. Firstly, we selected the top 1% retweeters in the dataset, which
from this point onward we will refer to as superspreaders, resulting in a
subset of 10,782 users. Despite being characterized by an extremely prolific
retweeting activity, in Fig. 1 we show that superspreaders also tend to
produce more original tweets than other users.

Secondly, we computed the TF-IDF vector of the IDs of the tweets retweeted by
each superspreader, thus obtaining for each a retweet-based feature vector.
This weighting allows us to place more importance to the action of retweeting
of unpopular tweets, which is a telltale sign of suspicious behavior. By
computing the pairwise cosine similarities between user vectors, we thus
obtained a weighted undirected similarity network, whose nodes are the
superspreaders we identified earlier and the weight of the edges between them
represents their similarity in retweet activity. We then extracted the
multiscale backbone [32] of the similarity network as to only retain
statistically relevant connections, resulting in a network containing 276,775
edges, and performed community detection with the Louvain clustering algorithm
[33]. On this dataset, Nizzoli et al. (2021) [20] detected seven clusters of
coordinated users, which they subsequently characterized politically by means
of their hashtag usage. In this work, we will focus on providing a toxicity-
based characterization for the three largest communities: the LAB and CON
clusters, populated by users supporting the Labour and Conservative party,
respectively, and the TVT cluster, a community promoting liberal democrats,
tactical voting, anti-Brexit, and anti-Tory campaigns. In addition to being
the largest, we argue that these communities are also the most representative
of the dataset, and thus most valuable for investigation: the first two have
distinctly politically slanted narratives, while the latter can serve as a
benchmark to validate the interplay between political alignment, toxicity, and
coordinated behavior. Finally, we applied the algorithm proposed in [20] to
assign each user a coordination score in the range $[0,1]$.

For the purposes of this study, we label as coordinated all superspreaders
having a coordination score above the median of the coordination score
empirical distribution, which is approximately equal to 0.8, resulting in a
set of 5,450 coordinated accounts. We consider the set of remaining users –
including all other superspreaders – to be non-coordinated.

###  2.2 Political leaning inference

We aim to investigate the role of political leaning and its relationship with
coordination and toxicity. The alignment of a user on the political spectrum
can be estimated by studying hashtag usage [34, 35]. Specifically, we built a
tweet-hashtag bipartite network starting from the sets of all tweets in our
dataset and the hashtags in them. Then, we projected this network onto a
hashtag co-occurrence network, whose nodes represent hashtags and an edge
between two hashtags indicates the two have appeared at least once in the same
tweet; the weight of the edge is a positive integer representing the number of
such co-occurrences. The co-occurrence network resulting from this projection
contains a total of 100,461 nodes and 822,420 edges. This network can then be
used to infer the leaning of each hashtag by applying a label propagation
algorithm, using the set of hashtags we defined for the data collection step
as an initial seed of known polarity in a way similar to [9]. The algorithm
builds upon the multiscale backbone extraction method [32] to identify
relevant connections (i.e., co-occurrences) among the network’s hashtags. All
nodes are initially assigned an “undefined” political leaning score, except
for the initial seed of hashtags of known polarity used for data collection
(Table 1). A single iteration $i$ of the algorithm consists of two main
operations: extraction of the backbone of the co-occurrence network with a
disparity filter $\alpha_{i}$; and simultaneous update of all hashtags with
undefined leaning at step $i$. The leaning assigned to a hashtag is equal to
the average of the leaning of its neighbors, weighted by the number of co-
occurrences with each. In this computation, all neighbors that weren’t already
assigned a leaning in a previous step are temporarily assigned a leaning of 0.
At step $i+1$, a larger disparity filter $\alpha_{i+1}>\alpha_{i}$ is used to
extract the backbone; this corresponds to a softer filtering that allows more
nodes to be part of the extracted network. The newly added hashtags and those
that haven’t been labeled in the previous steps will thus be assigned a
political leaning score. The algorithm stops when all hashtags have been
assigned a score or when a disparity filter equal to 1 is employed, meaning
the update is performed on the entire network. In the latter case, all
hashtags that still haven’t been updated at the end of the algorithm will be
assumed to be neutral and assigned a political leaning score of zero. The
length of the sequence of disparity filters will have an impact on the final
result of the label propagation, as it’s strongly dependent on the network it
is applied on and on the nodes used as the initial seed: defining an
excessively short sequence may lead to many neutral hashtags, while a long one
– although preferable – can become computationally expensive if the network
involved is large. For our purposes, we have found choosing a sequence of
values that scale logarithmically and span across several orders of magnitude
to be the optimal choice, in contrast to setting a fixed increment to be added
at each iteration.

Knowing the polarity of hashtags allows us to get an estimate of the political
leaning of each tweet, which we define as equal to that of the hashtag in it
with the highest score in absolute value (i.e., most polarized); this avoids
the leaning of polarized hashtags being averaged out in tweets where they are
used in conjunction with many politically neutral hashtags. Finally, we
estimate the political leaning of a user as the average political leaning
score of its tweets and retweets.

##  3 Results and Discussion

Figure 2: Joint distributions of different metrics for the superspreaders in
the three largest communities of the similarity network. Color intensity
indicates the number of users in a bin, with red regions highlighting peaks.
(a) Coordination score and toxicity of users, with the dotted line (median
coordination) indicating the threshold used to label a superspreader as
“coordinated”; (b) toxicity of users and weighted average of their
neighborhood’s toxicity; (c) cluster-normalized (least to most extreme)
political leaning of users and their toxicity; (d) political leaning of users
and weighted average of their neighborhood’s leaning.

###  3.1 Toxicity across coordinated communities

To analyze variations in toxicity across coordinated communities, each user
was assigned a toxicity score. This score is determined by averaging the
toxicity of a user’s top 10% most toxic original tweets and retweets. By
focusing on this metric, we aimed to gauge the peak toxicity a user can
manifest and disseminate [36, 18] rather than their standard activity levels.
Fig. 2(a) displays the joint Probability Density Function (PDF) of the
coordination scores of superspreaders from the three largest clusters within
the similarity network, aligned with their toxicity (as defined previously). A
discernible trend emerges: strongly-coordinated users tend to display below-
average toxicity. This implies a propensity among coordinated users to
disseminate content with low toxicity levels. This pattern is particularly
pronounced within the Labour and Conservative clusters, representing the most
distinctly politically aligned communities among superspreaders. Furthermore,
Fig. 2(a) highlights the coordination score threshold employed to classify a
superspreader as ‘coordinated’; subsequent analyses will center on this
specific user subset.

Fig. 2(b) depicts the joint PDF of each coordinated user’s toxicity against
the average toxicity of their neighboring users. These neighbors are factored
by their similarity (i.e., edge weight) to the user. The displayed PDF exposes
observable toxicity-based homophily in the CON cluster and, to a lesser
extent, in the TVT cluster, suggesting that users with comparable retweet
behaviors within these clusters also exhibit similar toxicity levels. To
confirm this intuition, we can consider the subgraph corresponding to each
cluster and estimate its assortativity [37] with respect to the toxicity of
the users within, resulting in a score of 0.54 for the CON cluster, 0.30 for
the TVT cluster, and 0.06 for the LAB cluster. Shuffling the toxicity scores
among users of the same cluster allows us to validate these estimates further
[38]; after 10,000 random shuffles, we obtain an average assortativity
coefficient approximately equal to zero and a Z-score $\gg 1$ for the observed
coefficients of the CON and the TVT clusters.

In Fig. 2(c), we present the joint distribution of user toxicity juxtaposed
with a normalized political leaning score. Interestingly, the distribution
pertaining to the coordinated users in the CON cluster suggests that the
toxicity they express and their political alignment are negatively correlated;
in fact, for the two metrics we obtain a Pearson’s correlation coefficient of
$\rho=-0.44$, statistically validated with a $t$-test ($p<0.001$) against the
hypothesis of no correlation. For additional context, Fig. 2(d) illustrates
the relationship between the political leaning of coordinated users and their
neighbors. The distributions for the LAB and TVT clusters are mostly centered
around a singular peak; this is especially evident in the latter, where users
predominantly employ non-polarized hashtags and thus form a distinct peak
around zero. On the other hand, the distribution of the CON cluster is
characterized by two separate peaks, one close to 0.5 – similarly to the LAB
cluster, although with opposite alignment – and the other close to zero. This
result suggests that the CON cluster might effectively house at least two sub-
communities, as denoted by the two observed peaks. The peak around 0.5 may be
comprised of coordinated users sharing tweets with a distinct political
connotation, which is determined by hashtags often co-occurring with the
conservative hashtags of polarity $(+1)$ used for data collection (Table 1).
On the other hand, the coordinated users distributed around zero may still be
sharing tweets with a clear conservative perspective, however, their focus
might be on topics characterized by hashtags that are not tightly linked to
our predefined set of conservative hashtags. This observed within-cluster bi-
modality is further strengthened by our previous analysis on the relationship
between polarity and toxicity: the users in these potential sub-communities
not only share tweets with different political content, as suggested by their
hashtag usage but also with different linguistic content, as suggested by the
toxicity they express. Specifically, users sharing more distinctly politically
aligned tweets do so with seemingly non-toxic language. This result further
highlights the nuanced behavior of coordinated users in regard to the topics
they promote and how they promote them from a linguistic viewpoint.

###  3.2 Toxicity in coordinated and non-coordinated users

Figure 3: Comparison between coordinated and non-coordinated users in terms of
expressed toxicity, defined as the average of the top 10% most toxic original
tweets or retweets. (a) Tweeting activity and user toxicity smoothed via a
LOESS curve (the shaded region indicates the corresponding 95% CI), with the
observed user toxicity distribution for both groups in miniature; (b)
bootstrap distribution of the average user toxicity; (c) bootstrap
distribution of the average user toxicity obtained by ignoring retweets.

In our subsequent analysis, we sought to measure the difference in toxicity
exhibited by coordinated and non-coordinated users, applying for both the
“user toxicity” definition we presented in 3.1. For what concerns this
comparison, we only considered users with a minimum of 5 tweets or retweets,
as including all users would generate a user toxicity distribution with a
heavy positive skew for both groups, thus making the comparison less valuable
for an assessment of the overall behavioral differences. Fig. 3(a) shows that
the toxicity expressed by coordinated users remains relatively unchanged
across different activity levels, suggesting that the more active coordinated
users don’t share content that is any more or less toxic. In contrast,
increased activity in non-coordinated users appears to correlate with reduced
toxic behavior.

Examining the user toxicity distributions for both user groups in Fig. 3(a)
further highlights this difference: coordinated users display a distribution
that is more sharply concentrated around its mean
($\overline{x}=0.43,\hat{\sigma}=0.12$), while non-coordinated users are
characterized by larger fluctuations ($\overline{x}=0.46,\hat{\sigma}=0.17$).
In addition, our results provide an indication that coordinated users manifest
on average lower toxicity than their non-coordinated counterparts, which we
have statistically validated ($p<0.001$) by means of the Anderson Darling
$k$-sample test with 10,000 simulations. We can quantify this discrepancy in
average toxicity via bootstrap resampling to account for the inherent
difference in sample size between the two groups. In Fig. 3(b), we report the
distributions resulting from 50,000 bootstrap replicates, which confirm our
previous observation that coordinated users are on average less toxic than the
non-coordinated ones; for the former, we obtain $\hat{\mu}=0.4269$ (95% CI:
$[0.4238,0.4300]$), while for the latter $\hat{\mu}=0.46129$ (95% CI:
$[0.45947,0.46308]$). We argue that this result follows from the difference in
tweeting activity of the two groups. In fact, we have defined coordinated
users as a subset of the most prolific retweeters in the dataset, hence their
activity being mostly characterized by sharing other accounts’ content
suggests that coordinated behaviors may favor promoting content that is less
toxic than average. To assess whether this result effectively stems from the
difference in tweeting patterns, we repeated the same analysis upon exclusion
of retweets, thus solely focusing on original tweets: out of the 5450
coordinated users we initially identified, this filtering reduced the set of
coordinated users to 4515. The densities obtained via bootstrap reported in
Fig. 3(c) indicate that extensive retweeting activity does play a role, and
also elicit an interesting observation: both coordinated and non-coordinated
users tend to post original content with a higher toxicity level compared to
that they disseminate through retweets. In fact, we measure $\hat{\mu}=0.5161$
(95% CI: $[0.5040,0.5281]$) for coordinated users, and $\hat{\mu}=0.4821$ (95%
CI: $[0.4759,0.4883]$) for the non-coordinated. In addition, our analysis
highlights that coordinated users produce original tweets that are distinctly
more toxic than those they spread via retweeting, whereas non-coordinated
users tend to produce and retweet content of similar toxicity. Indeed, unlike
our previous analysis that included retweets, the average toxicity of the
coordinated users is now higher than that estimated for the non-coordinated.
This suggests that coordinated behaviors aiming to maximize exposure might be
strategically orchestrated to disseminate toxic content while remaining wary
of triggering platform moderation mechanisms.

###  3.3 Influence of coordination on toxicity

Figure 4: Average toxicity of tweets produced by non-coordinated users,
obtained with bootstrap resampling. (a) Distributions of the average toxicity
produced following exclusive interactions with non-coordinated users or
coordinated users; (b) estimates with their 95% CI obtained by factoring in
the toxicity of the interactions, using a score of 0.6 as the threshold to
label a tweet as ‘toxic’, and their political leaning. Figure 5: Hourly
average toxicity of the tweets produced by non-coordinated users compared with
that of the tweets by coordinated users they have interacted with, overlaid
with LOESS curves (the shaded regions indicate their respective 95% CI) for
easier visualization. The size and visibility of a point are proportional to
the number of tweets observed within the corresponding hour. On the right
side, empirical distributions of the two metrics across the entire time
period.

We further examine whether coordinated activity directly affects the toxic
behaviors of non-coordinated users. To achieve this, we classify the tweets
each non-coordinated user has been involved in into two main categories:
‘productions’ (i.e., the original tweets the user has written) and
‘interactions’ (i.e., the tweets the user has retweeted or replied to). This
classification allows us to encode user activity within the context of the
electoral debate as a sequence of productions and interactions, which we can
study from a toxicity viewpoint. As we aim to investigate whether interacting
with toxic content influences the toxicity of the following productions, for
each user we discard the first sequence of tweets if they are original
productions and the last sequence if they are interactions. In addition, to
assess the role of CB, we only focus on sequences of interactions that involve
only coordinated or non-coordinated users. Finally, we assign to each sequence
of productions a toxicity score equal to the average toxicity of its tweets.
In Fig. 4(a), we report the density of the average toxicity produced by non-
coordinated users, obtained via bootstrap resampling, upon interacting with
both groups of users. The resulting estimates suggest that non-coordinated
users who have exclusively interacted with coordinated behavior tend to
manifest slightly higher toxicity levels than those who haven’t at all; we
obtain $\hat{\mu}=0.20274$ (95% CI: $[0.20124,0.20425]$) for the former, and
$\hat{\mu}=0.18918$ (95% CI: $[0.18794,0.19042]$) for the latter.

Fig. 4(b) displays how factoring in the toxicity of the content a user has
interacted with affects the previous outcome. In this analysis, we consider a
sequence of interactions to be ‘toxic’ if all the tweets that constitute it
have a toxicity score above 0.6; this is the threshold suggested by
PerspectiveAPI to classify a tweet as toxic, as it indicates with reasonable
confidence that more than half the readers would classify that message as
such. In contrast, a sequence is considered ‘non-toxic’ if all of its tweets
have a toxicity score below 0.6. Finally, we discard all sequences with a mix
of toxic and non-toxic tweets. Our results indicate that the level of toxicity
a user expresses upon interacting with toxic content doesn’t significantly
change when the author of that content is a coordinated user (the two samples
originate from the same population with $p=0.093$, Anderson Darling $k$-sample
test). In fact, the two bootstrap estimates overlap: $\hat{\mu}=0.2374$ (95%
CI: $[0.2293,0.2455]$) following interactions with coordinated users, and
$\hat{\mu}=0.2486$ (95% CI: $[0.2409,0.2561]$) with non-coordinated users. We
measure on the other hand a statistically significant difference ($p<0.001$)
in the case of interactions with non-toxic tweets, however, the increase is
only minimal: $\hat{\mu}=0.20127$ (95% CI: $[0.19970,0.20279]$) following
interactions with coordinated users, and $\hat{\mu}=0.18694$ (95% CI:
$[0.18570,0.18819]$) with non-coordinated users. This suggests that the
observed increase in produced toxicity should likely be attributed to the
nature of the tweets shared by coordinated users. In fact, as coordinated
efforts are typically orchestrated to disseminate and boost specific agendas,
the tweets they spread might be more polarizing or controversial in character
rather than inherently toxic. To investigate this further, we consider the
political alignment of the content a user has interacted with and compare it
to the user’s own alignment: if the average leaning of the interactions has an
opposite sign to that of the user, we consider that sequence of interactions
to have ‘opposite leaning’, otherwise ‘same leaning’. To avoid ties, we
exclude from this analysis all users with a political leaning score equal to
0. Surprisingly, Fig. 4(b) shows that users interacting with content of
opposite political alignment exhibit only marginally higher toxicity levels
than those interacting with politically aligned content. In addition, these
findings point towards coordinated behavior having only a minor role in
affecting the toxicity of tweets produced by non-coordinated users: both the
toxicity level and the political content of the tweets being shared seemingly
have a larger impact than the coordination degree of its author. In fact, our
measurements suggest that the ‘volume’ of tweets, which is characteristic of
coordinated behaviors, has a less pronounced effect on the toxic behavior of
non-coordinated users than the political message they convey and how they
convey it from a toxicity viewpoint.

An alternative approach we can employ to explore how CB might potentially
influence toxic production is to study the phenomenon from a strictly temporal
perspective. In this regard, we construct two-time series spanning the entire
time frame of our data: one is built by sequencing the hourly average toxicity
of original tweets produced by non-coordinated users, while the other by
averaging the toxicity of the tweets they have interacted with that have been
shared by coordinated users. Fig. 5 shows that while the former appears stable
and concentrated around a small range of values, the latter is characterized
by evident oscillations. This hints at coordinated users adopting a noticeably
more toxic and non-stationary tone in the context of specific events in the
campaign (such as TV debates between political leaders) and less toxic in
others. As we have suggested in 3.2, a possible explanation for this behavior
might reside in CB being specifically employed to inject inflammatory content
or trigger toxic responses while simultaneously avoiding the platform
moderation mechanism.

Additionally, from Fig. 5 we can observe that the two-time series nearly
mirror each other in proximity to election day, which might serve as
compelling evidence of one influencing the other. To verify this, we can
quantitatively measure the information flow between the two by applying the
transfer entropy method [39], which is based on the concept of entropy. Unlike
other methods such as Granger causality [40], transfer entropy can capture not
only non-linear dependencies between two time series but also the dominant
direction of the flow, by estimating the net gain in information about the
future observations of one derived from the past observations of the other. As
entropy requires discrete data, we discretize the average toxicity scores of
the two-time series and assign them to 4 bins with bounds equal to the
quantiles at $\mathbf{p}=(0.05,0.5,0.95)$ of their respective empirical
distributions. The selection of these quantiles is made to emphasize the tails
of the distributions, as they include the least and most toxic observations;
to this end, we compute the Rényi transfer entropy with a weighting parameter
of $q=0.5$, which puts more weights on the tails when calculating transfer
entropy [41]. Our findings indicate that the toxicity shared via CB affects
the toxic tendencies of non-coordinated users and that the information flow
from the former to the latter is statistically relevant ($p=0.034$), as tested
using the method proposed by Dimpfl and Peter (2013) [42] and implemented by
Behrendt et al. (2019) [41], whereas that in the opposite direction is not
($p=0.78$). This is an additional indication of CB’s role, suggesting that
even passive interactions with coordinated activities can significantly shape
the toxicity levels expressed by non-coordinated users.

Our findings expose a marginal yet measurable shift in the production of toxic
content when non-coordinated users engage with CB. However, this increase is
likely attributed to the nature of the content being shared, suggesting that
the ‘character’ of the message might be more influential than the extent of
coordination of the efforts that channeled it. Additionally, our results are
somewhat counterintuitive concerning the current assumption regarding
political polarization.

##  4 Conclusions

In the context of the 2019 UK General Elections, we systematically analyze the
interplay between Coordinated Behavior (CB) and content toxicity on Twitter.
Drawing from a dataset of 11 million tweets from 1 million unique users, we
aim to better understand the subtleties of this relationship on a significant
digital communication platform. Our findings underscore that users with higher
coordination scores predominantly circulate content with diminished toxicity
levels. This trend is pronounced in politically-affiliated clusters,
indicating that coordination, while strategic, does not necessarily result in
promoting harmful content. The primary aim of these coordinated users is
amplification and influence rather than explicit toxicity dissemination. Clear
patterns emerge when analyzing coordinated versus non-coordinated users: both
groups exhibit comparable average toxicity levels. However, coordinated users
maintain consistent toxicity levels regardless of their activity volume.
Conversely, increased activity among non-coordinated users typically aligns
with decreased toxic behavior, suggesting a more intentional toxicity
regulation by coordinated entities.

In examining the non-coordinated user group, we find that exposure to CB
subtly heightens their toxic behavior. Notably, this rise is not solely driven
by the volume of toxic exposure but is significantly influenced by the
character of the content. Political alignment, while a factor in toxicity,
does not show a direct linear relation with toxicity intensity. From a
temporal perspective, using Renyi entropy reveals the immediacy of CB’s impact
on non-coordinated user toxicity. A discernible flow from exposure to
production suggests CB’s subtle influence in shaping regular user behaviors.

In conclusion, our study highlights the nuanced manner in which CB functions
online. While coordination does not directly spread toxicity, its influence on
user behaviors is evident. The interplay between content nature and its
dissemination strategy is crucial. Future research should investigate the
motivations behind coordinated activities, the socio-political environments
nurturing them, and their broader effects on digital conversations.

## Tables

Table 1: Data collected via hashtags [20]. Neutral (N) hashtags have been assigned a political leaning score of $0$, whereas hashtags linked to the Labour (L) and Conservative (C) parties have been assigned a score of $-1$ and $+1$, respectively. Hashtag | Leaning | Users | Tweets
---|---|---|---
#GE2019 | N $(0)$ | 436,356 | 2,640,966
#GeneralElection19 | N $(0)$ | 104,616 | 274,095
#GeneralElection2019 | N $(0)$ | 240,712 | 783,805
#VoteLabour | L $(-1)$ | 201,774 | 917,936
#VoteLabour2019 | L $(-1)$ | 55,703 | 265,899
#ForTheMany | L $(-1)$ | 17,859 | 35,621
#ForTheManyNotTheFew | L $(-1)$ | 22,966 | 40,116
#ChangeIsComing | L $(-1)$ | 8,170 | 13,381
#RealChange | L $(-1)$ | 78,285 | 274254
#VoteConservative | C $(+1)$ | 52,642 | 238,647
#VoteConservative2019 | C $(+1)$ | 13,513 | 34,195
#BackBoris | C $(+1)$ | 36,725 | 157,434
#GetBrexitDone | C $(+1)$ | 46,429 | 168,911
Total |  | 668,312 | 4,983,499
Table 2: Data collected via accounts [20]. The two accounts linked to the Labour party (L) have been assigned a political leaning score of $-1$, whereas the two linked to the Conservative party (C) a score of $+1$. | Production | Interactions
---|---|---
Account | Leaning | Tweets | Retweets | Replies
@jeremycorbyn | L $(-1)$ | 788 | 1,759,823 | 414,158
@UKLabour | L $(-1)$ | 1,002 | 325,219 | 79,932
@BorisJohnson | C $(+1)$ | 454 | 284,544 | 382,237
@Conservatives | C $(+1)$ | 1,398 | 151,913 | 169,736
Total |  | 3,642 | 2,521,499 | 1,046,063

## References

  * [1] Carlo M Valensise, Matteo Cinelli, and Walter Quattrociocchi.  The drivers of online polarization: Fitting models to data.  Information Sciences, 642:119152, 2023.
  * [2] Michela Del Vicario, Gianna Vivaldo, Alessandro Bessi, Fabiana Zollo, Antonio Scala, Guido Caldarelli, and Walter Quattrociocchi.  Echo chambers: Emotional contagion and group polarization on facebook.  Scientific reports, 6(1):37825, 2016.
  * [3] Ferenc Huszár, Sofia Ira Ktena, Conor O’Brien, Luca Belli, Andrew Schlaikjer, and Moritz Hardt.  Algorithmic amplification of politics on twitter.  Proceedings of the National Academy of Sciences, 119(1):e2025334119, 2022.
  * [4] S. González-Bailón, D. Lazer, P. Barberá, and et al.  Asymmetric ideological segregation in exposure to political news on facebook.  Science, 381(6656):392–398, 2023.
  * [5] A.M. Guess, N. Malhotra, J. Pan, and et al.  How do social media feed algorithms affect attitudes and behavior in an election campaign?  Science, 381(6656):398–404, 2023.
  * [6] A.M. Guess, N. Malhotra, J. Pan, and et al.  Reshares on social media amplify political news but do not detectably affect beliefs or opinions.  Science, 381(6656):404–408, 2023.
  * [7] Michela Del Vicario, Alessandro Bessi, Fabiana Zollo, Fabio Petroni, Antonio Scala, Guido Caldarelli, H Eugene Stanley, and Walter Quattrociocchi.  The spreading of misinformation online.  Proceedings of the national academy of Sciences, 113(3):554–559, 2016.
  * [8] David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, et al.  The science of fake news.  Science, 359(6380):1094–1096, 2018.
  * [9] Alexandre Bovet and Hernán A Makse.  Influence of fake news in twitter during the 2016 us presidential election.  Nature communications, 10(1):7, 2019.
  * [10] Jonas L Juul and Johan Ugander.  Comparing information diffusion mechanisms by matching on cascade size.  Proceedings of the National Academy of Sciences, 118(46):e2100786118, 2021.
  * [11] Francesco Pierri, Luca Luceri, Nikhil Jindal, and Emilio Ferrara.  Propaganda and misinformation on facebook and twitter during the russian invasion of ukraine.  In Proceedings of the 15th ACM Web Science Conference 2023, pages 65–74, 2023.
  * [12] R. Kelly Garrett.  Echo chambers online?: Politically motivated selective exposure among internet news users.  Journal of Computer-Mediated Communication, 14(2):265–285, 2009.
  * [13] Christopher A Bail, Lisa P Argyle, Taylor W Brown, John P Bumpus, Haohan Chen, MB Fallin Hunzaker, Jaemin Lee, Marcus Mann, Friedolin Merhout, and Alexander Volfovsky.  Exposure to opposing views on social media can increase political polarization.  Proceedings of the National Academy of Sciences, 115(37):9216–9221, 2018.
  * [14] Joshua A Tucker, Andrew Guess, Pablo Barberá, Cristian Vaccari, Alexandra Siegel, Sergey Sanovich, Denis Stukal, and Brendan Nyhan.  Social media, political polarization, and political disinformation: A review of the scientific literature.  Political polarization, and political disinformation: a review of the scientific literature (March 19, 2018), 2018.
  * [15] Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini.  The echo chamber effect on social media.  Proceedings of the National Academy of Sciences, 118(9):e2023301118, 2021.
  * [16] Max Falkenberg, Alessandro Galeazzi, Maddalena Torricelli, Niccolò Di Marco, Francesca Larosa, Madalina Sas, Amin Mekacher, Warren Pearce, Fabiana Zollo, Walter Quattrociocchi, et al.  Growing polarization around climate change on social media.  Nature Climate Change, pages 1–8, 2022.
  * [17] Justin Cheng, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec.  Antisocial behavior in online discussion communities.  In Proceedings of the international aaai conference on web and social media, volume 9, pages 61–70, 2015.
  * [18] Matteo Cinelli, Andraž Pelicon, Igor Mozetič, Walter Quattrociocchi, Petra Kralj Novak, and Fabiana Zollo.  Dynamics of online hate and misinformation.  Scientific reports, 11(1):22083, 2021.
  * [19] Yonatan Lupu, Richard Sear, Nicolas Velásquez, Rhys Leahy, Nicholas Johnson Restrepo, Beth Goldberg, and Neil F. Johnson.  Offline events and online hate.  PLOS ONE, 18(1):1–14, 2023.
  * [20] Leonardo Nizzoli, Serena Tardelli, Marco Avvenuti, Stefano Cresci, and Maurizio Tesconi.  Coordinated behavior on social media in 2019 uk general election.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 443–454, 2021.
  * [21] Kate Starbird.  Disinformation’s spread: bots, trolls and all of us.  Nature, 571(7766):449–450, 2019.
  * [22] Luis Vargas, Patrick Emami, and Patrick Traynor.  On the detection of disinformation campaign activity with network analysis.  In Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop, CCSW’20, page 133–146, New York, NY, USA, 2020. Association for Computing Machinery.
  * [23] Kai Shu, Suhang Wang, Dongwon Lee, and Huan Liu.  Mining disinformation and fake news: Concepts, methods, and recent advancements.  In Disinformation, misinformation, and fake news in social media. Springer, 2020.
  * [24] Franziska B. Keller, David Schoch, Sebastian Stier, and JungHwan Yang.  Political astroturfing on twitter: How to coordinate a disinformation campaign.  Political Communication, 37(2):256–280, 2020.
  * [25] Diogo Pacheco, Pik-Mai Hui, Christopher Torres-Lugo, Bao Tran Truong, Alessandro Flammini, and Filippo Menczer.  Uncovering coordinated networks on social media: methods and case studies.  In Proceedings of the international AAAI conference on web and social media, volume 15, pages 455–466, 2021.
  * [26] Matteo Cinelli, Stefano Cresci, Walter Quattrociocchi, Maurizio Tesconi, and Paola Zola.  Coordinated inauthentic behavior and information spreading on twitter.  Decision Support Systems, 160:113819, 2022.
  * [27] David Schoch, Franziska B Keller, Sebastian Stier, and JungHwan Yang.  Coordination patterns reveal online political astroturfing across the world.  Scientific reports, 12(1):4572, 2022.
  * [28] Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu.  Defend: Explainable fake news detection.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 395–405, New York, NY, USA, 2019. Association for Computing Machinery.
  * [29] Somya Ranjan Sahoo and B.B. Gupta.  Multiple features based approach for automatic fake news detection on social networks using deep learning.  Applied Soft Computing, 100:106983, 2021.
  * [30] Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi.  Hate me, hate me not: Hate speech detection on facebook.  In Italian Conference on Cybersecurity, 2017.
  * [31] Google Jigsaw.  Perspective api, 2022.
  * [32] M. Ángeles Serrano, Marián Boguñá, and Alessandro Vespignani.  Extracting the multiscale backbone of complex weighted networks.  Proceedings of the National Academy of Sciences, 106(16):6483–6488, 2009.
  * [33] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre.  Fast unfolding of communities in large networks.  Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
  * [34] Juan Martinez-Romo, Lourdes Araujo, Javier Borge-Holthoefer, Alex Arenas, José A Capitán, and José A Cuesta.  Disentangling categorical relationships through a graph of co-occurrences.  Physical Review E, 84(4):046108, 2011.
  * [35] Alexandre Bovet, Flaviano Morone, and Hernán A Makse.  Validation of twitter opinion trends with national polling aggregates: Hillary clinton vs donald trump.  Scientific reports, 8(1):8673, 2018.
  * [36] Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec.  Anyone can become a troll: Causes of trolling behavior in online discussions.  In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, pages 1217–1230, 2017.
  * [37] Mark EJ Newman.  Mixing patterns in networks.  Physical review E, 67(2):026126, 2003.
  * [38] Aris Anagnostopoulos, Ravi Kumar, and Mohammad Mahdian.  Influence and correlation in social networks.  In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, pages 7–15, New York, NY, USA, August 2008. Association for Computing Machinery.
  * [39] Thomas Schreiber.  Measuring Information Transfer.  Physical Review Letters, 85(2):461–464, July 2000.
  * [40] C. W. J. Granger.  Investigating Causal Relations by Econometric Models and Cross-spectral Methods.  Econometrica, 37(3):424–438, 1969.
  * [41] Simon Behrendt, Thomas Dimpfl, Franziska J. Peter, and David J. Zimmermann.  RTransferEntropy — Quantifying information flow between different time series using effective transfer entropy.  SoftwareX, 10:100265, July 2019.
  * [42] Thomas Dimpfl and Franziska Julia Peter.  Using transfer entropy to measure information flows between financial markets.  Studies in Nonlinear Dynamics and Econometrics, 17(1), January 2013.