# Sparse then Prune: Toward Efficient Vision Transformers

Yogi Prasetyo  Novanto Yudistira  Agus Wahyu Widodo

###### Abstract

The Vision Transformer architecture is a deep learning model inspired by the
success of the Transformer model in Natural Language Processing. However, the
self-attention mechanism, large number of parameters, and the requirement for
a substantial amount of training data still make Vision Transformers
computationally burdensome. In this research, we investigate the possibility
of applying Sparse Regularization to Vision Transformers and the impact of
Pruning, either after Sparse Regularization or without it, on the trade-off
between performance and efficiency. To accomplish this, we apply Sparse
Regularization and Pruning methods to the Vision Transformer architecture for
image classification tasks on the CIFAR-10, CIFAR-100, and ImageNet-100
datasets. The training process for the Vision Transformer model consists of
two parts: pre-training and fine-tuning. Pre-training utilizes ImageNet21K
data, followed by fine-tuning for 20 epochs. The results show that when
testing with CIFAR-100 and ImageNet-100 data, models with Sparse
Regularization can increase accuracy by 0.12%. Furthermore, applying pruning
to models with Sparse Regularization yields even better results. Specifically,
it increases the average accuracy by 0.568% on CIFAR-10 data, 1.764% on
CIFAR-100, and 0.256% on ImageNet-100 data compared to pruning models without
Sparse Regularization. Code can be accesed here:
https://github.com/yogiprsty/Sparse-ViT

###### keywords:

Vision Transformer , Sparse Regularization , Pruning

††journal: Pattern Recognition Letters

[inst1]organization=Informatics Department, Faculty of Computer Science,
Universitas Brawijaya,addressline=Jalan Veteran 8, city=Malang,
postcode=65145, state=Malang, country=Indonesia

##  1 Introduction

Modern information technology advancements make it simpler and faster for
people to tackle a variety of difficulties. Humans produce a lot of data daily
due to its broad usage, including search engine activity, online photo and
video uploads, and more. One effective approach to automate data
classification is by utilizing Deep Learning technology. Recently, the most
popular deep learning architecture in digital image processing is the
Convolutional Neural Network (CNN) due to its success in the ILSVRC 2014
GoogleNet challenge[1]. However, this architecture generally requires a
relatively high computational time, making it possible for new research to
develop an architecture that can outperform CNN.

The self-attention-based architecture, especially the Transformer, has become
the preferred choice in natural language processing. In some cases, the
Transformer significantly outperforms convolutional and recurrent layer-based
architectures[2]. Not only in natural language processing, but the use of
Transformers is also starting to expand, including in digital image
processing. To apply the Vision Transformer, start by dividing the image into
patches or small pieces and provide the sequence of positions of these patches
as input to the Transformer layer. Patches of images are treated like tokens
in natural language processing applications. This method is five times faster
than EfficientNet, which is based on a Convolutional Neural Network. The
accuracy of the Vision Transformer model will increase significantly if it is
pre-trained with large data, such as ImageNet21k, before using it for simpler
cases[3].

In addition to researching better architectures, researchers have proposed
various methods to improve accuracy or speed up computing in Deep Learning
models. The widely used methods include Regularization and Pruning.
Regularization is an additional technique that aims to enhance the model’s
generalization, resulting in better results on test data[4]. On the other
hand, pruning is a popular technique used to obtain a more compact model by
removing weights considered less important[5]. This approach allows for faster
computation time while maintaining the model’s accuracy.

Based on the above background, this research applies Sparse Regularization and
Pruning methods to the Vision Transformer architecture for image
classification tasks utilizing the CIFAR 10 and CIFAR 100 datasets. The CIFAR
10 dataset comprises 60,000 images with a resolution of 32x32, divided into
ten distinct classes, each consisting of 6,000 images. In contrast, the CIFAR
100 dataset includes 60,000 images of the same resolution but encompasses a
broader spectrum of classes, specifically 100 classes. As an open-source
dataset extensively utilized in Deep Learning research, CIFAR datasets serve
as an ideal testbed for evaluating the proposed techniques and their impact on
image classification tasks.

##  2 Related Work

Research conducted by [3], titled "An Image is Worth 16x16 Words: Transformers
for Image Recognition at Scale," attempts to apply pre-training to the Vision
Transformer architecture using varying amounts of data. Consequently, when
trained on small data, the Vision Transformer model exhibits poor performance.
However, when trained on large data, this model demonstrates superior
performance and surpasses the ResNet model. Other research also shows the same
conclusion when the Vision Transformer is used to classify mask usage using
the MaskedFace-Net dataset. As a result, the pre-trained ViT-H/14 model
performed better than the ViT-H/14 model without pre-trained, with 1.87%
higher on test accuracy[6]. The findings of this research have inspired the
utilization of pre-trained models in the present study.

Much research has been done to improve Vision Transformer-based architecture.
The Data-efficient image Transformers (DeiT) [7] introduce a CNN model as a
teacher and apply knowledge distillation[8] to enhance the student model of
ViT in order to lower its dependence on a huge amount of data. To obtain
satisfactory results thus, DeiT only can be trained on ImageNet. Additionally,
the CNN models and distillation types selected may affect the final
performance. The Convolution enhanced Image Transformer (CeiT)[9] proposed to
overcome those limitations. This method works by combining the advantages of
CNN in low-feature extraction, strengthening the locality of the advantages of
Transformers in establishing long-range dependencies. The Convolutional Vision
Transformer (CvT)[10] proposed to improve the performance and efficiency of
ViTs by introducing convolutions into the ViT architecture. This design
introduces convolutions in two core sections of ViT: a hierarchy of
Transformers containing a new convolutional token embedding and a
convolutional Transformer block utilizing a convolutional projection.

Another method called sharpness-aware minimizer (SAM) was proposed and
utilized to explicitly smooths the loss geometry during the training
process[11]. SAM seeks to find solutions that the entire environment has low
losses rather than focusing on any single point. The Transformer-iN-
Transformer (TNT)[12] was proposed to enhance the ability of feature
representation in ViT by dividing the input images into several patches as
"visual sentence" and then divide those patches into sub-patches, as the
representation of "visual word". Apart from using conventional transformer
blocks to extract features and attention of visual sentences, sub-transformers
are also embedded into the architecture to excavate the features and details
of smaller visual words.

Another research conducted by [13], titled "Improvement of Learning for CNN
with ReLU Activation by Sparse Regularization," applies the Sparse
Regularization method to ReLU inputs. The findings demonstrate a significant
increase in accuracy, ranging from 9.98% to 12.12%, compared to models without
Sparse Regularization. Additionally, compared to the Batch Normalization
method, the accuracy improves by 4.64% to 6.87%. These compelling results have
inspired using the Sparse Regularization method in the present study.

The research conducted by [14], titled "Vision Transformer Pruning," applies
the Pruning method to the Vision Transformer architecture. The findings
demonstrate that this method reduces FLOPS (Floating Point Operations per
Second) by 55.5% when applied to the Vision Transformer Base 16 architecture.
Moreover, the accuracy only decreases by 2% on ImageNet-1K data. These
promising results have inspired using the Pruning method in the present study.

Previous studies have shown that Vision Transformer active neurons are highly
sparse. This implies that only a small portion of neurons within a layer or
network become active or "fire" at any given moment. In contrast, most neurons
remain inactive or have low activation values[11]. These findings indicate
that less than 10% of neurons in a Vision Transformer model have values
greater than zero. In simpler terms, this highlights the significant potential
of Vision Transformer models for network pruning.

##  3 Method

###  3.1 Datasets

We utilized the widely recognized CIFAR-10 and CIFAR-100 datasets as standard
benchmarks during the experiments. CIFAR-10 comprises RGB color images
categorized into ten classes for object classification, while CIFAR-100
consists of 100 different classes. Each image in these datasets had been
standardized to a resolution of 32x32 pixels. This consistency in size and
format allowed us to focus solely on the intricacies of the data and the
impact of various regularization methods on model performance. The
standardized resolution facilitated seamless comparisons between different
models and enabled us to discern subtle nuances that might have otherwise gone
unnoticed. CIFAR datasets is commonly used in research for evaluating
performance. The research conducted by [15] employs the CIFAR-10 and CIFAR-100
datasets to evaluate the effectiveness of this regularization method. By
conducting experiments on CIFAR datasets, researchers can gauge the impact of
regularization methods on model performance and compare them to existing
approaches.

###  3.2 Vision Transformer

The Transformer is an attention-based model published in 2017 to replace the
Recurrent layer-based architecture, addressing issues with gradients and
relatively long computation time[2]. The Vision Transformer is an architecture
inspired by the success of the Transformer in natural language processing. In
2020, Transformers were employed in image classification problems and
demonstrated superior performance compared to CNN-based architectures, such as
EfficientNet[3]. Figure 1 illustrates the architecture of the Vision
Transformer.

Unlike state-of-the-art architecture, The process begins by breaking the 2D
images into several parts and converting them into a sequence of flattened 2D
patches or 1-dimensional vector, shown in Equation 1, where $(H,W)$ is the
resolution of the original image, $C$ is the number of channels, $P$ is the
number of patches and $N=HW|P^{2}$. During this process, position embedding is
added to the vectors, and these vectors are combined with an additional vector
(class token) that will serve as the output of the Vision Transformer
architecture.

Images that have undergone the Patch and Embedding process will pass through
Transformer Encoder, which includes the Multi-Head Self-Attention (MSA) layer
and the Multi-Layer Perceptron (MLP). The normalization layer is applied
before the MSA and the MLP. Layer normalization is used to reduce the
computation time of the artificial neural network. This layer operates by
calculating the average and variance of each data input, which are then used
to normalize all inputs [16]. Equation 2 represents the formula for the
normalization layer, where $x_{i,k}$ is vector input.

Figure 1: Vision Transformer[3]

$x\in R^{HWC}\rightarrow x_{p}\in R^{N(P^{2}C)}$ (1)

$\hat{x}_{i,k}=\frac{x_{i,k}-\mu^{i}}{\sqrt{\sigma_{i}^{2}+\epsilon}}$ (2)

The result will pass through Multi-Head Attention, and this layer works by
dividing the input into several heads, allowing each head to learn a different
level of Attention. The heads’ outputs are combined and forwarded to the
Multi-Layer Perceptron. Equation 3 represents the formula for the Scaled Dot-
Product Attention, while Equation 4 represents the formula for the Multi-Head
Attention.

$head(Q,K,V)=softmax\left(\frac{QK^{t}}{\sqrt{d_{k}}}\right)V$ (3)

$MultiHead(Q,K,V)=Concat(head_{1}\,...\,head_{n})$ (4)

The last layer of transformer encoding is the multilayer perceptron (MLP), an
artificial neural network architecture with one or more hidden layers between
the input and output layers. Training an MLP model involves adjusting the
weights and biases between neurons at different layers to minimize the
difference between the model’s output and the desired output[17]. The MLP
layer used Gaussian Error Linear Unit (GELU). In certain datasets, these non-
linear activation functions can match and outperform linear functions such as
ReLU and ELU[18]. The equation for the GELU activation function can be seen in
Equation 5.

$GELU(x)=0.5x\left(1+tanh\left[\sqrt{2/\pi}\left(x+0.044715x^{3}\right)\right]\right)$
(5)

###  3.3 Sparse Regularization

Sparse regularization is a leading technique in deep learning research that
aims to improve the performance and efficiency of models by promoting sparsity
in the learned weights. This regularization method encourages a subset of the
weights to become zero or close to zero, resulting in a more compact and
interpretable model[13]. Also, this method has a similar effect to Batch
Normalization and can improve the model’s ability to predict more general
data. Evaluating sparseness can be done in various ways. However, in the
research, sparse regularization is applied to the ReLU input using Equation 6,
where $h_{k}$ is the input the $k$-th ReLU.

$S(h_{k})=log(1+h_{k}^{2})$ (6)

Once the sparse value is obtained, it is multiplied by $\lambda$, a parameter
that controls sparseness, before being added to the loss value $L$. Further
details can be found in Equation 7.

$E=L+\lambda\sum_{k}S(h_{k})$ (7)

(a)

(b)

Figure 2: Choices to put sparse regularization

Many explorations have taken place in the field of regularization techniques
within deep learning to determine optimal placements within models. Previous
studies have provided insights into the effectiveness of sparse regularization
in convolutional neural network (CNN) models, particularly when applied to the
input of ReLU[13]. Figure 2 demonstrates a range of interesting options for
strategically incorporating sparse regularization within the model
architecture, each offering unique insights and potential benefits.

Option 1 suggests an interesting placement for sparse regularization, where
sparsity is evaluated after multiplying queries and keys, followed by scaling,
also known as the similarity score. In option 2, sparsity is evaluated
precisely at the attention weight stage. This value is obtained after the
similarity score passes through the softmax activation function. Option 3
applied sparse regularization after multiplying attention weight with V
(value), also known as weighted value. Option 4, sparsity, is evaluated after
the linear or output layer in Multi-Head Attention. Last option, according to
previous research, we will apply sparse regularization to the input of GELU
activation function in MLP layer.

###  3.4 Pruning

Pruning is a method used to speed up computation in deep learning models by
removing the least important parameters. Various approaches for deleting
unneeded parameters can be used, such as using the $l_{1}$-norm or
$l_{2}$-norm to select the target parameters for deletion[5]. Pruning is
broadly classified into unstructured and structured pruning, each with its
strategy for parameter reduction[19].

Structured pruning operates by systematically removing structured components
within deep learning models. These components may manifest as filters[20],
channels, or even entire layers, enabling a more holistic approach to
parameter reduction[21]. By selectively excising these structured elements,
deep learning models can be streamlined, improving efficiency and reducing
computational complexity.

In contrast, unstructured pruning (algorithm 1) focuses on individual weights
within layers or filters. This surgical precision allows for the precise
removal of individual weights without disrupting the overall arrangement of
the deep learning model. Unstructured pruning can still achieve significant
parameter reduction while maintaining the accuracy of the pruned network[22],
resulting in faster computations and optimized model performance.

Input : model, pruning_ratio

Output : model

parameters $\leftarrow$ [];

for  _param in model.parameters()_ do

parameters.append(param);

all_parameters $\leftarrow$ torch.cat([param.data.view(-1) for param in
parameters]);

threshold $\leftarrow$ torch.kthvalue(torch.abs(all_parameters),
int(pruning_ratio * all_parameters.numel())).values;

for  _param in parameters_ do

param_data $\leftarrow$ param.data;

mask $\leftarrow$ torch.abs(param_data) >threshold;

pruned_param $\leftarrow$ param_data * mask.float();

param.data $\leftarrow$ pruned_param;

return _model_ ;

Algorithm 1 Global Pruning with L1 Unstructured Method

###  3.5 Transfer Learning

The Vision Transformer performs remarkably when complemented by a pre-training
process known as transfer learning [3]. Transfer learning involves training
the model on a large dataset before applying it to other data, leveraging the
knowledge acquired from the pre-training phase to enhance performance on new
tasks. In this context, the abundance of data and its diverse nature play
pivotal roles in facilitating effective transfer learning. Large amounts of
data and a wide diversity are the most influential factors for transfer
learning [23]. To exploit the full potential of transfer learning, this
research endeavors to capitalize on the Vision Transformer architecture, which
has undergone extensive training using ImageNet21k data. ImageNet21k boasts an
impressive collection of 14,197,122 annotated images, accurately organized
according to the WordNet hierarchy[24]. This vast and diverse dataset
encapsulates numerous objects, scenes, and concepts, providing rich visual
information for the Vision Transformer to learn from.

###  3.6 Model Training

The ViT architecture can be configured in multiple ways. To ensure unbiased
test results for each configuration, constant hyperparameters will be set
during training for the ViT architecture’s various configurations. For this
research, we will maintain the same setup used in the original ViT research,
as described in reference [3]. During the fine-tuning process, the following
hyperparameters were employed: a batch size of 64, a learning rate of 0.03, 20
epochs, a Cross-Entropy loss function[25], Stochastic Gradient Descent
optimizer, and GeLU activation function[18]. For transfer learning, we
utilized pre-trained weights initially trained on the ImageNet21K dataset[26].
The experiments were conducted using hardware specifications that included an
RTX 8000 GPU, an Intel(R) Xeon(R) Gold 6230R processor, and 255 GiB of RAM.

##  4 Experiment

To confirm the effectiveness of the implemented method, we have performed
experiments on the testing of the Vision Transformer using CIFAR-10 and
CIFAR-100 datasets.

###  4.1 ViT Architecture

According to the original research paper, the authors introduced several
Vision Transformer (ViT) model variants. They explored different model
configurations to investigate the impact of architectural choices on its
performance. There are three variants of the vision transformer model. The
first is ViT-Base. This variant serves as the baseline model, featuring a
transformer architecture with a moderate number of layers and attention heads.
It serves as a reference point for comparing the performance of other ViT
models. The Second is ViT-large. This variant extends the ViT model with more
layers and attention heads, increasing its capacity and potential for
capturing complex visual patterns. ViT-large aims to achieve higher accuracy
by leveraging a deeper and more parameter-rich architecture. The last is ViT-
Huge; as the name suggests, this variant represents an even more expansive and
powerful instantiation of the Vision Transformer. It features significantly
larger layers and attention heads, providing a massive capacity for learning
intricate visual representations. We have done the experiments using ViT-B/16
architecture. More detailed configuration and hyperparameter for Vision
Transformer are shown in the Tabel 1.

Table 1: Configurations and Hyperparameters for ViT-B-16 Configuration | Value
---|---
image resolution | 384 $\times$ 384
patch resolution | 16 $\times$ 16
learning rate | 0.001
weight decay | 0.0001
batch size | 16
hidden size | 768
mlp size | 3072
#heads | 12
encoder length | 12

###  4.2 Sparse Regularization Effect

We have performed experiments to evaluate the effectiveness of sparse
regularization. There are five scenarios for implementing sparse
regularization: similarity score, attention weight, weighted value, the output
layer, and the input GELU activation function at the MLP layer. The $\lambda$
value used is $\frac{1}{n\\_feature}$. Before testing, the vision transformer
model will be fine-tuned on the Cifar 10 and Cifar 100 datasets. The Vision
Transformer model without using the sparse regularization method will also be
fine-tuned with the same data for comparison. The comparison of sparse
regularization placement in the Vision Transformer model is shown in Table 2.

Table 2: Result on CIFAR-10 Layer | Sparse Position | Acc
---|---|---
- | - | 98.83
attention | similarity score | 98.57
attention | attention weight | 98.81
attention | weighted value | 98.73
attention | output | 98.33
MLP | input GELU | 98.52

The experiment used the CIFAR-10 dataset and the Vision Transformer model,
which had been fine-tuned for 20 epochs and applied with sparse
regularization. As a result, after calculating the Attention Weight, the model
achieved the highest accuracy of 98.81% when sparse regularization was applied
to the Self-Attention layer. On the other hand, the lowest accuracy of the
model with sparse regularization was 98.33% when the sparse regularization was
applied after the output layer calculation in the Self-Attention layer.
However, despite sparse regularization, the best results from the model still
could not outperform the baseline model, which achieved an accuracy of 98.81%.
The comparison of sparse regularization placement in the Vision Transformer
model is shown in Table 2.

Table 3: Result on CIFAR-100 Layer | Sparse Position | Acc
---|---|---
- | - | 92.39
attention | similarity score | 91.51
attention | attention weight | 92.52
attention | weighted value | 92.17
attention | output | 92.13
MLP | input GELU | 91.73

The second experiment was conducted using the CIFAR-100 dataset and the Vision
Transformer model, which had been fine-tuned for 20 epochs and applied with
sparse regularization. As a result, the model achieved the highest accuracy of
92.52% when sparse regularization was applied to the Self-Attention layer
after calculating the Attention Weight. On the other hand, the lowest accuracy
was 91.73% when the sparse regularization was applied to the MLP layer before
calculating the GELU activation function. The model with sparse regularization
achieved an accuracy increase of 0.12%, outperforming the baseline model. This
indicates that data with more classes makes the vision transformer model have
many sparse active neurons. The pruning method will work better if it is
applied to CIFAR-100 datasets.

###  4.3 Pruning Effect

To evaluate the impact of pruning on the performance of the Vision
Transformer, we have conducted a comprehensive series of experiments by
exploring the effects of varying pruning percentages on both the CIFAR-10 and
CIFAR-100 datasets. Through these experiments, we strive to gauge the
influence of pruning on accuracy and identify optimal pruning thresholds for
the Vision Transformer architecture. We use a range of pruning percentages to
achieve this goal, from 10% to 30% of the model weight. This systematic
approach allowed us to evaluate the pruned Vision Transformer’s performance
under different weight reduction levels. The pruning was applied globally,
ensuring a uniform impact across the entire model. The comparison of accuracy
results using the pruning method on CIFAR-10 and CIFAR-100 datasets is shown
in Figure 3.

Figure 3: Pruning effect on CIFAR-10 and CIFAR-100

The experiment was conducted using CIFAR-10 and CIFAR-100 datasets with the
ViT-B-16 model, which had been fine-tuned for 20 epochs. After training the
model, pruning was performed on all layers with weights. The weights were
determined to be pruned using the $l_{1}$-norm method. As a result, both
datasets showed a negative correlation between the decrease in accuracy and
the percentage of pruned parameters: the greater the number of pruned
parameters, the lower the accuracy. These results underscore the need for
careful consideration and optimization during pruning. While pruning offers
potential for model compression and computational efficiency, it must be done
judiciously to reduce the loss of accuracy. Researchers can navigate this
trade-off by leveraging techniques such as the L1-norm method to determine the
weights to trim, seeking the ideal balance between model cohesiveness and
performance.

###  4.4 Effect of Sparse then Prune

Based on previous experiments, the best accuracy was achieved by applying
sparse regularization after calculating the attention weight on the Self
Attention layer.

(a)

(b)

Figure 4: (a) Result on CIFAR-10. (b) Result on CIFAR-100

Therefore, in this test, the model was trained with sparse regularization
placed after calculating the attention weight on the Self Attention layer
before pruning. Similar to the previous test, the pruning percentage used
ranged from 10% to 30% and was applied globally.

Pruning performed on models with sparse regularization produces higher
accuracy than those without sparse regularization. This indicates that sparse
regularization can effectively distinguish between important and unimportant
parameters or weights. Consequently, when unimportant parameters are pruned or
deleted, the model with sparse regularization demonstrates better data
classification capabilities than the baseline model. Figures 4(a) and 4(b)
compare the accuracy of pruning with and without sparse regularization on the
Vision Transformer model. In the case of CIFAR-10, the average difference in
accuracy is 0.568%, while for CIFAR-100, the average difference in accuracy is
1.764%.

###  4.5 Result on CIFAR-10 and CIFAR-100

All models are pre-trained on ImageNet before being used in downstream tasks
with smaller datasets. A comparison of the Transformer-based specification
model can be seen in Table 4. The model with $\downarrow$10% means the model
is pruned with a 10% pruning ratio. Besides, the model with $\uparrow$384 is
fine-tuned on a bigger resolution, which is 384$\times$384 pixel.

Table 4: Model Specification Model | #params | Image |  |
---|---|---|---|---
Size | Hidden size | #heads |  |
ViT-B/16-Sparse | 86M | 384 | 768 | 12
ViT-B/16-Sparse$\downarrow$10% | 77M | 384 | 768 | 12
ViT-B/16-Sparse$\downarrow$15% | 73M | 384 | 768 | 12
ViT-B/16-Sparse$\downarrow$20% | 69M | 384 | 768 | 12
ViT-B/16-Sparse$\downarrow$25% | 64M | 384 | 768 | 12
ViT-B/16-Sparse$\downarrow$30% | 60M | 384 | 768 | 12
ViT-B/16-SAM | 87M | 244 | 768 | 12
DeiT-B$\uparrow$384 | 86M | 384 | 768 | 12
CeiT-S$\uparrow$384 | 24M | 384 | 768 | 12
TNT-B$\uparrow$384 | 65M | 384 | 40+640 | 4+10
Table 5: Accuracy comparison on CIFAR-10 and CIFAR-100 Model | CIFAR-10 | CIFAR-100 | Average
---|---|---|---
ViT-B/16-Sparse | 98.81 | 92.52 | 95.66
ViT-B/16-Sparse$\downarrow$10% | 98.71 | 91.66 | 95.18
ViT-B/16-Sparse$\downarrow$15% | 98.16 | 89.62 | 93.89
ViT-B/16-Sparse$\downarrow$20% | 96.68 | 84.88 | 90.78
ViT-B/16-Sparse$\downarrow$25% | 93.03 | 74.67 | 83.85
ViT-B/16-Sparse$\downarrow$30% | 85.09 | 59.47 | 72.28
ViT-B/16-SAM | 98.2 | 87.6 | 92.9
DeiT-B$\uparrow$384 | 99.1 | 90.8 | 94.95
CeiT-S$\uparrow$384 | 99.1 | 90.8 | 94.95
TNT-B$\uparrow$384 | 99.1 | 91.1 | 95.1

Table 5 report numerical result on CIFAR-10 and CIFAR-100 as a downstream
task. Interestingly, with almost the same parameters as defined in the
original paper [3], Vision Transformer with sparse regularization takes first
place on average accuracy.

###  4.6 Result on ImageNet-100

To provide a broader overview of the sparse regularization effect on pruning,
this experiment has also been carried out using ImageNet-100, a subset of
ImageNet1k. The sampling process was carried out randomly by taking 100
classes from ImageNet1k dataset. As a result, the ViT model with sparse
regularization can overcome the original ViT model with 0.12% higher accuracy.
The detail can be seen in Tabel 6.

Table 6: Accuracy on ImageNet-100 Model | Sparse Position | Accuracy
---|---|---
ViT-B/16 | - | 96.8
ViT-B/16-Sparse | Attention Weight | 96.92

Figure 5 shows the effect of pruning performed on the model that applies
sparse regularization. It can be seen that the result has a similar pattern to
the experiment that has been done on CIFAR-10 and CIFAR-100 data. This method
produces higher accuracy than pruning performed on models without sparse
regularization, with 0.25% higher accuracy on average.

Figure 5: Pruning effect on ImageNet-100

##  5 Conclusion

The implementation of sparse regularization will produce the best accuracy if
it is placed after the attention weight calculation on the self-attention
layer. Pruning on models with sparse regularization produces better accuracy
than pruning on models without sparse regularization.

The Vision Transformer model with sparse regularization can improve accuracy
by 0.12% on CIFAR-100 and 0.12% on ImageNet-100. Meanwhile, on CIFAR-10 data,
the model with sparse regularization has yet to outperform the baseline model.
While there is a negative correlation between pruning and accuracy, with the
accuracy decreasing as the percentage of pruned parameters increases, models
with sparse regularization tend to have a slightly higher average accuracy. In
particular. On CIFAR 10 data. Pruning on models with sparse regularization
achieves an average higher accuracy of 0.568%. Similarly, on CIFAR 100 data.
The average higher accuracy achieved through pruning on models with sparse
regularization is 1.764% and 0.256% higher on ImageNet-100.

In summary, our sparse regularization and pruning explorations have revealed a
delicate interplay between these techniques and model accuracy. Sparse
regularization, when strategically placed, can unlock the true potential of
deep learning models, enhancing accuracy in certain contexts. Furthermore, the
combination of sparse regularization and pruning presents a compelling
approach to mitigating the negative impact of pruning on accuracy.

## References

  * [1] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, CoRR abs/1409.4842 (2014).  arXiv:1409.4842.
URL http://arxiv.org/abs/1409.4842

  * [2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, CoRR abs/1706.03762 (2017).  arXiv:1706.03762.
URL http://arxiv.org/abs/1706.03762

  * [3] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, CoRR abs/2010.11929 (2020).  arXiv:2010.11929.
URL https://arxiv.org/abs/2010.11929

  * [4] J. Kukacka, V. Golkov, D. Cremers, Regularization for deep learning: A taxonomy, CoRR abs/1710.10686 (2017).  arXiv:1710.10686.
URL http://arxiv.org/abs/1710.10686

  * [5] L. Liebenwein, C. Baykal, B. Carter, D. Gifford, D. Rus, Lost in pruning: The effects of pruning neural networks beyond test accuracy, CoRR abs/2103.03014 (2021).  arXiv:2103.03014.
URL https://arxiv.org/abs/2103.03014

  * [6] H. D. Jahja, N. Yudistira, Sutrisno, Mask usage recognition using vision transformer with transfer learning and data augmentation, Intelligent Systems with Applications 17 (2023) 200186.  doi:https://doi.org/10.1016/j.iswa.2023.200186.
URL https://www.sciencedirect.com/science/article/pii/S266730532300011X

  * [7] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. Jégou, Training data-efficient image transformers & distillation through attention, CoRR abs/2012.12877 (2020).  arXiv:2012.12877.
URL https://arxiv.org/abs/2012.12877

  * [8] G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network (2015).  arXiv:1503.02531.
  * [9] K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, W. Wu, Incorporating convolution designs into visual transformers (2021).  arXiv:2103.11816.
  * [10] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, L. Zhang, Cvt: Introducing convolutions to vision transformers, CoRR abs/2103.15808 (2021).  arXiv:2103.15808.
URL https://arxiv.org/abs/2103.15808

  * [11] X. Chen, C. Hsieh, B. Gong, When vision transformers outperform resnets without pretraining or strong data augmentations, CoRR abs/2106.01548 (2021).  arXiv:2106.01548.
URL https://arxiv.org/abs/2106.01548

  * [12] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, Y. Wang, Transformer in transformer, CoRR abs/2103.00112 (2021).  arXiv:2103.00112.
URL https://arxiv.org/abs/2103.00112

  * [13] H. Ide, T. Kurita, Improvement of learning for cnn with relu activation by sparse regularization, in: 2017 International Joint Conference on Neural Networks (IJCNN), 2017, pp. 2684–2691.  doi:10.1109/IJCNN.2017.7966185.
  * [14] M. Zhu, K. Han, Y. Tang, Y. Wang, Visual transformer pruning, CoRR abs/2104.08500 (2021).  arXiv:2104.08500.
URL https://arxiv.org/abs/2104.08500

  * [15] X. Gastaldi, Shake-shake regularization, CoRR abs/1705.07485 (2017).  arXiv:1705.07485.
URL http://arxiv.org/abs/1705.07485

  * [16] J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization (2016).  arXiv:1607.06450.
  * [17] H. Ramchoun, M. Amine, J. Idrissi, Y. Ghanou, M. Ettaouil, Multilayer perceptron: Architecture optimization and training, International Journal of Interactive Multimedia and Artificial Inteligence 4 (2016) 26–30.  doi:10.9781/ijimai.2016.415.
  * [18] D. Hendrycks, K. Gimpel, Bridging nonlinearities and stochastic regularizers with gaussian error linear units, CoRR abs/1606.08415 (2016).  arXiv:1606.08415.
URL http://arxiv.org/abs/1606.08415

  * [19] J. van Amersfoort, M. Alizadeh, S. Farquhar, N. D. Lane, Y. Gal, Single shot structured pruning before training, CoRR abs/2007.00389 (2020).  arXiv:2007.00389.
URL https://arxiv.org/abs/2007.00389

  * [20] Y. He, G. Kang, X. Dong, Y. Fu, Y. Yang, Soft filter pruning for accelerating deep convolutional neural networks, CoRR abs/1808.06866 (2018).  arXiv:1808.06866.
URL http://arxiv.org/abs/1808.06866

  * [21] Y. He, L. Xiao, Structured pruning for deep convolutional neural networks: A survey (2023).  arXiv:2303.00566.
  * [22] C. Laurent, C. Ballas, T. George, N. Ballas, P. Vincent, Revisiting loss modelling for unstructured pruning (2020).  arXiv:2006.12279.
  * [23] K. Weiss, T. M. Khoshgoftaar, D. Wang, A survey of transfer learning, Journal of Big Data 3 (1) (2016) 9.  doi:10.1186/s40537-016-0043-6.
URL https://doi.org/10.1186/s40537-016-0043-6

  * [24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, Imagenet large scale visual recognition challenge, International Journal of Computer Vision 115 (3) (2015) 211–252.  doi:10.1007/s11263-015-0816-y.
URL https://doi.org/10.1007/s11263-015-0816-y

  * [25] Z. Zhang, M. R. Sabuncu, Generalized cross entropy loss for training deep neural networks with noisy labels, CoRR abs/1805.07836 (2018).  arXiv:1805.07836.
URL http://arxiv.org/abs/1805.07836

  * [26] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.  doi:10.1109/CVPR.2009.5206848.


# Dredge Word, Social Media, and Webgraph Networks for Unreliable Website
Classification and Identification

Evan M. Williams  Peter Carragher  Kathleen M. Carley
Carnegie Mellon University
5000 Forbes Ave.
USA, Pittsburgh, PA, 15213-3890
{emwillia, pcarragh<EMAIL_ADDRESS>
###### Abstract

In an attempt to mimic the complex paths through which unreliable content
spreads between search engines and social media, we explore the impact of
incorporating both webgraph and large-scale social media contexts into website
credibility classification and discovery systems. We further explore the usage
of what we define as dredge words on social media—terms or phrases for which
unreliable domains rank highly. Through comprehensive graph neural network
ablations, we demonstrate that curriculum-based heterogeneous graph models
that leverage context from both webgraphs and social media data outperform
homogeneous and single-mode approaches. We further demonstrate that the
incorporation of dredge words into our model strongly associates unreliable
websites with social media and online commerce platforms. Finally, we show our
heterogeneous model greatly outperforms competing systems in the top-k
identification of unlabeled unreliable websites. We demonstrate the strong
unreliability signals present in the diverse paths that users follow to
uncover unreliable content, and we release a novel dataset of dredge words.

Dredge Word, Social Media, and Webgraph Networks for Unreliable Website
Classification and Identification

Evan M. Williams and Peter Carragher and Kathleen M. Carley Carnegie Mellon
University 5000 Forbes Ave. USA, Pittsburgh, PA, 15213-3890 {emwillia,
pcarragh<EMAIL_ADDRESS>

##  1 Introduction

On February 24, 2022, the day Russia began its invasion of Ukraine, Jacob
Creech, a fringe Qanon conspiracy theorist, posted a series of unsubstantiated
claims on Twitter. These tweets insinuated that the US had created COVID-19,
and implied Russia’s invasion of Ukraine was actually being carried out to
shut down US-funded “biolabs” in Ukraine and to prevent another global
pandemic League (2022). Hours after the tweet, the conspiratorial website
InfoWars published an article promoting the tweet, crediting its author for
uncovering an “ulterior motive theory” League (2022). The tweet began
circulating to other platforms as Google queries for “US biolabs” and “Ukraine
Biolabs” spiked in the following days
111https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=Ukraine%20biolabs&hl=en.
As only conspiratorial sites covered the tweet in its early hours, those would
have likely been the only relevant search results until Snopes debunked the
claim later that day Evon (2022).

In some ways, this is a success story of the current paradigm of combatting
misinformation; fact-checkers responded rapidly to a quickly-spreading
falsehood. In other ways, this highlights the challenges present in reactive
misinformation interventions. Despite quick debunking by fact-checkers, the
conspiracy was nonetheless echoed by established news sources like Fox News,
and later amplified on the floor of the US SenateLeague (2022). Researchers
later found that a coordinated campaign of social media accounts boosted the
narrative on Twitter Alieva et al. (2022). As of 2024, despite the underlying
claims having been debunked by many fact-checkers Chappell and Yousef (2022);
Parachini (2022), a search engine audit study found that Google, Bing, and
Yandex all returned URLs on their first page of results that promote the
“Ukraine Biolabs” conspiracy Kuznetsova et al. (2024). This story also
highlights an important and understudied dimension of misinformation spread:
the interaction between social media and search engines.

An alternative to reactive fact-checking approaches are proactive algorithmic
content moderation approaches. These can include modifying the recommendation
and ranking systems to decrease the reach and virality of unreliable
information sources. In search engines, this can mean downranking articles
from unreliable domains, and on social media, this can mean ranking posts
containing unreliable domains lower in user newsfeeds. For proactive
approaches to succeed, platforms need systems that can rapidly detect and
discover unreliable domains. Both classification and detection are important
as unreliable websites can, and do, deploy a number of tactics to avoid
blacklists Carragher et al. (2024). In this work, we present a novel approach
to the tasks of unreliable domain detection and discovery that leverages
signals from both large-scale social media data and webgraph data.

Directing users to search engines can be an effective tactic to spread
misinformation as, since 2016, global surveys have consistently found that
individuals report higher trust in search engines than traditional media
Barometer (2024); McDuling (2015). Thus by pointing users to search engines,
either explicitly or implicitly, bad actors can create a false sense of
content reliability. Little is understood about these pathways, and research
has only recently begun exploring the ways social media users end up in data
voids. The concepts of problematic queries Golebiewski and Boyd (2019) and
keyword signaling Tripodi (2019) are highly related, but have only been
explored in case studies. The recently-proposed concept of search directives
provides a clear and observable path that could direct users to unreliable
content on search engines. A search directive is content that is explicitly
intended to prompt an online search, e.g., user $a$ directs user $b$ to "look
up Chemtrails on Google" Robertson et al. (2023). In the context of data
voids, search directives as a concept have two limitations: First, while
search directives can direct users to unreliable content, they can also direct
users to innocuous or helpful information. Second, as the search trends for
“Ukraine Biolabs” demonstrate, users can be driven to unreliable content
through search engines by querying phrases that appear without explicit search
directives. To help fill these gaps, we propose the concept of dredge
words—terms or keyphrases for which unreliable domains rank highly on search
engines.

By attempting to explicitly incorporate each of the paths that users take to
unreliable websites into GNNs, we seek to explore the dynamics that connect
the spread of unreliable content on social media and search engines. First, we
attempt to demonstrate domain (un)reliability signal present in large-scale
social media data and small-scale webgraph data with varying levels of
context. With respect to website credibility classification, we use graph
neural networks to demonstrate the domain reliability signal in user-domain
interaction networks on large-scale twitter data, on small-scale webgraph
networks, and homogeneous and heterogeneous combinations of the two. Through
comprehensive graph neural network ablations, we demonstrate that
heterogeneous graph models that leverage context from both webgraphs and
social media data outperform homogeneous and single-mode approaches. We
additionally demonstrate that designing a training curriculum based on
relative website reliability scores further increases model performance.

Finally, we provide the first exploration of dredge words on social
media—terms or keyphrases for which unreliable domains rank highly in search
engines. We demonstrate the usefulness of these dredge words in an unreliable
domain classification and discovery process that mimics how social media users
may transition from social platforms to the consumption of misinformation via
search engines. Surprisingly, we also find that dredge words frequently
surface social media URLs in top SERP positions, thus providing a
bidirectional path that often leads back to social media. We show our
heterogeneous model greatly outperforms competing systems in the top-k
discovery of unlabeled unreliable websites. We publicly release the code,
webgraph data collected for this project, a dataset of 3,939 dredge words for
46 unreliable news domains, and the resulting SERPs for each of the dredge
word queries222Artefacts will be publicly released following review period.

##  2 Related Works

###  2.1 Social Media

Classifying individual texts or articles is a core problem addressed by
research in misinformation detection. Such detection methods have typically
relied on website content and social media data Castelo et al. (2019); Chen
and Freire (2020); Silva et al. (2021). Castelo et al. proposed a topic-
agnostic detection system Castelo et al. (2019) that identifies unreliable
articles based on Linguistic Inquiry and Word Count features. Chen and Freire
adopted the method for the task of unreliable domain discovery Chen and Freire
(2020). Their discovery system capitalized on user tendencies within the
social graph, wherein a user who tweets a URL from a known unreliable source
is likely to also tweet URLs from yet unknown sources. Similarly, Silva et al.
combine website content and social context resulting in a misinformation page
detection method that leverages heterogeneous data types Silva et al. (2021),
with a focus on early detection across a broad range of topics.

###  2.2 Webgraphs

The predictive power of webgraphs and social media data have been demonstrated
separately in several contexts. Aswani et al. detect SEO manipulation by link-
building sites by clustering on Pagerank score and Domain Authority Aswani et
al. (2021). For detecting news site bias, Aires et al. Patricia Aires et al.
(2019) scrape cross-links from a list of prominent news sites from Media Bias
Fact Check (Zandt, 2022). Sehgal et al. explore a case where misinformation
was spread through coordinated hyperlink and social media networks Sehgal et
al. (2021) on a dataset of 1.4k URLs. Another recent case study demonstrated
the manipulated webgraph linkages of unreliable pseudo-thinktanks Williams and
Carley (2023). More recently, Carragher et al. (2024) proposed a webgraph-
based model for unreliable domain detection and discovery tasks. For
discovery, the authors use backlinking domains in the webgraph in a snowball
sampling approach. The authors demonstrate that GNN models trained on SEO
attributes and webgraph data proved effective for detection.

###  2.3 Combining Webgraphs and Social Media Context

Zhang and Cabage show that social media and SEO promotion have different
strengths; they find that social-sharing results in immediate but short-term
boosts to traffic, while the benefits of link-building are slower to realize,
but last longer Zhang and Cabage (2017). This hints that a combination of both
webgraph and social media data should be used in investigating misinformation
sources. However, to our knowledge, this is the first work that attempts to
combine the two contexts for unreliable domain classification or discovery.

##  3 Data

We construct datasets based on the paths that users can take to unreliable
websites. In our social media and webgraph data, we attempt to capture
explicit paths users take to unreliable websites—i.e. clicking a URL. The
dredge words we collect for a subset of the unreliable domains represent
implicit paths.

###  3.1 URL Labels

We use the domain credibility labels published by Lin et al. Lin et al.
(2023). The authors of the work align the domain reliability ratings of all
domains ranked by 6 expert groups and run imputation followed by principal
component analysis to generate aggregate ratings for 11,520 domains Lin et al.
(2023). To our knowledge, this is the most comprehensive public domain
reliability rating list as of spring of 2024. The domains in this list are
ranked by the calculated first principal component where low scores correspond
to high expert agreement on unreliability and high scores correspond to expert
agreement on reliability. While these scores are very sensible across wide
principal component score gaps, locally the relative orderings are less clear.
Treating website reliability as continuous also muddies the interpretability
and the discovery process, as there’s not a transparent reason a website
should have a score of 0.571 as opposed to 0.570. To address this issue, we
elected to binarize the website reliability labels. After manual inspection of
the data, we elected to consider the bottom two quintiles as unreliable
(corresponding to a principal component threshold of 0.5162).

###  3.2 Webgraph and features

As extracting complete webgraph information for these domains is
computationally challenging, researchers have generally needed to use third-
party tools to extract local webgraph attributes. We use the SEO toolkit
service Ahrefs333ahrefs.com to extract the 10 domains which link to each
target domain at the highest volume (the highest-volume back-linking domains).
We chose Ahrefs because researchers have previously shown the usefulness of
its data in domain reliability classification and discovery applications
Carragher et al. (2024); Williams and Carley (2023). We were unable to pull
backlinks for 193 of the domains. A random sample of 10 of 193 domains and
found 9 of them were either dead or unreachable. We therefore elected to drop
these 193 domains, leaving us with 11,327 unique reliability-labeled domains
and 32,431 unique unlabeled backlinking domains for a total of 43,758 domains.
For each of these 43,758 domains, we pull 23 attributes, which for an
individual domain, contains fields like the total number of backlinks and
outlinks, number of backlinks coming from .edu or .gov domains, and number of
referring pages 444https://ahrefs.com/api/documentation/metrics.

Similar to the backlink network constructed in Carragher et al. (2024), there
is clear assortativity across label-reliability groups as can be seen in
Figure 1. The prevalence of unlabeled nodes in our network overwhelms a node
attribute assortativity coefficient calculation $r_{A}=-0.336$. We re-
calculate node attribute assortativity on an induced subgraph that only
maintains edges between labeled nodes. The subgraph contains less structure
and 176 components, but nevertheless displays strong positive associativity
$r_{A}=0.376$. In other words, domains of the same (binary) reliability are
more likely to link to one another than linking to a domain of a different
reliability. This provides a strong justification for the use of network-based
models.

Figure 1: Webgraph colored by domain reliability labels. The network contains
6,861 reliable (blue) websites, 4,466 (red) unreliable websites, and 32,431
unlabeled (grey) backlinking websites.

###  3.3 Twitter Data

For social media context, we use a Twitter dataset constructed by querying
COVID-related keywords555coronavirus, Wuhan virus, Wuhanvirus, 2019nCoV, NCoV,
NCoV2019, covid-19, covid19, covid 19 via Twitter’s streaming API between
January 29, 2020 and June 26, 2022. Due to server issues and API limitations,
121 days over the time period have partial or missing data. However, these
gaps are spread relatively evenly over the time period, and still provide
strong coverage. Our final Twitter dataset contained 3.6 billion extracted
tweets.

From this set of 3.6 billion tweets, we pull all Tweets that mention or link
to one of our 11,520 websites, resulting in a dataset of 320M tweets that link
to over 840k unique domains. Of the original 11,520 labeled domains, only
5,504 appeared within the Twitter data. To reduce the noise in the dataset we
undertake several cleaning operations. Firstly, we drop all tweets that do not
explicitly include a link to the URL. Second, users with fewer than 10
observed tweets over the time period were dropped. To further condense the
dataset, we only include tweets which appeared at least 3 times in the
dataset. As reposting and retweeting are important influence metrics on
Twitter, this drops the tweets that likely did not receive as much attention.
We also dropped domains that were only linked to by a single user and users
that only linked to a single domain, as we hypothesized these pendulum nodes
would be of limited use given the size of the graph. As a result of the
expense of the webgraph attribute API, we chose to further restrict the
Twitter data to only include tweets that mention at least one of the 43,758
domains for which we extracted attributes. This corresponds to keeping tweets
that mention at least one of the 11,327 labeled domains for which we have
attributes or tweets that mention one of the 11,327 and co-mention any of the
32,431 unlabeled domains. Following these cleaning operations, we are left
with 555k users and 4.9M unique tweets which we observed tweeted or retweeted
91.2M times. This reduced dataset contains mentions of 2,475 reliability-
labeled domains and 714 unlabeled back-linking domains.

###  3.4 Dredge Words

With API constraints in mind, for the 100 websites with the lowest PC1 score
rank in Lin et al. (2023). We extract the top-ranked $k\leq 1,000$ Google
keyphrases from Ahrefs, for a total of 34,646 keyphrases. We then used
WebSearcher Robertson and Wilson (2020) to query each of these keyphrases on
Google and extracted the first 10 URLs returned in each SERP. We kept all
queries for which the target unreliable domain was returned in the top 10
Google search results. This resulted in our set of 3,939 dredge words spanning
46 unreliable domains. We created a separate query to find mentions of these
keyphrases in the 3.6B tweet covid dataset. This yielded 5.7 million tweets
containing dredge words. Many of the most common mentions were explicit
mentions of domain names or organization names (which can be identical to
twitter handles—like “gatewaypundit”, “infowars”, and “nvic” all received over
10,000 mentions. To filter down the data, we use regular expressions to ensure
each drudge word begins with a hashtag, starts a tweet, or is preceded by a
white space. This retains 213 dredge words in 421k tweets that qualitatively
contain less noise. Of these 421k tweets, only 9,788 (2%) explicitly linked to
the unreliable domain associated with the dredge word.

Figure 2: The top search results for the dredge word “silent assassination
through amplified neurons”. The query surfaces fringe reddit subreddits
followed by “beforeitsnews”, an unreliable news source.

Some examples of dredge words include “psychic attack” (28 Twitter mentions),
“akashic record” (12), “flu shot injury” (6), and “fallcabal” (5). While the
majority of the dredge words are in English, other languages are present in
the list, including Chinese, Hindi, and Arabic. We use this condensed twitter
dataset and the set of SERP resultss they yielded in our dredge-word-based
unreliable domain discovery process. While we were initially interested in
paths from social media to search engines, we find dredge word SERPs
surprisingly demonstrate a strong paths to social media. Youtube was by far
the most commonly-returned domain (4,304 times). Wikipedia, Reddit, Quora,
Twitter, Amazon, and Facebook were also among the most commonly-returned
domains, in part due to the widespread commercialization of pseudo-scientific
concepts. For a more in-depth analysis of dredge words see Appendix A.

##  4 Methods

In order to evaluate the impact of additional levels of context on our models,
we construct graphs that capture increasingly-granular levels of context
present in the data.

###  4.1 Single Mode Graphs

We define two single-mode graphs to evaluate the effectiveness of the models
using only webgraph data and only Twitter data. Let $\mathcal{W}$ be the set
of all labeled and unlabeled websites $\\{w_{1},w_{2},\dots,w_{n}\\}$
contained in the extracted webgraph data and define $\mathcal{U}$ as the set
of all Twitter users $\\{u_{1},u_{2},\dots,u_{n}\\}$ that link to domains
$\in\mathcal{W}$. For our webgraph data, we define
$\mathcal{G}_{w}=(\mathcal{W},A,Z,Y)$ where $A$ is a binary undirected
adjacency matrix where a $1$ or $0$ denotes the presence or absence of a
connection between website $i$ and website $j$. $Z$ is a node feature matrix,
and $Y$ is the label assigned to each node ($1$ = reliable, $0$ = unreliable,
$-1$ = unlabeled). We define a corresponding graph for Twitter user-to-website
connections $\mathcal{G}_{t}=(\mathcal{U},A,Z,Y)$ where nodes are users and
websites $\in\mathcal{U}$ and the set of edges captured by $A$ denote observed
edges between $\mathcal{U}$ and $\mathcal{W}$.

In the $\mathcal{G}_{w}$, the constructed webgraph, we define $Z$ as the
domain-level attributes extracted from Ahrefs. Following Carragher et al.
(2024), we log-normalize the features of each domain. However, the social
media network, $\mathcal{G}_{t}$ presents a challenge in defining $Z$, as
users do not have website-level attributes. For this graph, we therefore
define $Z_{n}$ as features calculated using using Node2Vec Grover and Leskovec
(2016) parameterized with a walk length to 20, context size to 10, and 10
walks per node, and an embedding dimension of 23, chosen because it is
identical to the number of features that we extract for each website from
Ahrefs. $\mathcal{G}_{t}$ presents an additional challenge in that not all
websites $\in\mathcal{W}$ are mentioned by users $\in\mathcal{U}$. We include
social-media-only baselines, but these are trained and evaluated on only a
network of user interactions with the 2,475 labeled domains mentioned in the
social media data, which make these results not directly comparable with our
other models.

###  4.2 Homogeneous Graph

We construct a homogeneous graph
$\mathcal{G}_{Hom}=\mathcal{G}_{w}\cup\mathcal{G}_{t}$ where $\cup$ denotes a
graph union operation. As node features for users and websites are different,
we consider only features derived from the graph structure in the homogeneous
model. Again, as features are different for users and websites, we calculate
features using node2vec over the graph $\mathcal{G}_{Hom}$ using the same
parameters as for the single-mode graph.

###  4.3 Heterogeneous Graph Construction

We construct a heterogeneous graph
$\mathcal{G}_{Het}=(\mathcal{W}\cup\mathcal{U},A,Z,\phi,Y)$ where $\phi$
denotes a type of relation: websites-to-websites or users-to-websites. As a
previous work has demonstrated the effectiveness of webgraph features in
reliability classification Carragher et al. (2024), we consider only two
different feature combinations. We consider a heterogeneous graph
$\mathcal{G}_{Het}$ where websites have log-normed attributes from ahrefs and
users are assigned the node2vec attributes calculated for $\mathcal{G}_{Hom}$.
We consider a second heteroegeneous graph, $\mathcal{G}_{Het,u}$ where each
user is assigned attributes extracted from their text content. For each user,
we randomly sample, without replacement, 10 tweets for each observed user and
embed them using multilingual distilBERT Sanh et al. (2019)666All models in
this paper were run or trained on a single NVIDIA GeForce RTX 3080.. This is a
naive approach that excludes a massive amount of context for some users,
particularly as we observed two users tweet over 1 million times in our
filtered dataset. Clearly, only considering 10 tweets is suboptimal, but it is
a standard practice in user embedding literature Pan and Ding (2019). A clear
path for future work is to explore the impact more advanced user embedding
strategies within this system.

###  4.4 Heterogeneous Dredge-word Graph Construction

We construct an additional heterogeneous graph $\mathcal{G}_{Het-d}$ that
attempts to model the path through which dredge words can drive users to
unreliable content on search engines. $\phi$ in this graph can denote
connections between dredge words and websites extracted from SERPs or
connections between dredge words and users $\in\mathcal{U}$. Dredge words
embeddings were extracted using multilingual distilBERT, and these embeddings
are used as features for dredge word nodes. Though the dredge word Twitter
dataset was collected separately, but we choose to restrict the users we
consider to those who also shared links to at least one known unreliable
domain to facilitate model comparability. We finally define $\mathcal{G}_{Het-
Dredge}$ as $\mathcal{G}_{Het-d}\cup\mathcal{G}_{Het}$.

###  4.5 Graph Neural Networks

A label-stratified 80/10/10 split on labeled websites is used to create
training, validation, and test sets. For our one-mode, heterogeneous, and
homogeneous experiments we use a simple 2-layer graph neural network using
GraphSAGE convolutions proposed in Hamilton et al. (2017). We ran experiments
using graph attention networks Veličković et al. (2017) and heterogeneous
graph transformers Hu et al. (2020), but found marginal accuracy gains at the
cost of longer training times. Consequently, we selected GraphSAGE layers.
Each model consists of a SAGEConv layer with dropout, with a hidden dimension
of 512, followed by a ReLU activation and a second SAGEConv layer with a log
softmax activation function. For each model, we train for a maximum of 1,000
epochs with early stopping based on validation loss and a patience of 50. We
use an Adam optimizer with a starting learning rate of 1e-3 and a cosine
annealing learning rate scheduler. The heterogeneous GNN, the most complex
network we consider, contains 3.2M parameters, and no model exceeded 5 minutes
of training time to reach our early stopping convergence condition.

###  4.6 Curriculum Learning

We assume that highly reliable and highly unreliable websites are easier to
differentiate in webgraphs than those that are mixed, mostly reliable, or
mostly unreliable. A manual examination of the labeled URL data leads us to
believe that this assumption is very reasonable when website content is
considered (see appendix B).

We implement a slightly-modified version of the "Baby Steps" learning
curriculum proposed and explored in Spitkovsky et al. (2010); Cirik et al.
(2016). As domain labels from Lin et al. (2023) contain unified principal
component scores of expert ratings of domain reliability, we can develop a
curriculum that first learns labels of extremely reliable and extremely
unreliable domains, and that gradually works from the extremes towards the
reliability boundary. Using the original labeled domain dataset $\mathcal{D}$,
we calculate quartiles of reliable labels and unreliable labels using
principal component scores, and define these as an ordered set of batches
$\\{d_{1},d_{2},\dots,d_{10}\\}$, where $d_{1}$ is the first quintile of the
most reliable domains and $d_{10}$ is the fifth quintile of the unreliable
domains. When using curriculum learning, we begin training the model
$\mathcal{M}$ using $\\{d_{1},d_{10}\\}$, and following convergence, the model
considers $\\{d_{1},d_{10}\\}\cup\\{d_{2},d_{9}\\}$. The model continues in
this fashion until it converges on all available data. In our Baby Steps
implementation, the we define model’s corresponding convergence as 10 epochs
without an improvement in validation loss. We note that this is the curriculum
convergence condition, and not the convergence condition of the model. More
generally, for a website reliability curriculum $\mathcal{C}$, an even-length
dataset $\mathcal{D}$, and a model $\mathcal{M}$, our implementation of the
Baby Steps curriculum can be expressed as the procedure in Algorithm 1.

Algorithm 1 Modified Baby Steps Curriculum

$\mathcal{M}$, $\mathcal{D}$, $\mathcal{C}$

sort($\mathcal{D}$, $\mathcal{C}$)

$B\leftarrow\emptyset$

for $i\in 0,1,\dots\frac{k}{2}$ do

while $\mathcal{M}$ not converged do

$B\leftarrow B\cup\mathcal{D}[i]\cup\mathcal{D}[\frac{k}{2}-i]$

$\mathcal{M}(B)$

return $\mathcal{M}$

###  4.7 Unreliable Domain Discovery

We implement and evaluate two distinct discovery processes. The first is GNN
discovery, where we take predictions for unlabeled domains in the graph from
the best performing GNN model, $\mathcal{E}_{u,w,c}$. We call the second
method “Dredge” Webgraph-Based Discovery ($\text{WG-BD}_{d}$), which mimics
the path social media users take to reach unreliable domains by observing
dredge words on social media, and querying these terms on a search engine.
Using WebSearcher Robertson and Wilson (2020), we create a candidate list of
domains from the top 10 SERP results for each of the dredge words we have
compiled from our Twitter dataset. We then pull Ahrefs attributes for each of
the candidate domains and use the SEO attribute classifiers from Carragher et
al. (2024) to filter down the candidate domains as detailed in the appendix.

We compare our discovery processes with the webgraph-based discovery (WG-BD)
process proposed in Carragher et al. (2024) and with the social media-based
discovery process (SM-BD) proposed in Chen and Freire (2020). Specifically, we
consider Precision@5, @10, @20, and we consider the partial F1 metric proposed
in Chen and Freire (2020). This means we run the discovery process twice, once
on the full domain list where we evaluate results manually with top-10 and
top-20 accuracies, and again on a restricted domain list to compute the
partial F1 metric. We additionally report partial precision and recall. We
note that dredge words were only considered for 46 of the least reliable
websites, so their inclusion in models is likely to harm the models, as those
features only exist for a small set of the 11,327 labeled domains. We
nonetheless report statistics and discovery evaluations.

Due to a lack of ground truth labels for evaluating newly discovered domains,
Partial F1 measures the ability of the discovery system to find unreliable
domains with respect to two known lists of unreliable sources, a seed list and
an evaluation list. In this case, the seed list is the PoliticalNews dataset
and the evaluation list is drawn from unreliable MBFC domains (Chen and
Freire, 2020). As such, partial F1 measures the ability of a discovery system
seeded on PoliticalNews to discover as many MBFC domains as possible (high
partial recall), without discovering domains that are not in the MBFC list
(high partial precision) (Castelo et al., 2019). We discuss the limitations of
the partial F1 metric in more detail in Limitations and Appendix C.

##  5 Results

###  5.1 Credibility Classification

We present Accuracy and F1 statistics for GNN ablations in Table 1.
$\mathcal{H}$ denotes a homogeneous GNN and $\mathcal{E}$ denotes a
heterogeneous GNN. The subscripts $u$ and $w$ denote the use of the user
network ($u$) and/or the webgraph network ($w$). We use graph-specific
($\mathcal{G}_{w}$, $\mathcal{G}_{t}$, or
$\mathcal{G}_{w}\cup\mathcal{G}_{t}$) Node2Vec features in all GNNs without
$f$ or $v$ subscripts. $f$ denotes the inclusion of webgraph features, $v$
denotes the inclusion of embedded user text, and $d$ denotes the model was
trained on $\mathcal{G}_{Het-Dredge}$. Finally, $c$ denotes the inclusion
learning curriculum in training. We note that the two social-media-only
models, $\mathcal{H}_{u}$ and $\mathcal{H}_{u,c}$ are trained and evaluated on
only the 2,475 labeled domains that were mentioned in the cleaned social media
data.

We find that the heterogeneous model $\mathcal{E}_{u,w,c}$, which incorporates
the user network, the webgraph, and curriculum learning outperforms all other
models with an F1 of $0.7777\pm.003$. While the accuracy of the Social Media
models is higher, those models are run only on a subset of the data, and thus
are not directly comparable.

Model | Accuracy | F1
---|---|---
$\mathcal{H}_{w}$ | 0.7686 ± .004 | 0.7545 ± .005
$\mathcal{H}_{w,f}$ | 0.7799 ± .002 | 0.7714 ± .002
$\mathcal{H}_{w,f,c}$ | 0.7791 ± .002 | 0.7705 ± .002
$\mathcal{H}_{u}^{*}$ | 0.8012* ± .005 | 0.7443* ± .008
$\mathcal{H}_{u,c}^{*}$ | 0.8113* ± .007 | 0.7582* ± .009
$\mathcal{H}_{u,w}$ | 0.7523 ± .005 | 0.7654 ± .005
$\mathcal{H}_{u,w,c}$ | 0.7696 ± .006 | 0.7559 ± .006
$\mathcal{E}_{u,w}$ | 0.7797 ± .010 | 0.7707 ± .012
$\mathcal{E}_{u,w,c}$ | 0.7865 ± .002 | 0.7777 ± .003
$\mathcal{E}_{u,w,f,v}$ | 0.7827 ± .007 | 0.7741 ± .008
$\mathcal{E}_{u,w,c,f,v}$ | 0.7738 ± .118 | 0.7657 ± .018
$\mathcal{E}_{u,w,c,f,v,d}$ | 0.7787 ± .004 | 0.7675 ± .007
Table 1: Mean Accuracy and F1 and standard deviations for each GNN ablation
over 10 runs. $\mathcal{H}$ denotes homogeneous and $\mathcal{E}$ denotes a
heterogeneous and subscripts denote included networks and features. * denotes
social-media-only—these were run and evaluated on the 2,475 labeled domains
mentioned in the Twitter data and thus are not directly comparable with other
models.

###  5.2 GNN Discovery

Two annotators77725-30 year-old PhD candidates studying misinformation
independently annotated the reliability of the top 20 predictions (sorted by
prediction confidence) of the top performing model, $\mathcal{E}_{u,w,c}$,
over the set of all unlabeled domains. Inter-annotator agreement in the first
round had a Krippendorff’s $\alpha=0.78$. The annotators then met and resolved
all disagreements. We compare our model with the webgraph-based discovery (WG-
BD) approach in Carragher et al. (2024). As that paper’s discovery process did
not rank discovered domains, we sort results by the misinformation classifier
proposed in the work. Again, two independent annotators ranked the results,
yielding a Krippendorff’s $\alpha=0.69$. For comparison with previous
unreliable domain discovery approaches, we additionally report partial F1,
defined in Chen and Freire (2020) as the set of true positive unreliable
domains with credibility labeled “mixed” or worse by Media Bias Fact Check.
Thresholding at a prediction confidence level of 0.7 (see Appendix C for
sensitivity analysis), the Partial F1 of our top-performing GNN model is 0.25,
which is largely in-line with Chen and Freire (2020) (0.29) and Carragher et
al. (2024) (0.28). We observe that $\mathcal{E}_{u,w,c}$ outperforms both
competing systems at each level of precision we consider.

###  5.3 Dredge Word Discovery

We consider two dredge word discovery approaches: one ($\text{Dredge WG-
BD}_{d}$) using the discovery approach proposed in (Carragher et al., 2024)
and the other using the GNN trained with dredge word context
($\mathcal{E}_{u,w,c,f,v,d}$). In the first approach, we observe that
$\text{WG-BD}_{d}$ yields higher precision than the webgraph based discovery
for P@5 and P@10, but lower precision for P@20. In the latter approach, as a
consequence of 1) only extracting dredge word networks for only 46 of the
least reliable domains 2) the dominance of social media and shopping sites in
the dredge word SERPs, the dredge word GNN learns to heavily associate social
media sites and online shopping platforms with unreliable domains. The 10 most
confident predictions the GNN returns contain 7 social media sites, amazon,
itunes, and s3.amazonaws. A deeper analysis of these results reveals that the
pseudo-scientific and conspiratorial dredge words are targeted by dubious
sellers, podcasters, and social media influencers (see Appendix A for a case
study). The model seems to have over-relied on these easy signals, which also
harmed its PF1. A deeper exploration of these relationships is warranted, and
would be a fruitful path for future work. As these websites are out-of-domain
for our task, but still often contain unreliable content, we chose not
evaluate Precision@k for this set.

Model | P@5 | P@10 | P@20 | PF1 | PP | PR
---|---|---|---|---|---|---
$\mathcal{E}_{u,w,c}$ | 1 | 1 | 0.9 | 0.25 | 0.18 | 0.35
$\mathcal{E}_{u,w,c,f,v,d}$ | * | * | * | 0.07 | 0.04 | 0.26
SM-BD | - | - | 0.7 | 0.29 | 0.24 | 0.37
WG-BD | 0.2 | 0.5 | 0.65 | 0.27 | 0.24 | 0.31
$\text{WG-BD}_{d}$ | 0.8 | 0.7 | 0.55 | 0.02 | 0.17 | 0.01
Table 2: Precision@K and Partial F1 (PF1), Partial Precision (PP), and Partial
Recall (PR) discovery evaluations from Chen and Freire (2020). We compare our
heterogeneous GNN and heterogeneous dredge word GNN with the social media-
based discovery (SM-BD) proposed in Chen and Freire (2020) and webgraph based
discovery (WG-BD) proposed in Carragher et al. (2024), and a modified version
of WG-BD that incorporates dredge words ($\text{WG-BD}_{d}$).

##  6 Conclusion

We propose a model for unreliable domain detection that incorporates many of
the diverse paths that users take to unreliable domains. We propose the idea
of Dredge Words and highlight their bidirectional connections with social
media. Finally, we demonstrate that our best-performing proposed model
outperforms competing systems on the task of unreliable domain discovery.

## Acknowledgements

The research for this paper was supported in part by the Office of Naval
Research, MURI: Persuasion, Identity, & Morality in Social-Cyber Environments
under grant N000142112749 and by the Knight Foundation. It was also supported
by the center for Informed Democracy and Social-cybersecurity (IDeaS) and the
center for Computational Analysis of Social and Organizational Systems (CASOS)
at Carnegie Mellon University. The views and conclusions are those of the
authors and should not be interpreted as representing the official policies,
either expressed or implied, of the ONR or the US Government.

##  7 Limitations

While we attempted to mitigate limitations where possible, several substantial
ones exist. First, the Twitter data were collected from 2020-2022, but the
dredge words used in this paper were based on Google SERPs in 2024. We cannot
say that the dredge words we extracted surfaced the same content in 2024 as
when the phrases were initially used on Twitter. However, given the
conspiratorial, niche, and pseudo-scientific nature of many of the dredge
words we identified, we speculate that they would be unlikely to surface
substantial reliable content, particularly as reliable sites have had several
years to attempt to debunk—and thereby rank for—dredge word queries. Future
work could explore the impacts of better temporal alignment, which would have
the added benefit of covering data voids formed around breaking news stories.

The Partial F1 metric we use for discovery evaluation is limited, as previous
work has shown many PoliticalNews and MBFC sites on which the metric relies
are dead or no longer active Carragher et al. (2024). However, to our
knowledge, no better metric exists for evaluating unreliable domain discovery
systems. This is a needed avenue for future work. Additionally, the current
process for extracting dredge words relies on paid third-party services. While
we validated each of the extracted dredge words to ensure they are ranking in
Google’s top-10, approaches to extract keyphrases from, for example, common-
crawl webgraphs would provide more democratization of this line of research.
We publish all dredge word data collected for this project in line with that
goal.

Finally, dredge words were only extracted for 46 of the least reliable 11,327
labeled domains. Their inclusion in our models is therefore not particularly
helpful. We elected to include the results from these models because they
point to an interesting and understudied phenomenon: bidirectional paths
between social media and search engines, which appear to be often targeted by
monetary interests. In future work, we plan to repeat this process with dredge
words extracted for a larger set of unreliable domains, along with keywords
extracted from reliable domains.

##  8 Ethical Considerations

To avoid unintentionally boosting the search rankings of unreliable domains,
we do not directly link to any referenced unreliable domains. For unreliable
sites, we link to either archive.org or third-party reliability assessments of
those domains. We further acknowledge the inherent challenges in determining
the “reliability” of websites, and note that even reliable websites can
occasionally publish inaccurate content. We defaulted entirely to third
parties in our original acquisition of labels, but emphasize that any
reliability-labeling ontology should be scrutinized to minimize harms. We
would argue that the system we proposed could be generalized to any reasonable
reliability ontology.

Acknowledging that the Twitter data likely contains examples of “sensitive
events” which could compromise the privacy or safety of users, we adhere to
Twitter’s terms of service and do not release this
data888https://developer.x.com/en/developer-terms/agreement-and-policy.
Finally, we acknowledge that by publicly releasing these dredge-word lists,
bad actors could use these terms to direct people to unreliable information.
However, once data voids are identified, they are often plugged Norocel and
Lewandowski (2023). We hope that by releasing this data, reliable websites can
target these phrases to reduce both the visibility and the ease of access of
unreliable content.

## References

  * Alieva et al. (2022) Iuliia Alieva, Lynnette Hui Xian Ng, and Kathleen M Carley. 2022.  Investigating the spread of russian disinformation about biolabs in ukraine on twitter using social network analysis.  In _2022 IEEE international conference on big data (big data)_ , pages 1770–1775. IEEE.
  * Aswani et al. (2021) R. Aswani, S.P. Ghrera, S. Chandra, and A.K. Kar. 2021.  A hybrid evolutionary approach for identifying spam websites for search engine marketing.  _Evolutionary Intelligence_ , 14(4):1803–1815.
  * Barometer (2024) Edelman Trust Barometer. 2024.  2024 edelman trust barometer global report.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 41–48.
  * Carragher et al. (2024) Peter Carragher, Evan M Williams, and Kathleen M Carley. 2024.  Detection and discovery of misinformation sources using attributed webgraphs.  _arXiv preprint arXiv:2401.02379_.
  * Castelo et al. (2019) Sonia Castelo, Thais Almeida, Anas Elghafari, Aécio Santos, Kien Pham, Eduardo Nakamura, and Juliana Freire. 2019.  A topic-agnostic approach for identifying fake news pages.  In _Companion proceedings of the 2019 World Wide Web conference_ , pages 975–980.
  * Chappell and Yousef (2022) Bill Chappell and Odette Yousef. 2022.  How the false russian biolab story came to circulate among the u.s. far right.
  * Chen and Freire (2020) Zhouhan Chen and Juliana Freire. 2020.  Proactive discovery of fake news domains from real-time social media feeds.  In _Companion Proceedings of the Web Conference 2020_ , pages 584–592.
  * Cirik et al. (2016) Volkan Cirik, Eduard Hovy, and Louis-Philippe Morency. 2016.  Visualizing and understanding curriculum learning for long short-term memory networks.  _arXiv preprint arXiv:1611.06204_.
  * Danilák (2014) Michal Danilák. 2014.  langdetect.  https://github.com/Mimino666/langdetect.
  * Evon (2022) Dan Evon. 2022.  Ukraine, us biolabs, and an ongoing russian disinformation campaign.
  * Golebiewski and Boyd (2019) Michael Golebiewski and Danah Boyd. 2019.  Data voids: Where missing data can easily be exploited.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable feature learning for networks.  In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 855–864.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30.
  * Hu et al. (2020) Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020.  Heterogeneous graph transformer.  In _Proceedings of the web conference 2020_ , pages 2704–2710.
  * Kuznetsova et al. (2024) Elizaveta Kuznetsova, Mykola Makhortykh, Maryna Sydorova, Aleksandra Urman, Ilaria Vitulano, and Martha Stolze. 2024.  Algorithmically curated lies: How search engines handle misinformation about us biolabs in ukraine.  _arXiv preprint arXiv:2401.13832_.
  * League (2022) Anti-Defamation League. 2022.  Unmasking “clandestine,” the figure behind the viral “ukrainian biolab” conspiracy theory.
  * Lin et al. (2023) Hause Lin, Jana Lasser, Stephan Lewandowsky, Rocky Cole, Andrew Gully, David G Rand, and Gordon Pennycook. 2023.  High level of correspondence across different news domain quality rating sets.  _PNAS nexus_ , 2(9):pgad286.
  * McDuling (2015) John McDuling. 2015.  Google is now a more trusted source of news than the websites it aggregates.
  * Norocel and Lewandowski (2023) Ov Cristian Norocel and Dirk Lewandowski. 2023.  Google, data voids, and the dynamics of the politics of exclusion.  _Big Data & Society_, 10(1):20539517221149099.
  * Pan and Ding (2019) Shimei Pan and Tao Ding. 2019.  Social media-based user embedding: A literature review.  _arXiv preprint arXiv:1907.00725_.
  * Parachini (2022) John V. Parachini. 2022.  Debunking russian lies about biolabs at upcoming u.n. meetings.
  * Patricia Aires et al. (2019) Victoria Patricia Aires, Fabiola G. Nakamura, and Eduardo F. Nakamura. 2019.  A link-based approach to detect media bias in news websites.  In _Companion Proceedings of The 2019 World Wide Web Conference_ , pages 742–745.
  * Robertson et al. (2023) Ronald E Robertson, Amy Dunphy, Shelby Grossman, Renée DiResta, and David Thiel. 2023.  Identifying search directives on social media.  _Journal of Online Trust and Safety_ , 2(1).
  * Robertson and Wilson (2020) Ronald E Robertson and Christo Wilson. 2020.  Websearcher: Tools for auditing web search.  In _Proceedings of the 2020 Computation+ Journalism Symposium (Boston, MA, USA)(C+ J 2020)_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Sehgal et al. (2021) Vibhor Sehgal, Ankit Peshin, Sadia Afroz, and Hany Farid. 2021.  Mutual Hyperlinking Among Misinformation Peddlers.  ArXiv:2104.11694 [cs].
  * Silva et al. (2021) Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie. 2021.  Embracing domain differences in fake news: Cross-domain fake news detection using multi-modal data.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 557–565.
  * Spitkovsky et al. (2010) Valentin I Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. 2010.  From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 751–759.
  * Tripodi (2019) Francesca Tripodi. 2019.  Devin nunes and the power of keyword signaling.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_.
  * Williams and Carley (2023) Evan M Williams and Kathleen M Carley. 2023.  Search engine manipulation to spread pro-kremlin propaganda.  _Harvard Kennedy School Misinformation Review_.
  * Zandt (2022) Dave Van Zandt. 2022.  Media bias fact check: A comprehensive media bias resource.
  * Zhang and Cabage (2017) Sonya Zhang and Neal Cabage. 2017.  Search Engine Optimization: Comparison of Link Building and Social Sharing.  _Journal of Computer Information Systems_ , 57(2):148–159.

##  Appendix A Dredge Word EDA

###  A.1 Dredge Word SERPs

To understand the distributions of websites surfaced in SERPs, we will provide
an illustrative example. One could imagine a user browsing a social media
platform and coming across a tweet or post like the one in Figure 3.

Figure 3: A tweet about “indigo children”.

A user encountering new, potentially perplexing information, might then be
curious and compelled to search “indigo children meaning” or just “indigo
children” on Google—both phrases are dredge words. We provide the results for
former query in Figure 4. We note that Google’s definition snippet expresses
pseudo-scientific concepts. YouTube videos, followed by a Reddit post from the
/r/Psychic subreddit were the fourth-to-the-seventh highest-ranked URLs when
we manually searched this dredge word phrase in June of 2024. Another user who
may have searched only “indigo children”, would have seen a snippet from
Wikipedia which correctly calls indigo children pseudoscientific. However, the
user would have seen four books for sale about indigo children alongside the
snippet, and the second highest-ranking return was an article from an
unreliable domain identifying the “13 signs you’re an indigo child”. Were the
user to then scroll down to the 6th result, they would see an Amazon link to a
book on indigo children.

Figure 4: SERP results for “Indigo Children Meaning”

.

###  A.2 Dredge EDA

We applied the python library langdetect Danilák (2014) to the list of 3,933
dredge words used in the paper, but we observe that automated language
extraction of dredge words is a challenge. We observed many dredge words are
disembodied fragments, names of people, or simply foreign words adopted by
various communities. For example, several Hindi-language-origin yoga poses
appear in the dredge word list: parivrtta anjaneyasana and parivrtta
utkatasana. These terms surface gaia, a conspiracy pseudoscience
website999https://mediabiasfactcheck.com/gaia/. Some websites rank highly for
the same terms in multiple langauges; the arabic and chinese words for (coral)
both surface creation, a website that frequently promotes
pseudoscience101010https://mediabiasfactcheck.com/christian-ministries-
international/. Qualitatively, hatespeech is fairly uncommon in the dredge
words we identified; 11 unique dredge word phrases contain the non-euphemistic
writing of the n-word and 3 others contain a homophobic slur.

##  Appendix B Curriculum EDA

Among the websites rated most reliable, the style, content, and widespread
name recognition would very likely result in most annotators to correctly
identifying the sites as reliable (e.g., reuters.com, nasa.gov,
smithsonianmag.com, nature.com). The landing pages of some of the least
reliable labeled domains are also typically easy to categorize, with websites
that contain toxic green backgrounds accompanied by blurry photos of George
Soros
111111https://web.archive.org/web/20240112053008/http://www.endgamethemovie.com/,
proclamations of white nationalism
121212https://web.archive.org/web/20240108044820/https://www.stormfront.org/forum/,
or full links to alien siting websites and paid psychic services
131313https://web.archive.org/web/20240116140224/https://rense.com/. Though
our webgraph model is content-agnostic, we hypothesize that the webgraph and
site attributes will exhibit similarly "easy" patterns across the extremes. To
test this hypothesis, we implement a curriculum learning batching procedure
Bengio et al. (2009).

##  Appendix C Partial F1 threshhold sensitivity

To investigate the partial F1 metric we experiment with various classifier
confidence thresholds. Naturally, as confidence increases, we see higher
precision and lower recall (Figure 5). The disparity between partial recall
and partial precision implies that our discovery process is not very precise.
However, in the manual evaluation of Precision@k, we find precision is
actually very high. This indicates a fundamental issue with the partial
precision metric; it underestimates model performance, particularly as the
MBFC reference list ages. This is in line with previous findings that
unreliable domain lists quickly become outdated (Carragher et al., 2024), and
it complicates the evaluation of discovery processes. Further work on domain
reliability discovery metrics is needed.

For Precision@k assessments, two annotators were asked “is this website
unreliable” and asked to provide a rationale for why or why not. Annotations
involved exploration of the domain and lateral reading—looking at what other
reliable sources have written about the domain. We publicly release these
annotations alongside annotator rationales.141414Following acceptance.

Figure 5: GNN discovery performance vs. classifier confidence reveals that the
Partial F1 metric is precision bounded.