# Q-Boost: On Visual Quality Assessment Ability of Low-level Multi-Modality
Foundation Models

###### Abstract

Recent advancements in Multi-modality Large Language Models (MLLMs) have
demonstrated remarkable capabilities in complex high-level vision tasks.
However, the exploration of MLLM potential in visual quality assessment, a
vital aspect of low-level vision, remains limited. To address this gap, we
introduce Q-Boost, a novel strategy designed to enhance low-level MLLMs in
image quality assessment (IQA) and video quality assessment (VQA) tasks, which
is structured around two pivotal components: 1) Triadic-Tone Integration:
Ordinary prompt design simply oscillates between the binary extremes of
$positive$ and $negative$. Q-Boost innovates by incorporating a ‘middle
ground’ approach through $neutral$ prompts, allowing for a more balanced and
detailed assessment. 2) Multi-Prompt Ensemble: Multiple quality-centric
prompts are used to mitigate bias and acquire more accurate evaluation. The
experimental results show that the low-level MLLMs exhibit outstanding zeros-
shot performance on the IQA/VQA tasks equipped with Q-Boost strategy.

Index Terms—  Multi-modality large language models, Image quality assessment,
Triadic-tone integration, Multi-prompt ensemble, Video quality assessment,
Zero-shot

##  1 Introduction

Multi-modality Large Language Models (MLLMs) have exhibited strong
capabilities in general visual perception and understanding, which have been
tested and confirmed in various vision-language tasks, including image
captioning, visual question answering, cross-modality grounding, as well as
standard vision tasks like image classification or segmentation. While the
focus has largely been on high-level visual content perception and
understanding, the proficiency of MLLMs in visual quality assessment remains
less understood.

Fig. 1: Motivation of Q-Boost. The Triadic-Tone Integration strategy helps
provide a more balanced and comprehensive assessment while the Multi-Prompt
Ensemble strategy helps improve the accuracy and reliability of evaluation.

Over the past decade, a significant amount of research has been dedicated to
exploring image quality assessment (IQA) and video quality assessment (VQA).
In the initial phases, researchers extract handcrafted, quality-aware features
from images or videos, subsequently regressing these features to derive
quality scores [1, 2, 3]. Subsequently, the advent and success of deep neural
networks (DNNs) have paved the way for the use of deep-learning models to
extract and regress quality features into corresponding scores [4, 5, 6, 7, 8,
9]. Regardless of the extraction technique, these quality assessment
methodologies predominantly depend on mean opinion scores (MOSs). The
fundamental learning approach involves minimizing the discrepancy between
these features and MOSs to obtain insightful knowledge for quality assessment.
While MOS regression is straightforward and enables models to directly assess
quality, it suffers from a lack of interpretability. In contrast, the text-
supervised MLLM can output quality assessment results in natural language
format and make explanations, which operate more similarly as human
perception. Given the rapid advancements in MLLM, integrating it into quality
assessment tasks is not only meaningful but also expands the horizon beyond
traditional MOS regression. This approach facilitates training AI through text
supervision to gain quality knowledge and designing prompts for MLLM to
predict quality scores effectively.

Fig. 2: The framework of the Q-Boost. The image and prompt are fed into the
MLLM, where the log probabilities (logits) are computed between the
[SCORE_TOKEN] and triadic-tone words (including ensembles of multiple
prompts). Then the logits of different tones are put through softmax operation
and integrated into the zero-shot quality score with weighted average pooling.

The design of prompts like “Rate the quality of the image” and “Score the
quality of the image from 1 to 5” is a natural approach for eliciting feedback
from MLLMs regarding image quality. Nevertheless, these strategies have been
shown to exhibit bias [10], as responses from MLLMs are often positively
skewed and extreme. It is more recommended to transfer the IQA task into a
binary classification problem on the probabilities of responding “Rate the
quality of the image” with good and bad from the MLLMs, where the
probabilities are quantified into quality scores via softmax. This softmax-
based evaluation strategy has been used to benchmark the assessment abilities
of various MLLMs [10] and even revealed that MLLMs are actually effective
zero-shot assessment assistants after multi-modality low-level dataset tuning
[11].

To further enhance this methodology, we introduce Q-Boost, a novel advancement
designed to significantly improve the performance of MLLMs on IQA [6, 4, 12]
as well as IQA-based video quality assessment (VQA) [13, 14, 15, 16, 17].
Q-Boost consists of two strategies: 1) The traditional prompt design, which
typically fluctuates between the binary extremes of $positive$ and $negative$,
neglects the fact that some images might be of medium quality. Q-Boost
proposes a significant innovation Triadic-Tone Integration (TTI) in this
context, by introducing $neutral$ prompts such as average, medium, and
acceptable. This not only enables a more balanced and comprehensive assessment
but also enriches the evaluation process with greater detail, moving beyond
the simplistic binary paradigm. 2) Q-Boost presents Multi-Prompt Ensemble
(MPE) to enhance the accuracy of the assessment results. On one hand, multi-
prompt expands the semantic scope, capturing nuances that a single word may
overlook. On the other hand, multi-prompt reduces ambiguity as different terms
carry slightly varied connotations, leading to more reliable evaluation. The
experimental results show that the MLLM with Q-Boost achieves state-of-the-art
zero-shot performance on both IQA/VQA datasets.

|  |  |
---|---|---|---
$\mathcal{{P}}_{log}(good)$ | $\mathcal{{P}}_{log}(average)$ | $\mathcal{{P}}_{log}(poor)$ | GT | $\mathcal{{P}}_{log}(good)$ | $\mathcal{{P}}_{log}(average)$ | $\mathcal{{P}}_{log}(poor)$ | GT | $\mathcal{{P}}_{log}(good)$ | $\mathcal{{P}}_{log}(average)$ | $\mathcal{{P}}_{log}(poor)$ | GT | $\mathcal{{P}}_{log}(good)$ | $\mathcal{{P}}_{log}(average)$ | $\mathcal{{P}}_{log}(poor)$ | GT
11.14 | 8.68 | 7.96 | 4.12/5 | 10.02 | 10.71 | 9.07 | 3.11/5 | 8.92 | 10.75 | 9.59 | 2.51/5 | 6.46 | 8.62 | 10.70 | 1.10/5

Table 1: Example log probabilities (logits) of good, average, and poor. It is
evident that images of lower quality exhibit a stronger correlation with
negative prompts, whereas those of higher quality demonstrate a greater
alignment with positive prompts. Images classified as being of medium quality
tend to show increased neutral logits.

##  2 The Q-Boost

The assessment framework of MLLM (Q-Boost) is illustrated in Fig. 2. First, we
describe the process of getting log probabilities (logits) of certain prompts
from MLLM. The image along with the prompt “Rate the quality of the image. The
quality of this image is [SCORE_TOKEN]” is input into the MLLM and the logits
can be derived as the probability distance between [SCORE_TOKEN] and the
certain prompt words such good and bad:

$\mathcal{P}_{log}(\mathcal{W})=\mathrm{MLLM}(\mathcal{T},\mathcal{I}),$ (1)

where $\mathcal{P}_{log}(\mathcal{W})$ represents the log probabilities of
certain prompt words, $\mathrm{MLLM}(\cdot)$ indicates the MLLM assessment
operation, $\mathcal{T}$ stands for the input prompt text, and $\mathcal{T}$
denotes the input image.

###  2.1 Triadic-Tone Integration (TTI)

In many text-prompted quality assessment works [12, 18], only binary extremes
(positive and negative) are taken into consideration, which is difficult to
distinguish the quality of images with medium quality as shown in Table 1.
Therefore, we introduce a third tone neutral to such as average, medium, and
acceptable to provide a more balanced and comprehensive evaluation:

$\displaystyle\mathcal{Q}_{pos},$
$\displaystyle\mathcal{Q}_{neu},\mathcal{Q}_{neg}=$ (2)
$\displaystyle\gamma(\mathcal{\bar{P}}_{log}(\mathcal{W}_{pos}),\mathcal{\bar{P}}_{log}(\mathcal{W}_{neu}),\mathcal{\bar{P}}_{log}(\mathcal{W}_{neg})),$

where
$\mathcal{\bar{P}}_{log}(\mathcal{W}_{pos}),\mathcal{\bar{P}}_{log}(\mathcal{W}_{neu}),\mathcal{\bar{P}}_{log}(\mathcal{W}_{neg})$
represent the averaged logits of positive, neutral, negative prompt words (see
in Sec. 2.2), $\gamma(\cdot)$ indicates the softmax function. Then the
integrated quality scores can be obtained as follows:

$\mathcal{Q}=w_{1}\mathcal{Q}_{pos}+w_{2}\mathcal{Q}_{neu},$ (3)

where $w_{1}=1.0$ and $w_{2}=0.5$ are the weighted parameters for
$\mathcal{Q}_{pos}$ and $\mathcal{Q}_{neu}$.

###  2.2 Multi-Prompt Ensemble (MPE)

Generally speaking, multi-prompt enhance the semantic range, grasping
subtleties that a single word might miss. Conversely, they minimize ambiguity
since different terms have slightly different implications, resulting in a
more dependable evaluation. Therefore, we propose to use a group of synonym
words instead of a single word to describe one certain tune:

$\bar{\mathcal{P}}_{log}(\mathcal{W}_{\alpha})=\frac{1}{n_{\alpha}}\sum{\mathcal{P}}_{log}(\mathcal{W}_{\alpha}),$
(4)

where $\alpha\in\\{pos,neu,neg\\}$,
$\bar{\mathcal{P}}_{log}(\mathcal{W}_{\alpha})$ represents the averaged logits
of tune $\alpha$, and $n_{\alpha}$ stands for the number of the words for
describing tune $\alpha$.

###  2.3 Video Evaluation

The process described above can directly get the quality score of a single
image. For videos, we extract the frames every second and average the frames’
logits as the video logits:

$\bar{\mathcal{P}}_{log}^{v}(\mathcal{W}_{\alpha})=\frac{1}{n_{v}}\sum_{i=1}^{n_{v}}\bar{\mathcal{P}}_{log}^{i}(\mathcal{W}_{\alpha}),$
(5)

where $\alpha\in\\{pos,neu,neg\\}$, $n_{v}$ indicates the length of the video,
$\bar{\mathcal{P}}_{log}^{i}(\mathcal{W}_{\alpha})$ represents the $i$-th
frame’s logits, and $\bar{\mathcal{P}}_{log}^{v}(\mathcal{W}_{\alpha})$ stands
for the video’s logits. Then the video quality can be calculated with the same
interaction process described above.

Table 2: Review of the IQA/VQA datasets.

Type | Dataset | Num. | Content
---|---|---|---
IQA | KonIQ-10k [19] | 10,741 | In-the-wild
KADID-10K [20] | 10,125 | Artificially Distorted
CGIQA-6K [21] | 6,000 | Computer Graphics Images
VQA | LIVE-VQC [22] | 585 | In-the-wild
KoNViD-1k [23] | 1,200 | In-the-wild
Youtube-UGC [24] | 1,380 | In-the-wild
| MaxWell (val) [14] | 909 | In-the-wild

Table 3: Results on the IQA datasets. Both mPO-7B (Q-Instruct) and mPO-7B
(Q-Boost) utilize the Q-Instruct [11]-tuned weights. mPO-7B (Q-Instruct) is
validated without the Q-Boost strategy. About 48.92% images of the KonIQ-10k
are included in the Q-Instruct and 0% for other datasets. Best zero-shot
performance is marked in bold.

Datasets | KonIQ-10k | KADID-10K | CGIQA-6K
---|---|---|---
Model | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
Opinion-Aware Approaches
DBCNN [4] | 0.884 | 0.875 | 0.856 | 0.851 | 0.662 | 0.664
HyperIQA [5] | 0.917 | 0.906 | 0.845 | 0.852 | 0.636 | 0.660
MUSIQ [6] | 0.915 | 0.937 | 0.572 | 0.584 | 0.625 | 0.641
Zero-shot Approaches
NIQE [1] | 0.316 | 0.377 | 0.374 | 0.428 | 0.075 | 0.056
IL-NIQE [2] | 0.537 | 0.523 | 0.558 | 0.534 | 0.108 | 0.082
CLIP-IQA [12] | 0.468 | 0.505 | 0.501 | 0.520 | 0.294 | 0.300
mPO-7B (Q-Instruct [11]) | 0.902 | 0.888 | 0.698 | 0.684 | 0.629 | 0.644
mPO-7B (Q-Boost) | 0.912 | 0.923 | 0.702 | 0.699 | 0.630 | 0.648

Table 4: Results on the VQA datasets. The best and second zero-shot
performance are marked in bold and underline.

Datasets | LIVE-VQC | KoNViD-1k | YouTube-UGC | MaxWell (val)
---|---|---|---|---
Model | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
Opinion-Aware Approaches |  |
TLVQM [25] | 0.799 | 0.803 | 0.773 | 0.768 | 0.669 | 0.659 | 0.661 | 0.652
VSFA [7] | 0.773 | 0.795 | 0.773 | 0.775 | 0.724 | 0.743 | 0.676 | 0.678
VIDEVAL [26] | 0.752 | 0.751 | 0.783 | 0.780 | 0.779 | 0.773 | 0.597 | 0.601
Zero-shot Approaches |
(Spatial, Classical) NIQE [1] | 0.596 | 0.628 | 0.541 | 0.553 | 0.278 | 0.290 | 0.312 | 0.301
(Temporal, Classical) TPQI [27] | 0.636 | 0.645 | 0.556 | 0.549 | 0.111 | 0.218 | 0.361 | 0.377
(CLIP-based) SAQI [18] | 0.629 | 0.638 | 0.608 | 0.602 | 0.585 | 0.606 | 0.541 | 0.559
(Ensemble of Three Above) BVQI [18] | 0.784 | 0.794 | 0.760 | 0.760 | 0.525 | 0.556 | 0.637 | 0.648
Spatial) mPO-7B (Q-Instruct [11]) | 0.720 | 0.657 | 0.692 | 0.603 | 0.539 | 0.540 | 0.554 | 0.515
(Spatial) mPO-7B (Q-Boost) | 0.741 | 0.793 | 0.801 | 0.803 | 0.723 | 0.705 | 0.682 | 0.692

###  2.4 Inference Cost

Q-Boost does not introduce additional computational complexity, as the entire
computation process essentially takes place after the input prompt is entered
and the language backbone returns the [SCORE_TOKEN]. The subsequent MPE simply
involves tokenizing the words used, and then calculating the logits on the
[SCORE_TOKEN]. As the computational cost of the softmax and score weighting of
TTI is much less than the inference of the MLLM, the additional computation
cost of the Q-Boost is almost negligible. This is different from CLIP-based QA
methods [12, 18], in which each prompt word needs to pass through the text
backbone independently. Thus, applying a similar multi-prompt strategy to
CLIP-based QA methods would significantly increase inference costs.

##  3 Experiment

###  3.1 Implementation Details

####  3.1.1 Experimental Setup

The mPLUG-Owl-2-7B [28] (abbr. as mPO-7B) with CLIP-ViT-Large-14 vision
backbone and LLaMA2-7B language backbone is used for validation. The mPO-7B
weights are loaded from the instruction-tuning process of Q-Instruct [11] and
frozen during the inference process. The mPO-7B (Q-Instruct) indicates using
Q-Instruct-tuned weights without the Q-Boost strategy. The strategy-enhanced
MLLM model is denoted as mPO-7B (Q-Boost). Only the TTI strategy is employed
for IQA while both TTI and MPE strategies are utilized for VQA (reasons are
discussed in Sec. 3). In the MPE strategy, (good \+ high \+ fine
$\leftrightarrow$ average \+ medium \+ acceptable $\leftrightarrow$ poor \+
low \+ bad) are used as the groups of words for describing (positive
$\leftrightarrow$ neutral $\leftrightarrow$ negative) tunes. The weighted
parameters $w_{1}$ and $w_{2}$ mentioned in Sec. 2.1 are set as 1 and 0.5
respectively.

(a) SRCC (b) PLCC

Fig. 3: The SRCC (Fig. (a)) and PLCC (Fig. (b)) performance comparison of the
best zero-shot competitor, mPO-7B (Q-Instruct), and mPO-7B (Q-Boost), where
the index values are calculated as $\frac{SRCC}{SRCC_{max}}$ and
$\frac{PLCC}{PLCC_{max}}$. It can be seen that mPO-7B (Q-Boost) achieves the
best performance in general and significantly boosts the performance of mPO-7B
(Q-Instruct) on the VQA datasets. Table 5: Experimental results for the
ablation study. good $\leftrightarrow$ / $\leftrightarrow$ poor indicates
using only binary extremes prompts good and poor, good $\leftrightarrow$
average $\leftrightarrow$ poor represents using Triadic-Tone Integration
without Multi-Prompt Ensemble, g+h+f $\leftrightarrow$ a+m+ac
$\leftrightarrow$ p+l+b stands for utilizing prompts of (good+high+fine
$\leftrightarrow$ average+medium+acceptable $\leftrightarrow$ poor+low+bad),
which employs both strategies.

Datasets | IQA | VQA
---|---|---
KonIQ-10k | KADID-10K | CGIQA-6K | LIVE-VQC | KoNViD-1k | YouTube-UGC | MaxWell (val)
Prompts | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
good $\leftrightarrow$ / $\leftrightarrow$ poor | 0.902 | 0.888 | 0.698 | 0.684 | 0.629 | 0.644 | 0.720 | 0.657 | 0.692 | 0.603 | 0.539 | 0.539 | 0.554 | 0.516
good $\leftrightarrow$ average $\leftrightarrow$ poor | 0.912 | 0.923 | 0.702 | 0.699 | 0.630 | 0.648 | 0.740 | 0.725 | 0.766 | 0.701 | 0.577 | 0.583 | 0.611 | 0.589
g+h+f $\leftrightarrow$ a+m+ac $\leftrightarrow$ p+l+b | 0.898 | 0.910 | 0.704 | 0.678 | 0.558 | 0.577 | 0.741 | 0.793 | 0.801 | 0.803 | 0.723 | 0.705 | 0.682 | 0.692

####  3.1.2 Datasets Introduction

3 IQA datasets and 4 VQA datasets are utilized for performance comparison,
which are briefly reviewed in Table 2. About 48.92% images of the KonIQ-10k
[19] dataset are included in the Q-Instruct [11]-tuning stage and no images of
the rest datasets are seen by the mPO-7B (Q-Instruct & Q-Boost). Additionally,
the CGIQA-6K [21] dataset contains two separate sub-sets which consist of
3,000 game images and 3,000 movie images respectively and we average the
performance of the sub-sets for the final exhibition.

####  3.1.3 IQA/VQA Competitors

Both opinion-aware (need supervised training) and zero-shot methods (training-
free) are included for comparison.

$\bullet$ The opinion-aware IQA methods consist of DBCNN [4], HyperIQA [5],
and MUSIQ [6]. $\bullet$ The zero-shot IQA methods contain NIQE [1], IL-NIQE
[2], and CLIP-IQA [12]. $\bullet$ The opinion-aware VQA methods include TLVQM
[25], VSFA [7], and VIDEVAL [26]. $\bullet$ The zero-shot VQA methods contain
NIQE [2], TPQI [27], and SAQI [18].

###  3.2 Performance Discussion

The experimental performance on the IQA and VQA datasets is illustrated in
Table 2.3 and Table 2.3, from which we can obtain several interesting
findings. Finding 1: Both mPO-7B (Q-Instruct) and mPO-7B (Q-Boost) have
outperformed the existing zero-shot NR-IQA methods by a large margin and are
even competitive with the option-aware approaches on the IQA datasets. This
reveals the powerful potential of MLLM in IQA tasks. Moreover, compared to
traditional MOS (mean opinion score) regression training models, MLLM is text-
supervised, which means that using text supervision instead of MOS regression
can achieve similar or even better zero-shot performance. This also opens up a
possible new training path in the QA field, allowing us to train and enhance
AI’s QA capabilities in a more intuitive and explanatory manner. Finding 2:
The mPO-7B (Q-Boost) significantly advances the mPO-7B (Q-Instruct) and
achieves impressive zero-shot performance on the VQA datasets even without
considering temporal quality information. The mPO-7B (Q-Boost) model
outperforms all compared methodologies, encompassing supervised VQA
techniques, on both the KoNViD-1k and MaxWell (val) datasets. This
demonstrates the superior assessment capabilities and remarkable
generalization proficiency of MLLM when augmented with Q-Boost in the realm of
VQA. The progress of Q-Boost stems from leveraging the advantages of
diversity, eliminating biases, and providing a more comprehensive and thorough
assessment.

###  3.3 Ablation Study

To quantify the contributions of the TTI and MPE strategies, we conduct the
ablation study in this section, the results of which are shown in Table 5. 1)
It is observed that the TTI strategy yields a modest enhancement in the
performance of IQA), whereas the MPE strategy appears to have negligible
impact. This phenomenon could be attributed to the more explicit nature of
IQA, wherein the ensemble of prompts may inadvertently introduce elements of
confusion. Consequently, we recommend the exclusive use of TTI as the Q-Boost
approach for optimizing IQA tasks. 2) Both strategies make significant
contributions to enhancing the VQA understanding of MLLM. This is because the
MLLM’s weights are pre-trained with image-text pairs, with a gap to the video
data. Thus these strategies effectively mitigate biases arising from this data
gap and utilize the diversity of prompts to enable more accurate and
comprehensive assessment.

##  4 Conclusion

In conclusion, this paper has made significant strides in bridging the gap in
the application of MLLM for visual quality assessment, a crucial area in the
realm of low-level vision tasks. Our introduction of Q-Boost marks a pivotal
advancement in this field. The unique incorporation of Triadic-Tone
Integration, which departs from the traditional binary prompt approach to
include a neutral perspective, provides a more nuanced and balanced
evaluation. Additionally, the Multi-Prompt Ensemble technique effectively
mitigates biases and enhances assessment accuracy. The experimental results
unequivocally illustrate that MLLM equipped with the Q-Boost strategy exhibits
outstanding zero-shot performance on both IQA and VQA tasks. We believe that
the Q-Boost strategy will serve as a catalyst for further advancements in the
quality assessment field and inspire new methodologies in the broader context
of MLLM for low-level vision.

## References

  * [1] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik,  “Making a “completely blind” image quality analyzer,”  IEEE SPL, 2012.
  * [2] Lin Zhang, Lei Zhang, and Alan C Bovik,  “A feature-enriched completely blind image quality evaluator,”  IEEE TIP, 2015.
  * [3] Zicheng Zhang, Wei Sun, Xiongkuo Min, Wenhan Zhu, Tao Wang, Wei Lu, and Guangtao Zhai,  “A no-reference evaluation metric for low-light image enhancement,”  in IEEE ICME, 2021.
  * [4] Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang,  “Blind image quality assessment using a deep bilinear convolutional neural network,”  IEEE TCSVT, vol. 30, no. 1, pp. 36–47, 2020.
  * [5] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang,  “Blindly assess image quality in the wild guided by a self-adaptive hyper network,”  in CVPR, 2020, pp. 3667–3676.
  * [6] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang,  “Musiq: Multi-scale image quality transformer,”  in CVPR, 2021, pp. 5148–5157.
  * [7] Dingquan Li, Tingting Jiang, and Ming Jiang,  “Quality assessment of in-the-wild videos,”  in ACM MM, 2019, p. 2351–2359.
  * [8] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin,  “Exploring video quality assessment on user generated contents from aesthetic and technical perspectives,”  in ICCV, 2023.
  * [9] Zicheng Zhang, Wei Wu, Wei Sun, Danyang Tu, Wei Lu, Xiongkuo Min, Ying Chen, and Guangtao Zhai,  “Md-vqa: Multi-dimensional quality assessment for ugc live videos,”  in CVPR, 2023.
  * [10] Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin,  “Q-bench: A benchmark for general-purpose foundation models on low-level vision,”  2023\.
  * [11] Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, Geng Xue, Wenxiu Sun, Qiong Yan, and Weisi Lin,  “Q-instruct: Improving low-level visual abilities for multi-modality foundation models,” 2023.
  * [12] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy,  “Exploring clip for assessing the look and feel of images,”  in AAAI, 2023.
  * [13] Dingquan Li, Tingting Jiang, and Ming Jiang,  “Quality assessment of in-the-wild videos,”  in ACM MM, 2019, pp. 2351–2359.
  * [14] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin,  “Towards explainable in-the-wild video quality assessment: a database and a language-prompted approach,”  ACM MM, 2023.
  * [15] Haoning Wu, Chaofeng Chen, Liang Liao, Jingwen Hou, Wenxiu Sun, Qiong Yan, Jinwei Gu, and Weisi Lin,  “Neighbourhood representative sampling for efficient end-to-end video quality assessment,” 2023.
  * [16] Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin,  “Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,”  in ECCV, 2022.
  * [17] Haoning Wu, Chaofeng Chen, Liang Liao, Jingwen Hou, Wenxiu Sun, Qiong Yan, and Weisi Lin,  “Discovqa: Temporal distortion-content transformers for video quality assessment,”  IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2023.
  * [18] Haoning Wu, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Erli Zhang, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin,  “Exploring opinion-unaware video quality assessment with semantic affinity criterion,”  in International Conference on Multimedia and Expo (ICME), 2023.
  * [19] Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe,  “Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,”  IEEE TIP, vol. 29, pp. 4041–4056, 2020.
  * [20] Hanhe Lin, Vlad Hosu, and Dietmar Saupe,  “Kadid-10k: A large-scale artificially distorted iqa database,”  in QoMEX, 2019, pp. 1–3.
  * [21] Zicheng Zhang et al.,  “Subjective and objective quality assessment for in-the-wild computer graphics images,”  ACM TOMM, 2023.
  * [22] Zeina Sinno and Alan Conrad Bovik,  “Large-scale study of perceptual video quality,”  IEEE TIP, vol. 28, no. 2, pp. 612–627, 2019.
  * [23] Vlad Hosu, Franz Hahn, Mohsen Jenadeleh, Hanhe Lin, Hui Men, Tamás Szirányi, Shujun Li, and Dietmar Saupe,  “The konstanz natural video database (konvid-1k),”  in QoMEX, 2017, pp. 1–6.
  * [24] Yilin Wang, Sasi Inguva, and Balu Adsumilli,  “Youtube ugc dataset for video compression research,”  in 2019 MMSP, 2019.
  * [25] Jari Korhonen,  “Two-level approach for no-reference consumer video quality assessment,”  IEEE TIP, vol. 28, no. 12, pp. 5923–5938, 2019.
  * [26] Zhengzhong Tu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, and Alan C. Bovik,  “Ugc-vqa: Benchmarking blind video quality assessment for user generated content,”  IEEE TIP, vol. 30, pp. 4449–4464, 2021.
  * [27] Liang Liao, Kangmin Xu, Haoning Wu, Chaofeng Chen, Wenxiu Sun, Qiong Yan, and Weisi Lin,  “Exploring the effectiveness of video perceptual representation in blind video quality assessment,”  in ACM MM, 2022.
  * [28] Qinghao Ye et al.,  “mplug-owl: Modularization empowers large language models with multimodality,” 2023.


# Impact of the COVID-19 outbreak on the Italian Twitter vaccination debate: A
network–based analysis

Veronica Lachi  University of Siena, Siena, Italy  Giovanna Maria Dimitri
University of Siena, Siena, Italy  Alessandro Di Stefano  University of
Teesside, United Kingdom  Pietro Liò  University of Cambridge, Cambridge,
United Kingdom  Monica Bianchini  University of Siena, Siena, Italy  Equal
contribution  Chiara Mocenni  University of Siena, Siena, Italy  Equal
contribution

###### Abstract

Vaccine hesitancy, or the reluctance to be vaccinated, is a phenomenon that
has recently become particularly significant, in conjunction with the
vaccination campaign against COVID-19. During the lockdown period, necessary
to control the spread of the virus, social networks have played an important
role in the Italian debate on vaccination, generally representing the easiest
and safest way to exchange opinions and maintain some form of sociability.
Among social network platforms, Twitter has assumed a strategic role in
driving the public opinion, creating compact groups of users sharing similar
views towards the utility, uselessness or even dangerousness of vaccines. In
this paper, we present a new, publicly available, dataset of Italian tweets,
TwitterVax, collected in the period January 2019–May 2022. Considering monthly
data, gathered into forty–one retweet networks — where nodes identify users
and edges are present between users who have retweeted each other —, we
performed community detection within the networks, analyzing their evolution
and polarization with respect to NoVax and ProVax users through time. This
allowed us to clearly discover debate trends as well as identify potential key
moments and actors in opinion flows, characterizing the main features and
tweeting behavior of the two communities.

_K_ eywords Twitter, COVID-19, vaccination, Pro/NoVax community detection,
social networks

##  1 Introduction

Vaccines are one of the most powerful weapons to fight infectious diseases.
The use of vaccines has, in fact, helped to drastically reduce epidemic
mortality rates in the 20th century [1]. Nonetheless, we can observe the
presence of an ongoing phenomenon, called vaccine hesitancy, which can be
identified as a delay in acceptance or a clear refusal of vaccination, despite
the availability of vaccination services [2]. Indeed, vaccination skepticism
is a phenomenon that has existed since the first vaccine became available.
However, vaccine hesitancy is currently a growing global attitude — supported
and amplified by the ease of finding controversial information on the Internet
— that can pose a problem in avoiding the outbreak of communicable diseases.
Therefore, recognizing the importance of the phenomenon and the high risk to
which it exposes the world population, especially in disadvantaged areas, is
essential [3]. For this purpose, the Strategic Advisory Group of Experts
(SAGE) on Immunization of the World Health Organization (WHO) has established,
since 2012, a specific working group on the subject, led by a joint WHO/Unicef
Secretariat group.
The reasons that lead to hesitancy or refusal have been widely studied in the
last few decades. For example, in [4], it is shown that increasing levels of
perceived economic hardship are associated with vaccine hesitancy, just as
less parental education is significantly associated with vaccine refusal for
children. However, not only socio–economic conditions are linked to vaccine
hesitancy, which is a multifaceted phenomenon where cognitive, psychological
and cultural factors all play a critical role [5, 6, 7, 8].
Recently, due to the outbreak of the COVID-19 pandemic, the issue of vaccine
hesitancy has become particularly important. Despite overwhelming evidence
showcasing the effectiveness of vaccines [9], surpassing even alternative
measures like contact tracing and lockdowns[10, 11], a large swath of the
population has continued to refuse inoculation, endangering public health and
economic and social life [12]. In this specific situation, in fact, social
opinions on COVID-19 vaccines were negatively affected also by concerns
related to the unprecedented speed with which they were developed [13], with
sometimes confusing media communications and unregulated social media
information sources fomenting such anti–vaccination sentiment — not to mention
conspiracy theories, which include population control through 5G technology or
the extermination of humanity through vaccines [14, 15, 16]. This phenomenon
has been exasperated by the presence of so–called echo chambers, virtual
environments where like–minded people reinforce their opinions through
repeated interactions that amplify their political leanings, beliefs and
attitudes [17, 18, 19, 20, 21]. Such a mechanism can lead to increasingly
polarized debates and phenomena of extremism [22]. To get an idea of the
diffusion of certain forms of thinking, the Annual Report 2021 by Censis
(Centro Studi Investimenti Sociali, https://www.censis.it/rapporto-annuale-
censis) reveals that: ”For 5.9% of Italians (about 3 million) Covid does not
exist, for 10,9% the vaccine is useless, for 31.4% it is an experimental drug
and people who get vaccinated act as guinea pigs, for 12.7% science produces
more harm than good [$\dots$], for 19.9% 5G is a sophisticated tool to monitor
people”.
Similarly to many other political and social issues, social networks represent
a natural source of aggregation and a powerful medium for the largely
uncontrolled dissemination of information.Undoubtedly, social media has had a
considerable impact on the health sector, as the user sentiment can be used to
understand collective panic or for the dissemination of reliable and
unreliable medical claims [23]. Additionally, social media has a recognized
role in healthcare campaigns, both targeted at businesses and by government
agencies and non-profits to combat rumors, encourage behavioral change, and
share information, enabling the audience to engage and share feedback.
Specifically, Twitter is one of the most used social media [24] and can also
be identified as a popular source of health information. For these reasons, it
can provide realistic insights on the society perception regarding
vaccination. Twitter administration itself explicitly tried to prevent the
dissemination of misleading information regarding COVID-19, publishing a
series of rules for users (https://help.twitter.com/it/rules-and-
policies/medical-misinformation-policy), yet leaving to their common sense the
respect of the code of ethics on the matter.
The aim of this work is that of exploiting Twitter data to study the evolution
over the past three years of the Italian vaccine debate since, recently,
vaccines have become a very political issue in Italy [25]. The contribution of
this paper is manifold. First, we show that, with the outbreak of COVID-19,
the vaccine debate has changed: it has significantly intensified and,
furthermore, the discussion has gone from being widespread among all
interested users (mostly parents of preschoolers) to being concentrated in the
hands of a few influential hubs. Second, we have detected and monitored the
NoVax and ProVax communities. The relative proportion of the two communities
has not changed significantly over time and ProVax users are the most numerous
but also the least active. Moreover, we have identified core NoVax users as
well as core ProVax users, demonstrating that the former outnumber the latter
— who have, anyway, more followers, mostly among verified users. Finally, we
provide a new dataset of Italian tweets, TwitterVax, collected between January
2019 and May 2022, which is publicly available at
https://github.com/veronicalachi/TwitterVax.111Despite being in contrast with
the Twitter ToS, we made the decision to publicly share the tweet texts. We
did so because we firmly believe that this act brings significant benefits to
the research community.
The paper is organized as follows. In Section 2, we present an overview of
related works in the context of vaccine debates and social network analysis.
In Section 3, we describe our approach to data collection and analysis, while
in Section 4 we present and discuss the obtained results. Finally, Section 5
collects some conclusions and outlines future research perspectives.

##  2 Related works

Several works in the literature — many of which make use of network–based
approaches — investigate the structure and characteristics of the vaccination
debate.
In [26], an analysis of Twitter data and official vaccination coverage rates
showed that vaccine opinions from Twitter users could affect the vaccination
decision–making process; moreover, the application of a community detection
algorithm led to the identification of two user communities: one in support of
vaccination, including important and influential users, and one against
vaccination, characterized by a lower level of interaction. Using a structural
network approach, in [27], it was shown that vaccination skeptics, as well as
advocates, reside in their own distinct echo chambers and that the structure
of these two communities is different, with skeptics organized into highly
connected clusters and supporters characterized by the presence of influential
hub users. A social network approach was also used in [28] to demonstrate that
NoVax users retweeted frequently each other while ProVax users formed a
fragmented network, with fewer connections; moreover, while the ProVaxes were
mostly healthcare workers, the NoVaxes were mainly parents and activists. In
[29], seven different communities were identified, including health workers,
writers and journalists, anti–establishment people and international
vaccination advocates; contents shared by the healthcare community hardly
reached other communities, while messages tweeted by anti–establishment users
managed to filter to other communities.
If several works have been proposed studying the structure of the vaccine
debate and the interactions between vaccination advocates and skeptics, still
little research has been conducted on how the outbreak of the COVID-19
pandemic has influenced this controversial discussion. In [30], a comparison
of tweets from the four months prior to the onset of COVID-19 and tweets from
the four months following the outbreak of the pandemic reveals that vaccine
opponents on Twitter increased by 80%. Finally, in [31], the COVID-19 debate
is analyzed by focusing on people who interacted and shared opinions on
Twitter. Based on a sentiment dataset, called COVIDSenti, composed of 90 000
COVID-19–related tweets collected in the early stages of the pandemic (from
February to March 2020), it was shown that, at the onset of the pandemic,
people favored lockdown while, as expected, sentiment shifted by mid–March.
As far as the Italian scenario is concerned, so far few works have been
published. In [32], a new collected dataset was used to describe the
polarization and volumes of tweets, only focusing on a few months just after
the vaccination campaigns (between October 2020 and January 2021). The dataset
was crawled using only the four words "vaccino/i" and "vaccinazione/i" and
manually annotated, showing a higher percentage of NoVax users. Furthermore,
in [33], sentiment analysis was proposed for COVID-19 vaccine hesitancy posts,
collected from several different social networks, with an attention towards
the COVID-19 vaccine or specifically the booster shot. In [34], more than 316
million Twitter messages were gathered between October 2019 and March 2021 to
identify the amount of disinformation flows among users of different
countries. Indeed, the tweets were written in more than eighteen languages
(including Italian), with a focus on the global debate rather than considering
national realities. Finally, in [35], a comprehensive study was conducted by
collecting a dataset of 16,223,749 Italian tweets spanning from September 2019
to December 2021. The main objective was to investigate the impact of
unprecedented experiences and measures related to the COVID-19 pandemic on
polarization in vaccine discussions. The results obtained clarified that,
despite the outbreak of COVID-19, the echo chamber phenomenon within vaccine
discussions persisted.
The aim of the present work is to extend the research on the Italian
vaccination discussion, particularly characterizing the structural evolution
of the debate and investigating the changes in the NoVax and ProVax
communities in terms of size, productivity, and core users.

##  3 Materials & methods

Twitter, a widely recognized social networking platform, had its data publicly
accessible via its application programming interface (API) until February
2023. In this paper, we use a network–based approach to study the evolution of
Italian public opinion on vaccines over time. Transforming the unstructured
information available from Twitter into graph data has allowed us not only to
employ common metrics used on networks to estimate their topological features
but also to apply some powerful algorithms for community detection.

###  3.1 Data Collection

To perform data collection, the Twitter API v2 has been employed. In
particular, the Twitter tweets/search/all
endpoint222https://developer.twitter.com/en/docs/twitter-
api/tweets/search/api-reference/get-tweets-search-all has been used, which
allows us to search the full archive of tweets and to filter them using a set
of keywords. By crawling the Twitter archive based on twenty Italian
vaccine–related terms as keywords, we were able to collect 9,068,389 tweets —
contributing to the vaccination debate — posted by 300,653 users from 1
January 2019 to 31 May 2022. In particular, the twenty keywords selected to
build the TwitterVax dataset were: vaccino, vaccini, vaccinazione,
vaccinazioni, vaccinare, vaccinarsi, vaccinatevi, vacciniamoci, vaccinando,
vaccinale, vaccinali, vaccinati, vaccinate, vaccinata, vaccinato, va@@ino,
va..ino, vaxino, vaxxino, #iomiovaccino. Some of the words, vaxxino for
example, were specific jargon related to the debate and were frequently used
in tweets by both the ProVax and NoVax communities. In contrast to the
approach taken in [35], we have made the decision to exclusively employ
Italian keywords in our analysis. In Figure 1, the most common words in our
dataset are shown, where the font size is a quantitative indicator of the
amount of occurrences of a specific word in the TwitterVax collection.
Moreover, we further filtered the tweets to return Italian texts only.
Finally, we decided to study the dynamics of users’ opinions about vaccines by
dividing the entire period into forty–one sub–periods, of the duration of one
month each. The one–month sampling resulted heuristically the one that best
preserves the information on the evolution of the community structure.
Interestingly, when using the Search API, some content is omitted from the
dataset, especially content that has been deleted or posted by suspended
users. The findings of [36] shed light on this issue, revealing that a
considerable proportion (72%) of tweets captured through the Streaming Search
API at the time of posting were not retrieved via the Historical Search API.
This stark contrast in retrieval rates serves as a strong indication that a
significant amount of Twitter content is swiftly removed from the platform,
either due to user deletions or Twitter’s enforcement actions against
violations of community norms. This phenomenon highlights the dynamic nature
of the platform, wherein a considerable portion of tweets is ephemeral,
quickly disappearing from public view.

Figure 1: Keywords used in the Twitter API calls. The font size identifies the
number of times the various words occur in our dataset.

###  3.2 Network construction and indicators

Using the data from each month, we constructed an ordered set of forty–one
networks. In each graph, nodes represent the active users of the platform in
that period, while undirected edges connect users that have retweeted each
other at least once. This approach addresses a limitation highlighted in [35],
where an edge was considered only if there were at least two retweets,
resulting in the exclusion of over half of the initial users. Since retweeting
is the act of sharing another user’s post without modifying it, people tend to
retweet content they approve and to build a relationship with users with the
same opinion about certain topics [37, 38, 39]. Therefore, relationships
between users induce an “agreement network”, in which groups of people with
similar leaning are strongly connected to each other.
In order to reduce the dimension of the networks, tough keeping the most
significant information, only the largest connected components have been used.
In this way, the change in the discussion volume can be observed and the
evolution of the network structure — showing how the debate evolves in time —
can be described, by measuring its connectivity features. In particular, the
following metrics were considered in the analysis.

  * •

Density — It represents the proportion of possible relationships in the
network that are actually present [40] or, in other words, it is calculated as
the ratio of the number of edges in a graph over the number of edges of the
complete graph with the same number of nodes. Therefore, it provides a measure
of how dense a graph is in terms of edge connectivity.

  * •

Average clustering coefficient — The local clustering coefficient of a node in
a graph quantifies how close its neighbours are to being a clique (i.e., a
complete graph). In other words, when computed on a single node, the
clustering coefficient is a measure of how complete the neighborhood of a node
is. Therefore, the overall level of clustering in a network is measured by
averaging the clustering coefficient [40, 41] over all the network nodes.The
clustering coefficient of a single node is calculated as:

$c_{v}=\frac{2T(v)}{deg(v)(deg(v)-1)}$

where $T(v)$ is the the number of connected triangles including node $v$ and
$deg(v)$ is the degree of $v$.

  * •

s–metric — The s–metric [42] is a structural metric that provides a measure of
the extent to which a graph is scale–free. It is calculated as:

$s(\mathcal{G})=\sum_{(v,u)\in\mathcal{E}}deg(v)\times deg(u)$

where $\mathcal{G}=(\mathcal{V},\mathcal{E})$ is a graph, $\mathcal{V}$ is the
set of its vertices or nodes, $\mathcal{E}$ is the set of its edges, and
$deg(v)$ is the degree of the node $v$. A scale–free network [43] is a network
whose degree distribution follows a power law, at least asymptotically. The
main characteristic of scale–free networks is the presence of nodes with a
degree that significantly exceeds the average degree.

###  3.3 Community detection and user labeling

Since the vaccination discussion is highly contentious, it is reasonable to
assume that, at different stages, each active user takes a specific position
in the debate. In particular, given the high polarization of this topic [44],
we hypothesize that just two cohorts of people populate the platform: the
vaccination skeptics (NoVax) and the vaccination advocates (ProVax).
Classifying each user into these two categories for each month is not an easy
task. The simplest and most reliable way to label a user as NoVax or ProVax is
to analyze the content of their tweets, which is impossible to do manually,
due to the large amount of text. Furthermore, despite significant advancements
in natural language processing (NLP) methods for analyzing tweets, also for
COVID-19 contents [45, 46], this area of research continues to present ongoing
challenges. The brevity of tweets, constrained to a maximum of 280 characters,
coupled with their complex and unusual semantics, poses significant obstacles.
As a result, the effective utilization of NLP techniques in this domain
remains a subject of active investigation. Thus, we decided to use the
structure of the network to infer the opinion of the users.
Identifying communities of highly connected users within a network is one of
the most popular and challenging problems in network science [47, 48]. To
perform community detection we have used the multi–level graph partitioning
algorithm METIS [49]. The choice of METIS was driven by several reasons.
Firstly, the extensive usage of METIS in significant studies within the
literature [38, 27] ensures the comparability of our results with existing
research. Additionally, METIS has proven its effectiveness in handling retweet
networks [38]. Furthermore, the partitions generated by METIS consistently
exhibit superior quality, with improvements ranging from 0% to 50% compared to
spectral partitioning algorithms [50]. Indeed, extensive experimentation
across diverse graphs has demonstrated METIS’s superiority in terms of speed,
surpassing other widely utilized partitioning algorithms by one to two orders
of magnitude [51]. Finally, a clustering algorithm was recommended in which
the number of communities was defined in advance, as we aimed to specifically
identify two communities (ProVaxes and NoVaxes).
The idea behind METIS is to create successively smaller graphs
$\mathcal{G}_{1}$, $\mathcal{G}_{2}$, $\ldots$, $\mathcal{G}_{k}$ from
$\mathcal{G}_{0}=(\mathcal{V},\mathcal{E})$, to obtain a partition of
$\mathcal{G}_{k}$ in a short time, and project the partition back onto
$\mathcal{G}_{0}$, while refining it at each step. In particular, METIS
consists of two stages: coarsening and refinement.

  1. 1.

Coarsening — The original graph, $\mathcal{G}_{0}$, is transformed into
sequentially smaller graphs $\mathcal{G}_{1}$, $\mathcal{G}_{2}$, $\ldots$,
$\mathcal{G}_{k}$, such that
$|\mathcal{V}_{0}|>|\mathcal{V}_{1}|>|\mathcal{V}_{2}|>\ldots>|\mathcal{V}_{k}|$.
If $\mathcal{G}_{k}$ is meant to be a good representation of
$\mathcal{G}_{0}$, a good partitioning of $\mathcal{G}_{k}$ represents a
fairly good partitioning of $\mathcal{G}_{0}$. $\mathcal{G}_{k}$ is small
enough to make partitioning very quick.

  2. 2.

Refinement — The partition $\mathcal{P}_{k}$ is projected back onto
$\mathcal{P}_{k-1}$, $\ldots$, $\mathcal{P}_{0}$. After each projection
$\mathcal{P}_{i}$, the partitioning is refined using a greedy algorithm. Each
partition $\mathcal{P}_{i}$, for $0\leq i\leq k-1$, is refined before
projecting to $\mathcal{P}_{i-1}$.

Following the procedure adopted in [27], we have applied METIS repeatedly one
hundred times, choosing two as the number of communities. This leads to a
vector of one hundred elements, with entries equal to 0 or 1, representing the
partition assignment for each user. Averaging the assigned partition across
the vector allows to obtain a score between 0 and 1, which represents the
probability that the corresponding user belongs to one of the two partitions.
The hyper–parameter of METIS, that is the relative size of the partitions, was
tuned by maximizing the number of users whose leaning score is in the 95%
confidence interval between 0 and 1. Finally, the tweets of the 10% of users
characterized by the most extreme scores have been read in order to classify
them manually as NoVax or ProVax. Every remaining user is assigned to the same
partition of the “extreme” user with the closest score. Using the resulting
communities of users, our aim is to monitor the change in the relative size of
the two partitions and to measure the polarization over time, where the
polarization is the relative density of the in–group agreement with respect to
the out–group agreement. Following [52], we calculated, for each timestamp
$t$, the polarization score $S_{t}$ as:

$S_{t}=\frac{(E_{n}+E_{p}-E_{o})}{(E_{n}+E_{p}+E_{o})}$ (1)

where $E_{n}$ is the edge density in the NoVax partition — calculated as the
number of observed edges within the community divided by the total amount of
the possible edges in the NoVax partition, i.e.,
$|\mathcal{V}_{n}|\times(|\mathcal{V}_{n}|-1)/2$ —, $E_{p}$ is the edge
density in the ProVax partition and, finally, $E_{o}$ is the density of edges
connecting the two partitions — calculated as the number of edges linking
NoVax and ProVax users normalized by the total amount of possible edges
between the two communities, i.e. $\mathcal{V}_{n}\times\mathcal{V}_{p}$. The
closer $S_{t}$ is to $1$, the denser the in–partition edges are compared with
the out–partition edges and the more polarized is the debate. Conversely, a
polarization score near $-1$ indicates that the ties of the network are
equally distributed within and between communities, meaning that the
discussion is not significantly polarized.

###  3.4 Vaccination debate analysis through the global multiplexity matrix

In order to provide a view of the evolution of the communities in time, we
constructed the global multiplexity matrix $M$, which has proven to be
successful in dynamical network analysis [53, 54]. In our case, both rows and
columns of $M$ correspond to active users, while each entry $M(i,j)$ counts
the number of times that users $i$ and $j$ belong to the same community in the
forty–one periods considered in our analysis. In fact, the monthly networks
would not really represent a multiplex. Multiplex networks [55, 56, 57] are,
indeed, made up of the same set of nodes over time. Therefore, we decided to
construct $M$ based on all the users who participated in the discussion at any
moment during the considered period of time — which is not an issue since our
analysis aimed to select only those nodes that always belong to the same
community, during the whole period. This revealed core NoVax users, as well as
core ProVax users over time.
Finally, we have identified whether the core communities (for NoVaxes and
ProVaxes) were composed by verified users or not, also counting the number of
connections of the corresponding nodes. Verified users — which in Twitter are
indicated with a blue tick — are people (journalists, actors, conductors,
singers, etc.) or companies whose identity has been checked by Twitter,
guaranteeing the profile authenticity. Therefore, identifying if the most
active and influential users, from both sides of the debate, belong to this
category represents an important issue. The number of verified users’
followers is also a fundamental indicator to quantify how much they are able
to spread information.

##  4 Results and discussion

###  4.1 Temporal evolution of the network structure

The retweeting relationship between users allowed us to construct forty–one
retweet networks, one for each month. The giant connected component present in
all the monthly networks includes more than 90% of the respective total number
of nodes. As can be deduced from Figure 3, both nodes and edges follow an
increasing trend over time until the beginning of 2022, witnessing a huge
growth of people interacting on vaccines. Such an increase in the volume of
the discussion, after the COVID-19 outbreak, was easily foreseeable. In
particular, a real soar of users and exchange of tweets was registered around
the 21st month considered in our analysis. This is not surprising, as October
2020 coincides with the onset of the second and more aggressive wave of
COVID-19 and, in turn, the spread of rumors about the imminent release of
anti–Covid vaccines. In fact, after a few months during which the debate was
weaker (see also Figure 6, where the bottleneck in the clustering leaning
score is visible), a strong peak can be observed in the number of nodes and
edges in early autumn 2020. After that, the volume of the debate seems to
remain high, until a final declining phase around April–May 2022, due to the
pandemic control with the consequent decrease of interest on the topic.
Interestingly, we detected a positive cross–correlation (with lag equal to
$-1$) between the time series of the Twitter users engaged in vaccine–related
discussions and the number of Google searches, based on the same keywords,
obtained from Google Trend (Figure 2).

Figure 2: Cross-Correlation between the time-series of the number of nodes and
the time-series of the research interest on Google Trend.

Figure 3: Evolution of the number of nodes (left) and edges (right) of the
retweet network in time (during the forty–one months considered). The plots
clearly show how the beginning of the second wave of COVID-19, which caused
the second lockdown and the highest number of deaths in Italy, has changed the
dimension of the debate significantly.

However, it is interesting to monitor how the structure of the debate has
evolved over time. For this purpose, we computed, for each monthly network,
some structural metrics: density, mean clustering coefficient, mean
node–degree and s–metric. Their values over time are plotted in Figure 4.
Retweet networks show a decrease in the density metric through early 2022
(Figure 4, top left). Thus, even if there is a growth over time in both the
number of nodes and edges, they do not have the same decreasing rate: as the
number of users grows, the connection between them becomes more sparse, making
the network less similar to a complete graph. This tendency is confirmed also
by the negative trend of the average clustering coefficient (Figure 4, top
right), which implies a decrease in the probability of detecting triangle
patterns within the network. Such findings, together with the rise of the
average node–degree (Figure 4, bottom left), lead to conclude that the
structure of the debate significantly changed over time. In particular,
although all the forty–one networks show a scale–free pattern — as revealed by
the power–law degree distribution (Figure 5) and by the low clustering
coefficient [58] —, at the beginning, the discussion looks more uniformly
spread all over the users, and later on it gets more and more concentrated in
the hands of a small number of influential nodes. These hubs play a key role
in the diffusion of opinions about vaccines. More specifically, hubs are nodes
characterized by a very high node–degree and most other users tend to retweet
them massively, building poor connections with low-degree nodes. The
increasing s–metric score (Figure 4, bottom right) confirms that the network
becomes more scale–free as months go by.
Opposite conclusions can be drawn for the structural changes of the debate
since the beginning of 2022: the end of the vaccination campaign and the
spread of the less aggressive Omicron variant seem to turn off the debate
(Figure 3) and to make it more similar — in terms of structure — to the
before–Covid era.
Monitoring the evolution of the network structure over time is crucial, as
changes in the structure can significantly impact the spread of information
and disinformation [59]. Indeed, such changes lead to a more efficient spread
of information, since a sparser network with reduced node clusters allows for
faster and wider dissemination of information. This is because the information
can travel quickly across the network without getting stuck in highly
clustered regions or bottlenecks. However, this also means that disinformation
can spread more efficiently as well, especially if the malicious actors
responsible for spam or fallacious content have managed to infiltrate the
network’s key nodes. Disinformation campaigns can take advantage of a sparse
network structure to target a wider audience, and the reduced clustering can
make it difficult for fact–checkers and debunkers to combat the spread of
false information.

Figure 4: Evolution of network metrics over time. In all the plots, the $x$
axis shows the month numbers, while the $y$ axis describes the values of the
four network measures considered (density, average clustering coefficient,
average degree and s–metric). The yellow box indicates the transition period,
reported also in Figure 3, approximately ranging from July 2020 to February
2021. Figure 5: Degree distribution of the retweet networks at months 1, 21,
31, 41. All the networks are scale–free.

###  4.2 Temporal evolution of the community composition and network
polarization

As previously explained, we applied the METIS algorithm one hundred times to
produce two partitions on the forty–one retweet networks. The result of the
procedure is presented in the heat–map of the leaning scores shown in Figure
6. Each row of the heat–map corresponds to a month (between 1 and 41) while
each column is a number between 0 and 1 (represented with at most two decimal
places), defining a possible leaning score. Indeed, the leaning score (see
Section 3.3) describes the probability that a user belongs to one of the two
communities. The color of each entry represents the fraction of users that, in
the corresponding sub–period, share that leaning score: the lighter the color
the higher the percentage of users characterized by that score. Intuitively, a
huge fraction of users assigned to one of the two extremes witnesses a highly
partitioned network. Based on Figure 6, as the months go by, there is a higher
percentage of users with extreme leaning scores. This suggests that the
network structures become increasingly partitioned, enabling the algorithm to
more accurately detect communities. However, a bottleneck is present
approximately in correspondence of August 2020–February 2021. In this period,
a new lockdown started, during which each region was assigned a color
corresponding to the spread of the virus and consequently to the severity of
the restrictions imposed on the population. This situation corresponds to a
highly uncertain time frame and is coherent with the abrupt increase in the
debate size, shown in Figures 3 and 4 (highlighted by the yellow box). Another
plausible explanation for this bottleneck could be the concurrent rollout of
the vaccination campaign, which garnered heightened public attention and may
have acted as a connecting factor between the two communities during that
particular period. After the bottleneck, a continuous increase of the
clustering leaning score is observed, indicating a stronger polarization of
the debate.

Figure 6: Heat–map of the leaning scores. The $x$ axis represents the leaning
score value (which varies in [0,1]), while the $y$ axis shows the numbering of
the months. The lighter the color the greater the percentage of users sharing
the same leaning score.

In order to characterize the two communities found by the algorithm, the
tweets of the users with the most extreme scores were qualitatively read and
assessed. For each month, users with similar extreme leaning scores tweeted
similar content, guaranteeing the fairness of the partitioning procedure.
Every other node of the network is then classified based on the community
membership of the “extreme” user with the closest score. Four examples of the
retweet networks, with the respective communities, are reported in Figure 7.
From now on the blue and orange colors will indicate ProVax and NoVax users,
respectively.

Figure 7: Graph representation of the retweet networks at months 1, 21, 31,
41; the blue and orange nodes represent the ProVax and NoVax users,
respectively.

Once the two communities were identified, it was possible to study the
relative proportion of vaccination skeptics and advocates for each sub–period
(Figure 8). Before the 27th month (March 2021), the relative proportion of
ProVax and NoVax users did not follow a particular trend and, mostly, the
former were more numerous than the latter. Since April 2021, however, vaccine
advocates have become progressively less active, being outnumbered by skeptics
from January 2022.

Figure 8: Relative proportion of NoVax (orange) and ProVax (blue) users over
the forty–one months, from January 2019 to May 2022.

An interesting insight we could gather from our analysis is also given by the
number of tweets exchanged within the two communities. As shown in Figure 9,
NoVax users were generally more productive than ProVax users. The latter
tweeted more than the former only during the first five months of the
vaccination campaign. Starting from the 28th month (April 2021) the
productivity of the two groups begins to follow two different trends: NoVax
users become increasingly productive, while the number of tweets from ProVax
users decreased significantly.

Figure 9: Total number of tweets from NoVax (orange) and ProVax (blue) users
for each month.

Furthermore, with the beginning of the COVID-19 vaccination campaign, the
subsequent political issues around the green pass, and the introduction of
mandatory vaccination law for some categories of workers, the vaccination
skeptics became not only more present in the debate, but also more and more
active. Going into detail, the four peaks of the ProVax curve correspond
exactly to the administration of the vaccine to medical personnel (December
2020) and to the first, second and third dose for the entire population
(indicatively, April, July and November 2021). In the case of NoVax users,
instead, the two most significant peaks relate to the imposition of the green
pass obligation (August 6, 2021) and the order of the Ministry of Health
(February 8, 2022) for the cessation of the obligation to wear outdoor masks,
while the local minimum in April 2022 corresponds to the end of the state of
emergency. The correspondence between the peaks of ProVax and NoVax activities
and national–level events suggests a substantial difference of base motivation
between the two groups, with the NoVax being mostly driven by self–interest
considerations. Amidst the vast body of research on vaccine hesitancy,
particularly in the context of the COVID-19 pandemic [60, 61, 12, 62], some
studies have focused on exploring the impact of social trust levels on
vaccination rates [63, 64, 65]. In particular, our findings align with those
of a study conducted in Italy [66], which examined the relationship between
vaccination status, social context, social trust, and adherence to core
institutional structures, such as the rule of law and collective commitments.
The study revealed that individuals with higher levels of social trust are
less likely to remain unvaccinated against COVID-19. Conversely, unvaccinated
individuals show less support for honoring other collective commitments
unrelated to COVID-19, compared to vaccinated individuals, ceteris paribus.
These findings highlight the importance of considering a social contract
perspective, alongside the social context, in the study of vaccine hesitancy.
As such, they have significant implications for guiding policymakers in
developing effective strategies to promote vaccination. Specifically, they
suggest that appeals emphasizing individual benefits may be more successful in
encouraging vaccination compared to appeals centered around collective
responsibility.
Finally, the polarization score described in Eq. (1) was computed for each
sub–period (Figure 10). The polarization has followed an increasing trend over
time with an oscillating phase corresponding to late 2020 and early 2021. This
confirmed that in the month of the so–called “Vax–day” (December 2020) not
only the network is less clearly divided but the debate is also less
polarized. These observations provide further evidence supporting the
assertions made in [35], namely that the arrival of COVID-19 has failed to
alleviate the echo chamber effect in vaccine discussion. On the contrary, our
analysis indicates that the advent of COVID-19 has exacerbated the
polarization of the debate.

Figure 10: Polarization score over the forty–one months considered.

###  4.3 Possible strategies to slow down vaccine misinformation

By identifying communities over time, we can develop effective strategies to
curb the spread of vaccine misinformation and limit the growth of the NoVax
community. One promising approach is to pinpoint nodes with a central role in
the NoVax community and disrupt their influence through node attacks, such as
account blocking. To determine node importance, we employed the betweenness
centrality metric for NoVax nodes. Analyzing the betweenness of NoVax nodes
enabled us to examine the pathways of misinformation spreading. Specifically,
we analyzed the subgraph induced by the NoVax nodes at each time step and
calculated the distribution of betweenness among the nodes in the subgraph.
Our analysis revealed that all the distributions followed a power–law trend,
indicating that only few nodes had high betweenness at each time step. We then
determined the number of nodes with extremely high betweenness by calculating
the percentage of nodes outside the 95th percentile of the distribution for
each time step (Figure 11). Although the number of nodes with high betweenness
increased over time, they constituted a small portion of the total nodes.
Therefore, account blocking could be an effective strategy to hinder their
ability to spread misinformation. Other node attacks, based on metrics such as
degree and closeness, could also fragment the NoVax community.

Figure 11: Fraction of NoVax users characterized by a betweenness centrality
outside the 95th percentile.

###  4.4 Characterization of the core NoVax/ProVax users

The global multiplexity matrix $M$, of dimension 300,653$\times$300,653, is
found to be sparse. Each entry $M(i,j)$ is an integer between 0 and 41
representing the number of months in which the two users $i$ and $j$ belong to
the same community. First, we extracted all pairs of users who have been in
the same community throughout the period, i.e. pairs of users whose relative
matrix entry equals 41. Then, we used the extracted information to construct a
new network connecting only such selected users. The resulting graph (Figure
12) has seven connected components, whose dimensions are shown in Table 1.

Figure 12: Users that belong to the same community for forty–one months, forty months and thirty–nine months. The two colors (blue/orange) represent the two core communities (ProVax/NoVax) identified among these subgroups. Table 1: Dimension and composition of the seven connected components composed by users who belong to the same community for forty–one months; the percentage is calculated as the relative number of times that the nodes inside the connected components represent NoVax/ProVax users  # Nodes in Connected Component | NoVax | ProVax
---|---|---
183 | 100% | 0%
10 | 2.4% | 97.6%
8 | 95.1% | 4.9%
4 | 4.8% | 95.2%
4 | 4.9% | 95.1%
2 | 5.1% | 94.9%
2 | 97.5% | 2.5%

It is worth noting that each component is fully connected because requiring
that nodes belong to the same community forty–one times — which is the maximum
possible — is a transitive membership relation. As we can see in Table 1, the
largest connected component is totally composed of users that have always been
NoVax, whereas ProVax convinced users populate the second largest component.
The other five connected components are composed by few users that have
changed their mind about vaccines at least one time during the forty–one
months. However, it is also possible that some convinced and active users have
not tweeted during some periods. Considering this, we decided to relax the
constraint by requiring that nodes belong to the same community just forty
times. In this case, the graph (Figure 12) has four connected components — no
more fully–connected —, with the largest one composed by the core NoVax users
(Table 2).

Table 2: Dimension and composition of the four connected components composed by users who belong to the same community for forty months  # Nodes in Connected Component | NoVax | ProVax
---|---|---
312 | 96.5% | 3.5%
60 | 3.2% | 96.8%
2 | 95.1% | 4.9%
2 | 95% | 5%

Decreasing the number of periods to thirty–nine, the two core NoVax and ProVax
sets became larger and more stable: the graph at this point contains only two
connected components (Figure 12), whose dimension and composition are shown in
Table 3. Based on Table 3, we can also conclude that the NoVax supporters are
more numerous and stable over time, whereas the core set of ProVax users is
significantly smaller. Anyway, the most convinced and active ProVax users have
100200 followers on average, while NoVaxes have generally less followers
(44067), even if they are more constant and more present in the debate.
Furthermore, none of the users who make up the NoVax backbone are verified
users, while 15.85% of vaccine advocates are.
Another noteworthy observation pertains to the structure of the core NoVax and
ProVax users. As depicted in Figures 10b and 10c, it is evident that the NoVax
users exhibit a significantly more homogeneous structure compared to the
ProVax users. This characteristic may be closely tied to their efficacy in
disseminating false information. As highlighted in [67], the act of sharing
false information, even if subsequently debunked, tends to correlate with
individuals either postponing vaccination or outright rejecting it.

Table 3: Dimension and composition of the two connected components composed by users who belong to the same community for thirty–nine months # Nodes in Connected Component | NoVax | ProVax
---|---|---
405 | 95.6% | 4.4%
97 | 3.3% | 96.7%

Figure 13 shows a selection of the most popular users on both sides.
Interestingly, among ProVaxes, the most followed user is Dr. Roberto Burioni,
a famous Italian virologist who played an important role in health information
during the COVID-19 pandemic and even before. The most influential and active
NoVax user is instead Byoblu, a counter–information blog founded by Claudio
Messora, a well–known member of the NoVax community in Italy, as well as an
advocate of conspiracy theories on the health dictatorship.

Figure 13: Selection of the three most followed users in the ProVax (blue) and
NoVax (orange) core sets. The texts are translated from Italian to English
using Google Translate.

##  5 Conclusion

In Italy, the debate on vaccines has always been very heated and COVID-19 has
contributed to exacerbating it. Therefore, the discussion triggered by the
beginning of the pandemic on social networks provides an accurate picture of
the perception of Italians on this burning issue.
In order to study the temporal evolution of the vaccination debate during the
last three years, we collected the new dataset TwitterVax, using the Twitter
API and crawling tweets related to several vaccine keywords. This first
collection step allowed us to exploit Twitter data from a network perspective,
in order to understand the changes in the structure, scale and polarization of
the vaccine dispute in Italy, starting from January 2019 to May 2022. Based on
the collected data, we showed that, with the outbreak of COVID-19, the debate
has not only dramatically intensified, but has also become more concentrated
in the hands of a few influential hubs, who have played a vital role in
disseminating vaccine information. However, this trend seems to have reversed
since the beginning of 2022, testifying that the change in the structure of
the discussion was probably due to the COVID-19 emergency. We successfully
detected the NoVax and ProVax communities, demonstrating that the relative
proportion of the two user cohorts does not change significantly over time. In
particular, the NoVax community is often the least populated but also the most
active in terms of the number of tweets. Moreover, we computed a polarization
score between the two user groups, demonstrating an increasing level of
polarization. Finally, using a multiplexity approach, we identified the core
of NoVax and ProVax users. In this way, we have verified that core NoVaxes are
more than core ProVaxes who, however, have more followers on average and a
higher percentage of verified users.
Future perspectives include the possibility to collect a more complete data
set, covering a larger time period, as well as process the data according to a
higher resolution. It could also be interesting to analyze higher order motifs
in temporal multiplex networks, which could give us further information on the
structure of the interaction networks [68, 69]. Thereby, other strategic
events that have had a profound impact on the debate can be identified and
analyzed, also using different network approaches and evaluation metrics.
Furthermore, our analysis did not consider the presence of artificial robots
(bots) in the vaccine debate on Twitter. Identifying and filtering bot users
can be a challenging task[70, 71], which is beyond the scope of this work.
However, analyzing the percentage of bots and investigating their influence on
diffusion and polarization processes would be valuable and will be the subject
of future research.

## References

  * [1] Centers for Disease Control, Prevention (CDC, et al.  Impact of vaccines universally recommended for children–united states, 1990-1998.  MMWR. Morbidity and mortality weekly report, 48(12):243–248, 1999\.
  * [2] Melanie Schuster, Juhani Eskola, Philippe Duclos, et al.  Review of vaccine hesitancy: Rationale, remit and methods.  Vaccine, 33(34):4157–4160, 2015.
  * [3] Clarissa Simas and Heidi J. Larson.  Overcoming vaccine hesitancy in low–income and middle–income regions.  Nature Reviews Disease Primers, 7:41, 2021.
  * [4] Chiara Bertoncello, Antonio Ferro, Marco Fonzo, Sofia Zanovello, Giuseppina Napoletano, Francesca Russo, Vincenzo Baldo, and Silvia Cocchio.  Socioeconomic determinants in vaccine hesitancy and vaccine refusal in italy.  Vaccines, 8(2):276, 2020.
  * [5] Matthew Browne, Patricia Thomson, Matthew Justus Rockloff, and Gordon Pennycook.  Going against the herd: psychological and cultural factors underlying the ‘vaccination confidence gap’.  PLoS one, 10(9):e0132562, 2015.
  * [6] Matthew J Hornsey, Emily A Harris, and Kelly S Fielding.  The psychological roots of anti-vaccination attitudes: A 24-nation investigation.  Health psychology, 37(4):307, 2018.
  * [7] Jamie Murphy, Frédérique Vallières, Richard P Bentall, Mark Shevlin, Orla McBride, Todd K Hartman, Ryan McKay, Kate Bennett, Liam Mason, Jilly Gibson-Miller, et al.  Psychological characteristics associated with covid-19 vaccine hesitancy and resistance in ireland and the united kingdom.  Nature communications, 12(1):1–15, 2021.
  * [8] Tiffany D Pomares, Alison M Buttenheim, Avnika B Amin, Caroline M Joyce, Rachael M Porter, Robert A Bednarczyk, and Saad B Omer.  Association of cognitive biases with human papillomavirus vaccine hesitancy: a cross-sectional study.  Human Vaccines & Immunotherapeutics, 16(5):1018–1023, 2020.
  * [9] Jerome H Kim, Florian Marks, and John D Clemens.  Looking beyond covid-19 vaccine phase 3 trials.  Nature medicine, 27(2):205–211, 2021.
  * [10] Giulia Cencetti, Gabriele Santin, Antonio Longa, Emanuele Pigani, Alain Barrat, Ciro Cattuto, Sune Lehmann, and Bruno Lepri.  Using real-world contact networks to quantify the effectiveness of digital contact tracing and isolation strategies for covid-19 pandemic.  medRxiv, pages 2020–05, 2020.
  * [11] Vincenzo Alfano and Salvatore Ercolano.  The efficacy of lockdown against covid-19: a cross-country panel analysis.  Applied health economics and health policy, 18:509–517, 2020.
  * [12] Malik Sallam.  Covid-19 vaccine hesitancy worldwide: a concise systematic review of vaccine acceptance rates.  Vaccines, 9(2):160, 2021.
  * [13] Shingai Machingaidze and Charles Shey Wiysonge.  Understanding covid-19 vaccine hesitancy.  Nature Medicine, 27(8):1338–1339, 2021.
  * [14] Cornelia Betsch, Noel T Brewer, Pauline Brocard, Patrick Davies, Wolfgang Gaissmaier, Niels Haase, Julie Leask, Frank Renkewitz, Britta Renner, Valerie F Reyna, et al.  Opportunities and challenges of web 2.0 for vaccination decisions.  Vaccine, 30(25):3727–3733, 2012.
  * [15] Talha Burki.  Vaccine misinformation and social media.  The Lancet Digital Health, 1(6):e258–e259, 2019.
  * [16] Anna Kata.  A postmodern pandora’s box: anti-vaccination misinformation on the internet.  Vaccine, 28(7):1709–1716, 2010.
  * [17] Wesley Cota, Silvio C Ferreira, Romualdo Pastor-Satorras, and Michele Starnini.  Quantifying echo chamber effects in information spreading over political communication networks.  EPJ Data Science, 8(1):1–13, 2019.
  * [18] Michela Del Vicario, Alessandro Bessi, Fabiana Zollo, Fabio Petroni, Antonio Scala, Guido Caldarelli, H Eugene Stanley, and Walter Quattrociocchi.  The spreading of misinformation online.  Proceedings of the National Academy of Sciences, 113(3):554–559, 2016.
  * [19] Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis.  The effect of collective attention on controversial debates on social media.  In Proceedings of the 2017 ACM on Web Science Conference, pages 43–52, 2017.
  * [20] Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis.  Political discourse on social media: Echo chambers, gatekeepers, and the price of bipartisanship.  In Proceedings of the 2018 world wide web conference, pages 913–922, 2018.
  * [21] Kieron O’Hara and David Stevens.  Echo chambers and online radicalism: Assessing the internet’s complicity in violent extremism.  Policy & Internet, 7(4):401–422, 2015.
  * [22] Mehdi Moussaïd, Henry Brighton, and Wolfgang Gaissmaier.  The amplification of risk in experimental diffusion chains.  Proceedings of the National Academy of Sciences, 112(18):5631–5636, 2015.
  * [23] Aldo Hernandez-Suarez, Gabriel Sanchez-Perez, Karina Toscano-Medina, Victor Martinez-Hernandez, Hector Perez-Meana, Jesus Olivares-Mercado, and Victor Sanchez.  Social sentiment sensor in twitter for predicting cyber-attacks using l1 regularization.  Sensors, 18(5):1380, 2018.
  * [24] Brooke Auxier and Monica Anderson.  Social media use in 2021.  Pew Research Center, 1:1–4, 2021.
  * [25] Federico Brandmayr.  Public epistemologies and intellectual interventions in contemporary italy.  International Journal of Politics, Culture, and Society, 34(1):47–68, 2021.
  * [26] Gema Bello-Orgaz, Julio Hernandez-Castro, and David Camacho.  Detecting discussion communities on vaccination in twitter.  Future Generation Computer Systems, 66:125–136, 2017.
  * [27] Alessandro Cossard, Gianmarco De Francisci Morales, Kyriaki Kalimeri, Yelena Mejova, Daniela Paolotti, and Michele Starnini.  Falling into the echo chamber: the italian vaccination debate on twitter.  In Proceedings of the International AAAI conference on web and social media, volume 14, pages 130–140, 2020.
  * [28] Elena Milani, Emma Weitkamp, and Peter Webb.  The visual vaccine debate on twitter: A social network analysis.  Media and Communication, 8(2):364–375, 2020.
  * [29] Roel O Lutkenhaus, Jeroen Jansz, and Martine PA Bouman.  Mapping the dutch vaccination debate on twitter: Identifying communities, narratives, and interactions.  Vaccine: X, 1:100019, 2019.
  * [30] Erika Bonnevie, Allison Gallegos-Jeffrey, Jaclyn Goldbarg, Brian Byrd, and Joseph Smyser.  Quantifying the rise of vaccine opposition on twitter during the covid-19 pandemic.  Journal of communication in healthcare, 14(1):12–19, 2021.
  * [31] Usman Naseem, Imran Razzak, Matloob Khushi, Peter W. Eklund, and Jinman Kim.  COVIDSenti: A large–scale benchmark twitter data set for COVID-19 sentiment analysis.  IEEE Transactions on Computational Social Systems, 8:1003–1015, 2021.
  * [32] Davide Gori, Francesco Durazzi, Marco Montalti, Zeno Di Valerio, Chiara Reno, Maria Pia Fantini, and Daniel Remondini.  Mis-tweeting communication: a vaccine hesitancy analysis among twitter users in italy.  Acta Bio Medica: Atenei Parmensis, 92(Suppl 6), 2021.
  * [33] Elena Bellodi, Alessandro Bertagnon, and Marco Gavanelli.  Comparing emotion and sentiment analysis tools on italian anti-vaccination for covid-19 posts.  In Proceedings of the Sixth Workshop on Natural Language for Artificial Intelligence (NL4AI 2022) co-located with 21th International Conference of the Italian Association for Artificial Intelligence (AI* IA 2022), 2022.
  * [34] Jacopo Lenti, Kyriaki Kalimeri, André Panisson, Daniela Paolotti, Michele Tizzani, Yelena Mejova, and Michele Starnini.  Global misinformation spillovers in the online vaccination debate before and during covid-19.  arXiv e-prints, pages arXiv–2211, 2022.
  * [35] Giuseppe Crupi, Yelena Mejova, Michele Tizzani, Daniela Paolotti, and André Panisson.  Echoes through time: Evolution of the italian covid-19 vaccination debate.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 102–113, 2022.
  * [36] Juergen Pfeffer, Angelina Mooseder, Luca Hammer, Oliver Stritzel, and David Garcia.  This sample seems to be good enough! assessing coverage and temporal reliability of twitter’s academic api.  arXiv preprint arXiv:2204.02290, 2022.
  * [37] Pablo Barberá, John T Jost, Jonathan Nagler, Joshua A Tucker, and Richard Bonneau.  Tweeting from left to right: Is online political communication more than an echo chamber?  Psychological science, 26(10):1531–1542, 2015.
  * [38] Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis.  Quantifying controversy on social media.  ACM Transactions on Social Computing, 1(1):1–27, 2018.
  * [39] Panagiotis Metaxas, Eni Mustafaraj, Kily Wong, Laura Zeng, Megan O’Keefe, and Samantha Finn.  What do retweets indicate? results from user survey and meta-review of research.  In Proceedings of the international AAAI conference on web and social media, volume 9, pages 658–661, 2015.
  * [40] Maarten Van Steen.  Graph theory and complex networks.  An introduction, 144, 2010.
  * [41] Duncan J Watts and Steven H Strogatz.  Collective dynamics of ‘small-world’networks.  nature, 393(6684):440–442, 1998.
  * [42] Lun Li, David Alderson, John C Doyle, and Walter Willinger.  Towards a theory of scale-free graphs: Definition, properties, and implications.  Internet Mathematics, 2(4):431–523, 2005.
  * [43] Albert-László Barabási and Réka Albert.  Emergence of scaling in random networks.  science, 286(5439):509–512, 1999.
  * [44] Ana Lucía Schmidt, Fabiana Zollo, Antonio Scala, Cornelia Betsch, and Walter Quattrociocchi.  Polarization of the vaccination debate on facebook.  Vaccine, 36(25):3606–3612, 2018.
  * [45] Martin Müller, Marcel Salathé, and Per E Kummervold.  Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter.  arXiv preprint arXiv:2005.07503, 2020.
  * [46] Anna Nguyen, Antonio Longa, Massimiliano Luca, Joe Kaul, and Gabriel Lopez.  Emotion analysis using multilayered networks for graphical representation of tweets.  IEEE Access, 10:99467–99478, 2022.
  * [47] Santo Fortunato and Darko Hric.  Community detection in networks: A user guide.  Physics reports, 659:1–44, 2016.
  * [48] Fanzhen Liu, Shan Xue, Jia Wu, Chuan Zhou, Wenbin Hu, Cecile Paris, Surya Nepal, Jian Yang, and Philip S Yu.  Deep learning for community detection: progress, challenges and opportunities.  arXiv preprint arXiv:2005.08225, 2020.
  * [49] George Karypis and Vipin Kumar.  A fast and high quality multilevel scheme for partitioning irregular graphs.  SIAM Journal on scientific Computing, 20(1):359–392, 1998.
  * [50] Stephen T Barnard and Horst D Simon.  Fast multilevel implementation of recursive spectral bisection for partitioning unstructured problems.  Concurrency: Practice and experience, 6(2):101–117, 1994.
  * [51] George Karypis and Vipin Kumar.  Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices.  1997\.
  * [52] Ted Hsuan Yun Chen, Ali Salloum, Antti Gronow, Tuomas Ylä-Anttila, and Mikko Kivelä.  Polarization of climate politics results from partisan sorting: Evidence from finnish twittersphere.  Global Environmental Change, 71:102348, 2021.
  * [53] Tiago Azevedo, Giovanna Maria Dimitri, Pietro Lió, and Eric R Gamazon.  Multilayer modelling of the human transcriptome and biological mechanisms of complex diseases and traits.  NPJ systems biology and applications, 7(1):1–13, 2021.
  * [54] Desislava Hristova, Alex Rutherford, Jose Anson, Miguel Luengo-Oroz, and Cecilia Mascolo.  The international postal network and other global flows as proxies for national wellbeing.  PloS one, 11(6):e0155976, 2016.
  * [55] Giovanna Maria Dimitri, Shruti Agrawal, Adam Young, Joseph Donnelly, Xiuyun Liu, Peter Smielewski, Peter Hutchinson, Marek Czosnyka, Pietro Lió, and Christina Haubrich.  A multiplex network approach for the analysis of intracranial pressure and heart rate data in traumatic brain injured patients.  Applied network science, 2(1):1–12, 2017.
  * [56] Federico Battiston, Vincenzo Nicosia, and Vito Latora.  The new challenges of multiplex networks: Measures and models.  The European Physical Journal Special Topics, 226(3):401–416, 2017\.
  * [57] Federico Battiston, Vincenzo Nicosia, and Vito Latora.  Structural measures for multiplex networks.  Physical Review E, 89(3):032804, 2014.
  * [58] Xiaoyan Wu and Zonghua Liu.  How community structure influences epidemic spread in social networks.  Physica A: Statistical Mechanics and its Applications, 387(2-3):623–630, 2008.
  * [59] James P Gleeson, Kevin P O’Sullivan, Raquel A Baños, and Yamir Moreno.  Effects of network structure, competition and memory time on social spreading phenomena.  Physical Review X, 6(2):021019, 2016.
  * [60] Gianmarco Troiano and Alessandra Nardi.  Vaccine hesitancy in the era of covid-19.  Public health, 194:245–251, 2021.
  * [61] Metin Yigit, Aslinur Ozkaya-Parlakay, and Emrah Senel.  Evaluation of covid-19 vaccine refusal in parents.  The Pediatric infectious disease journal, 40(4):e134–e136, 2021\.
  * [62] Mohammad S Razai, Umar AR Chaudhry, Katja Doerholt, Linda Bauld, and Azeem Majeed.  Covid-19 vaccination hesitancy.  Bmj, 373, 2021.
  * [63] Debendra Nath Roy, Mohitosh Biswas, Ekramul Islam, and Md Shah Azam.  Potential factors influencing covid-19 vaccine acceptance and hesitancy: A systematic review.  PloS one, 17(3):e0265496, 2022.
  * [64] Will Jennings, Gerry Stoker, Hannah Bunting, Viktor Orri Valgarsson, Jennifer Gaskell, Daniel Devine, Lawrence McKay, and Melinda C Mills.  Lack of trust, conspiracy beliefs, and social media use predict covid-19 vaccine hesitancy.  Vaccines, 9(6):593, 2021.
  * [65] Nasim Asadi Faezi, Pourya Gholizadeh, Moussa Sanogo, Amadou Oumarou, Maad Nasser Mohamed, Yacouba Cissoko, Mamadou Saliou Sow, Bakary Sayon Keita, Youssouf AG Mohamed Baye, Pasquale Pagliano, et al.  Peoples’ attitude toward covid-19 vaccine, acceptance, and social trust among african and middle east countries.  Health promotion perspectives, 11(2):171, 2021.
  * [66] Sarah E Kreps and Douglas L Kriner.  Resistance to covid-19 vaccination and the social contract: evidence from italy.  npj Vaccines, 8(1):60, 2023.
  * [67] Francesco Pierri, Brea L Perry, Matthew R DeVerna, Kai-Cheng Yang, Alessandro Flammini, Filippo Menczer, and John Bryden.  Online misinformation is linked to early covid-19 vaccination hesitancy and refusal.  Scientific reports, 12(1):5966, 2022.
  * [68] Chunying Li, Xiaojiao Guo, Weijie Lin, Zhikang Tang, Jinli Cao, and Yanchun Zhang.  Multiplex network community detection algorithm based on motif awareness.  Knowledge-Based Systems, 260:110136, 2023.
  * [69] Stefano Boccaletti, Ginestra Bianconi, Regino Criado, Charo I Del Genio, Jesús Gómez-Gardenes, Miguel Romance, Irene Sendina-Nadal, Zhen Wang, and Massimiliano Zanin.  The structure and dynamics of multilayer networks.  Physics reports, 544(1):1–122, 2014.
  * [70] Eiman Alothali, Nazar Zaki, Elfadil A Mohamed, and Hany Alashwal.  Detecting social bots on twitter: a literature review.  In 2018 International conference on innovations in information technology (IIT), pages 175–180. IEEE, 2018.
  * [71] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini.  The rise of social bots.  Communications of the ACM, 59(7):96–104, 2016.