11institutetext: West Virginia University, Morgantown, USA 22institutetext:
University of Pittsburgh, Pittsburgh, USA 33institutetext: University of
Aberdeen, Aberdeen, UK

# TE-SSL: Time and Event-aware Self Supervised Learning for Alzheimer’s
Disease Progression Analysis

Jacob Thrasher 11 Alina Devkota 11 Ahmed P. Tafti 22 Binod Bhattarai 33
Prashnna Gyawali 11 for the Alzheimer’s Disease Neuroimaging Initiative

###### Abstract

Alzheimer’s Disease (AD) represents one of the most pressing challenges in the
field of neurodegenerative disorders, with its progression analysis being
crucial for understanding disease dynamics and developing targeted
interventions. Recent advancements in deep learning and various representation
learning strategies, including self-supervised learning (SSL), have shown
significant promise in enhancing medical image analysis, providing innovative
ways to extract meaningful patterns from complex data. Notably, the computer
vision literature has demonstrated that incorporating supervisory signals into
SSL can further augment model performance by guiding the learning process with
additional relevant information. However, the application of such supervisory
signals in the context of disease progression analysis remains largely
unexplored. This gap is particularly pronounced given the inherent challenges
of incorporating both event and time-to-event information into the learning
paradigm. Addressing this, we propose a novel framework, Time and Event-aware
SSL (TE-SSL), which integrates time-to-event and event and data as supervisory
signals to refine the learning process. Our comparative analysis with existing
SSL-based methods in the downstream task of survival analysis shows superior
performance across standard metrics. The full code can be found here:
https://github.com/jacob-thrasher/TE-SSL ††*Data used in preparation of this
article were obtained from the Alzheimer’s Disease Neuroimaging Initiative
(ADNI) database (adni.loni.usc.edu). As such, the investigators within the
ADNI contributed to the design and implementation of ADNI and/or provided data
but did not participate in analysis or writing of this report. A complete
listing of ADNI investigators can be found here

###### Keywords:

Alzheimer’s Survival Analysis Self-supervised learning.

##  1 Introduction

Advancement in deep-learning-based medical image analysis have shown
remarkable promise in revolutionizing the study of Alzeimer’s Dementia (AD)
through medical imaging [20, 2, 19, 6], potentially aiding in the management
and treatments of the estimated 6.7 million people aged 65+ who live with AD
in the United States [3]. For instance, Chang et al. (2023) [6] developed MRI-
based deep learning framework for differentiating between Alzheimer’s disease,
temporal lobe epilepsy and healthy controls. These approaches have been shown
to be successful in identifying subtle patterns in brain images that may be
challenging for human detection. While significant strides have been made in
utilizing computer-assisted tools for detecting Alzheimer’s Disease, the focus
on progression analysis, such as time-to-event prediction, has been
comparatively less prominent. However, the ability to accurately predict the
trajectory of Alzheimer’s over time is crucial for early diagnosis and can
dramatically transform the clinical workflow. Tools capable of forecasting
disease progression offer invaluable insights for personalized patient care,
enabling timely interventions and better management of the disease’s impact on
patients’ lives.

Representation learning within deep learning has demonstrated remarkable
potential in extracting meaningful features from data, significantly enhancing
the performance of downstream tasks [5, 4, 15]. This approach has been
particularly transformative in medical imaging, where it has contributed to
breakthroughs in disease detection and diagnosis [12]. Self-supervised
learning (SSL) and its variants stand out as key strategies in learning these
useful representations without relying on labeled data [15, 7, 22]. When
labels are available, incorporating them into SSL frameworks can further
refine model performance by effectively clustering data points from the same
class closer together while distancing those from different classes [11].
Despite its success across domains, including medical image analysis [21, 9,
17], the application of SSL in analyzing disease progression, such as in
Alzheimer’s disease, remains under explored. Furthermore, there appears to be
a significant gap in the literature, as no existing studies have leveraged the
supervisory signals provided by available labels, such as event indicators, to
enhance the capabilities of self-supervised learning models specifically for
progression analysis in Alzheimer’s diseases. Our research aims to bridge this
gap by integrating these supervisory signals into our self-supervised learning
framework, thereby potentially improving downstream tasks in progression
analysis.

Toward this goal, we initially explored the potential of self-supervised
learning (SSL) in the progression analysis of Alzheimer’s disease. As
anticipated, the initial results showcased improved outcomes that further
encouraged us to incorporate supervisory signals to enhance representation
learning for AD progression analysis. We integrated the event label to better
guide the representation learning specifically tailored for progression
analysis. Recognizing the significance of time-to-event information, we
developed a novel self-supervised learning framework, Time and Event-aware SSL
(TE-SSL), which incorporates both event and time-to-event labels as additional
supervisory signals. Through evaluation on the ADNI dataset, TE-SSL
demonstrated an improvement in the downstream performance of time-to-event
prediction. Overall, our contributions are:

  1. 1.

The use of supervisory signals in the form of event occurrence for progression
analysis, offering a novel approach to understanding disease dynamics.

  2. 2.

A novel framework, TE-SSL, that uses time and event labels for SSL training.

  3. 3.

Demonstrated improved downstream performance for time-to-event prediction
across different metrics, showcasing the practical efficacy of our approach in
enhancing the predictive capabilities for Alzheimer’s disease progression.

##  2 Methods

We consider a set of labeled training examples $\mathcal{X}$ with the
corresponding labels $\mathcal{Y}$. Since, we are interested in modeling
disease progression, we consider a setup of survival analysis, where the
labels $\mathcal{Y}$ contains both time and event information. Specifically,
for each instance $i$, $\mathcal{Y}_{i}=(T_{i},\delta_{i})$, where $T_{i}$
denotes the time to event or censoring and $\delta_{i}$ is the event
indicator, with $\delta_{i}=1$ if the event (disease progression) occurred,
and $\delta_{i}=0$ if the data is censored. Here, censoring refers to
instances where the event has not yet occurred. Importantly, this does not
mean the event will never occur, only that it was not observed during the
study. The goal of survival analysis is to model the survival function
$S(t)=P(T>t)$, which estimates the probability of an event not occurring by
time $t$. Building upon this setup, we first learn appropriate representations
using our proposed self-supervised learning approach, TE-SSL (section 2.1),
which is a contrastive learning paradigm that utilizes the additional time
labels afforded by a survival analysis to strengthen the pull of elements at
similar stages of development to improve feature extraction. Fig. 1 provides
an illustration of our proposed SSL framework. We then finetune the network
with a task-specific objective function to predict survival outcomes at
individual time points (section 2.2).

Figure 1: Schematic diagram of the proposed time- and event-aware SSL, where
$\Delta_{*,*}$ represents the time difference between two data elements
$z_{*}$

###  2.1 TE-SSL: Time and Event-aware Self Supervised Learning

We provide a background of contrastive self-supervised learning, a learning
paradigm designed to use unlabeled data to learn representations by performing
some pretext task. We then provide details for enhancing the SSL learning
paradigm by introducing supervisory signals, before finally laying out our
proposed TE-SSL which leverage both event and time-to-event labels.

####  2.1.1 Self-supervised learning

Self-supervised learning (SSL) is an unsupervised method for learning
representations through pretext tasks, without labeled data. Formally, given
training examples $\mathcal{X}$, they are transformed into a modified version
$\tilde{\mathcal{X}}$ using a set of transformations $\mathcal{T}$ to create
multi-viewed examples. Transformations $\mathcal{T}$ include a set of
augmentation functions that do not alter the intrinsic information contained
within the data, thus creating a separate view of the original data. SSL
frameworks are trained with these multi-viewed batches. For each index $i\in
I\equiv{1,2,...,2N}$ in such a multi-viewed batch, let $j$ represent the
corresponding index of the transformed pair to sample $i$. In such a setup, a
loss function for the self-supervised learning objective can be represented
as:

$\mathcal{L}_{\text{SSL}}=-\sum_{i\in I}\log{\frac{\exp{(z_{i}\cdot
z_{j}/\tau)}}{\sum_{a\in A(i)}\exp{(z_{i}}\cdot z_{a}/\tau)}}$ (1)

where $z_{*}$ is the corresponding projection of the input $\mathbf{x}$,
$\tau$ is a temperature parameter, and $A(i)\equiv I-\\{i\\}$. Since augmented
images are semantically identical to one another, this method provides anchor
points for the model to "pull" together two augmented views while "pushing"
other samples in a batch.

####  2.1.2 Supervisory signal for SSL training

The standard SSL setup described above cannot maximize similarity between
samples of the same class (e.g., disease category or event vs. censored) in a
batch due to the absence of a supervisory signal. While it may seem that
standard SSL could eventually differentiate such attributes over time, recent
work has shown that incorporating supervisory signals significantly enhances
the standard SSL setup [11].

To incorporate supervisory signals using available label information (in our
case, event labels), the SSL objective (in Eqn. 1) can be generalized as
follows:

$\mathcal{L}_{\text{E-SSL}}=\sum_{i\in I}\frac{-1}{|P(i)|}\sum_{p\in
P(i)}\log{\frac{\exp{(z_{i}\cdot z_{p}/\tau)}}{\sum_{a\in
A(i)}\exp{(z_{i}\cdot z_{a})/\tau}}}$ (2)

where $P(i)\equiv\\{p\in A(i)|\tilde{y}_{p}=\tilde{y}_{i}\\}$ represents the
set of indices for all positive samples in the multi-viewed batch that are
distinct from $i$, and $|P(i)|$ is the cardinality of this set. If sample $i$
has an event indicator during the study period, this objective classifies all
samples with an event indicator as positive cases and those that are censored
as negative cases, and vice-versa. Clinically, this would translate to
maximizing feature representations of multiple patients’ imaging data (within
a batch) who eventually convert to AD.

####  2.1.3 Time-to-event and Event labels for SSL training

Progression analysis uniquely benefits from having both event indicators and
time-to-event information as labels, providing a comprehensive view of patient
outcomes. When considering disease progression, it is reasonable to assume
that the features of two patients at similar stages of progression will be
more similar than those from a late-stage and early-stage patient pair. We
therefore hypothesize that utilizing both types of labels in SSL training
enhances the learning process, leading to nuanced models that can accurately
predict the event timing and occurrence, thus significantly improving disease
progression analysis.

Towards this, we devise weighing schemes for each sample in the batch relative
to the anchor point, based on their time-to-event information. We calculate
the time difference between the anchor point and other samples in a batch,
using the maximum and minimum differences to assign weights to each pair.
These weights determine the strength of feature similarity, enforcing that
patients at similar stages in development will have a stronger pull than an
early/late stage patient pair. Our proposed SSL learning objective function
incorporating both time-to-event and labels is represented as:

$\mathcal{L}_{\text{TE-SSL}}=\sum_{i\in I}\frac{-1}{|P(i)|}\sum_{p\in
P(i)}\log{\frac{\omega_{i,p}\exp{(z_{i}\cdot z_{p}/\tau)}}{\sum_{a\in
A(i)}\omega_{i,a}\exp{(z_{i}\cdot z_{a})/\tau}}}$ (3)

where the weight term $\omega_{*,*}$ for each anchor point is calculated as:

$\omega_{i,j}=\frac{\alpha-\beta}{s-l}\Delta_{i,j}+\frac{\beta-\alpha}{s-l}s+\alpha$
(4)

where $\Delta_{i,j}=|T_{i}-T_{j}|$ is the time difference associated with data
points $i$ and $j$. Additionally, we compute $\Lambda=\\{\Delta_{i,j}|i,j\in
I,i\neq j\\}$ and take $s=\min{\Lambda}$ and $l=\max{\Lambda}$ to establish
the maximum and minimum time span differences between samples in the batch.
Finally, $\alpha$ and $\beta$ serve as hyperparameters defining the maximum
and minimum weight values, respectively. Specifically, the pair with the
smallest time difference is assigned the highest weight, $\alpha$, and the
pair with the largest time difference receives the lowest weight, $\beta$.

###  2.2 Time-to-event prediction

To leverage the feature space learned with SSL frameworks, we construct a deep
learning framework consisting of an encoder network $\mathcal{E}(\cdot)$ and a
projection head $\mathcal{P}(\cdot)$. During pretraining (Section 2.1),
multiviewed data (original and it’s augmented copy $\tilde{\mathcal{X}})$ is
passed through the encoder module to obtain
$r=\mathcal{E}(\tilde{\mathcal{X}})\in\mathbb{R}^{d_{E}}$, where $d_{E}$ is
the dimension of $r$. A final representation
$z=\mathcal{P}(r)\in\mathbb{R}^{d_{P}}$ ($d_{P}$ is the dimension $z$) is
computed and normalized for the pretraining procedure. After pretraining,
$\mathcal{P}(\cdot)$ is discarded and replaced with task-specific head, which
is then finetuned together with the encoder network on the time-to-event
objective.

For our task head, we adopt the DeepHit framework [13], a deep learning
approach to survival analysis. With DeepHit framework, instead of predicting a
single hazard coefficient for a given input, we output a distribution of
hazards at discrete time points. This allows the model to learn the first
hitting times (predicted time until the occurrence of the first event of
interest for each subject) directly without making assumptions about the
underlying form of the data. In specific, the model learns to minimize the
loss function $\mathcal{L}_{total}=\mathcal{L}_{1}+\mathcal{L}_{2}$, where
$\mathcal{L}_{1}$ is the log-likelihood of the distribution of the hitting
time, defined as

$\mathcal{L}_{1}=-\sum^{N}_{i=1}[\mathbbm{1}(\delta_{i}=1)*\log{h_{i}^{T_{i}}}+\mathbbm{1}(\delta_{i}\neq
1)*\log{(1-\hat{F}(T_{i}|x_{i})}]$ (5)

where, $\mathbbm{1}$ is an indicator function evaluating to 1 iff
$\delta_{i}=1$ (event occurred), $h_{i}^{T_{i}}$ corresponds to the predicted
hazard for input $X_{i}$ at time $T_{i}$, and $\hat{F}(T_{i}|x_{i})$ is the
estimated cumulative incidence function (CIF) which approximates the
probability that the event will occur on or before time $T_{i}$.
$\mathcal{L}_{2}$ incorporates a combination of cause-specific ranking loss
functions and is defined as:

$\mathcal{L}_{2}=\gamma\sum_{i\neq
j}A_{i,j}\cdot\exp(\hat{F}(T_{i}|x_{i}),\hat{F}(T_{i}|x_{j}))$ (6)

where $\gamma$ is a hyperparameter which indicates the intensity of the
ranking loss and $A_{i,j}=\mathbbm{1}(T_{i}<T_{j})$ represents an indicator
function which evaluates to $1$ if a pair $(i,j)$ experience an event at
different times.

##  3 Experiments and Results

####  3.0.1 ADNI dataset:

Our data consists of a cohort of 493 unique patients in the Alzheimer’s
Disease Neuroimaging Initiative (ADNI) [16] dataset. Each subject has one or
more visits containing a 3D T1-weighted MR Image, yielding a total of 2007
data points. Patients are diagnosed as being cognitively normal (CN), having
mild cognitive impairment (MCI), or Alzheimer’s dementia (AD) at every visit.
We define converters as subjects whom were CN or MCI during their initial
visit, but developed AD within the duration of the study. Additionally, each
visit contains the number of months since the baseline observation, which acts
as the time-to-event signal. The data were preprocessed via the pipeline laid
out by [14] and divided based on the unique participants to avoid data
leakage. For patients with multiple visits, we treat each visit as a unique
data point.

####  3.0.2 Implementation details:

We utilize a 3D CNN adapted from [14] as our backbone MRI encoder for both
pretraining and finetuning tasks. The encoder takes in
$X\in\mathbb{R}^{N\times 96\times 96\times 96}$ and outputs a representation
$\mathcal{E}(X)=r\in\mathbb{R}^{N\times 1024}$, where $N$ is the batch size.

Pretrain phase: Contrastive based SSL techniques require large batch sizes to
train properly. Due to hardware constraints, we selected $N=16$ and
accumulated gradients for 8 iterations before backpropagation to simulate a
batch size of 128. The encoded representation $r$ is then passed through the
projection head to achieve $z=\mathcal{P}(r)\in\mathbb{R}^{N\times 128}$. We
assigned a temperature of $\tau=0.07$ for all contrastive loss functions and
optimized the model using LARS with a learning rate of $.3\times N/256=.15$
[8] and a momentum of 0.9.

Finetune phase: We first discard the projection head of the pretrained model
and replace it with a task-specific $\mathcal{P}_{t}(\cdot)$ to predict the
probability mass function (PMF) associated with the input. We then apply the
Adam optimizer with a learning rate of $1e-4$ during training.

Evaluation: We compute the Time-dependent Concordance Index (C-td) [1], which
measures the extent to which the ordering of actual survival times of pairs
agrees with the ordering of their predicted risk. Additionally, we evaluate
the Integrated Brier Scores (IBS) [10], which measures an overall assessment
of the model’s performance across all available times considered in the study.
These metrics are calculated using the Pycox library.

Table 1: Comparison between different time-to-event prediction approach for C-td $\uparrow$ and IBS $\downarrow$ scores. Bold and underline denote 1st and 2nd best values, respectively.  Methods | C-td $\uparrow$ | IBS $\downarrow$
---|---|---
No pretraining | 0.7329 | 0.2099
SSL | 0.7511 | 0.1985
E-SSL | 0.7720 | 0.1997
TE-SSL | 0.7873 | 0.1889
Figure 2: t-SNE analysis of representations across different SSL frameworks.
Individual points, if not censored, are labeled with different time-to-event
groups.

###  3.1 Results

Our primary results in Table 1 showcase a comparison between our E-SSL and TE-
SSL frameworks and baseline models: No Pretraining and SSL. For fairness, all
models, including E-SSL and TE-SSL, were finetuned using the same model as No
Pretraining. We trained each model with three random seeds and reported their
average results. Our frameworks outperform others in both C-td (higher is
better) and IBS (lower is better) metrics. Notably, E-SSL introduces the novel
use of event labels in SSL training for progression analysis, while TE-SSL’s
innovative incorporation of both time-to-event and event information leads to
the best performance, highlighting its efficacy in progression analysis.

In Fig. 2, we present t-SNE plots [18] to examine the feature spaces learned
by the SSL frameworks considered in this study. The left panel displays the
feature space resulting from standard SSL training, which lacks the clear
separation between censored patient data and patient data with events observed
in the middle and right panels, where supervisory signals enhance SSL
training. Among the two SSL frameworks with supervisory signals, the
incorporation of both time-to-event and event information appears to achieve
superior separability.

###  3.2 Ablation analysis

We conducted the ablation analysis to better understand the roles of $\alpha$
and $\beta$ in our proposed TE-SSL (Eqn. 4), which serve to define the
intensity of the weight values for a pair of inputs $(i,j)$. The difference
between $\alpha$ and $\beta$ dictates how strongly to differentiate distant
pairs. For instance, in a batch of $N$ samples if $(i,j)$ represents the pair
with the largest time difference, setting $\beta=0$, effectively considers $j$
as a negative sample relative to the anchor $i$. Therefore, we explored
sensible configurations of $\alpha$ and $\beta$, with $1\leq\alpha\leq 1.5$
and $0.5\leq\beta\leq 1$, noting that an $\alpha-\beta<0.5$ would
inappropriately diminish the negative impact of distant pairs. The results,
presented in Table 2, demonstrate the method’s relative stability within these
selected ranges. It is also noteworthy that TE-SSL outperforms both standard
SSL and the baseline no-pretraining time-to-event prediction model in five out
of six experiments, highlighting its efficacy.

Table 2: Model performance on different values of $\alpha$ and $\beta$ for TE-SSL pretraining.  $\alpha$ | $\beta$ | C-td $\uparrow$ | IBS $\downarrow$
---|---|---|---
1 | 0.5 | 0.7636 | 0.1954
1 | 0.7 | 0.7581 | 0.1954
1 | 0.9 | 0.7883 | 0.1889
1.1 | 1 | 0.7714 | 0.1981
1.3 | 1 | 0.761 | 0.1931
1.5 | 1 | 0.733 | 0.2027

##  4 Conclusion

We introduce the Time and Event-aware SSL framework, which integrates both
event and time-to-event information to guide the learning process of feature
representations. As demonstrated, our approach surpasses existing self-
supervised learning methods, including those supervised versions that
incorporate only the event label. This underscores the critical importance of
utilizing both event and time-to-event information in the progression analysis
of Alzheimer’s disease. Our evaluation using the ADNI dataset showcases the
practical applicability and effectiveness of our proposed method,
significantly contributing to the advancement of AD progression study.

####  4.0.1 Acknowledgments.

This research was supported by West Virginia Higher Education Policy
Commission’s Research Challenge Grant Program 2023 and DARPA/FIU AI-CRAFT
grant. Data collection and sharing for the Alzheimer’s Disease Neuroimaging
Initiative (ADNI) is funded by the National Institute on Aging (National
Institutes of Health Grant U19 AG024904). The grantee organization is the
Northern California Institute for Research and Education. In the past, ADNI
has also received funding from the National Institute of Biomedical Imaging
and Bioengineering, the Canadian Institutes of Health Research, and private
sector contributions through the Foundation for the National Institutes of
Health (FNIH) including generous contributions from the following: AbbVie,
Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon
Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir,
Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company;
EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech,
Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy
Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research &
Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale
Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis
Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda
Pharmaceutical Company; and Transition Therapeutics.

####  4.0.2 Disclosure of interests.

The authors have no competing interests to report.

## References

  * [1] Laura Antolini, Patrizia Boracchi, and Elia Biganzoli.  A time-dependent discrimination index for survival data.  Statistics in medicine, 24(24):3927–3944, 2005.
  * [2] Akhilesh Deep Arya, Sourabh Singh Verma, Prasun Chakarabarti, Tulika Chakrabarti, Ahmed A Elngar, Ali-Mohammad Kamali, and Mohammad Nami.  A systematic review on machine learning and deep learning techniques in the effective diagnosis of alzheimer’s disease.  Brain Informatics, 10(1):17, 2023.
  * [3] No authors listed.  2023 alzheimer’s disease facts and figures.  ALZHEIMER’S ASSOCIATION REPORT, 2023.
  * [4] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  Advances in neural information processing systems, 32, 2019.
  * [5] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [6] Allen J Chang, Rebecca Roth, Eleni Bougioukli, Theodor Ruber, Simon S Keller, Daniel L Drane, Robert E Gross, James Welsh, Anees Abrol, Vince Calhoun, et al.  Mri-based deep learning can discriminate between temporal lobe epilepsy, alzheimer’s disease, and healthy controls.  Communications Medicine, 3(1):33, 2023.
  * [7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  2 2020.
  * [9] Nanqing Dong, Michael Kampffmeyer, and Irina Voiculescu.  Self-supervised multi-task representation learning for sequential medical images.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 779–794. Springer, 2021.
  * [10] Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher.  Assessment and comparison of prognostic classification schemes for survival data.  Statistics in medicine, 18(17-18):2529–2545, 1999.
  * [11] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  Advances in neural information processing systems, 33:18661–18673, 2020.
  * [12] Rayan Krishnan, Pranav Rajpurkar, and Eric J Topol.  Self-supervised learning in medicine and healthcare.  Nature Biomedical Engineering, 6(12):1346–1352, 2022.
  * [13] Changhee Lee, William Zame, Jinsung Yoon, and Mihaela Van Der Schaar.  Deephit: A deep learning approach to survival analysis with competing risks.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [14] Sheng Liu, Chhavi Yadav, Carlos Fernandez-Granda, Narges Razavian, Adrian V Dalca, Matthew Mcdermott, Emily Alsentzer, Sam Finlayson, Michael Oberst, Fabian Falck, and Brett Beaulieu-Jones.  On the design of convolutional neural networks for automatic detection of alzheimer’s disease, 2020.
  * [15] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6707–6717, 2020.
  * [16] R. C. Petersen, P. S. Aisen, L. A. Beckett, M. C. Donohue, A. C. Gamst, D. J. Harvey, C. R. Jack, W. J. Jagust, L. M. Shaw, A. W. Toga, and et al.  Alzheimer’s disease neuroimaging initiative (adni).  Neurology, 74(3):201–209, Jan 2010.
  * [17] Luca Rettenberger, Marcel Schilling, Stefan Elser, Moritz Böhland, and Markus Reischl.  Self-supervised learning for annotation efficient biomedical image segmentation.  IEEE Transactions on Biomedical Engineering, 2023.
  * [18] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [19] Samuel L Warren and Ahmed A Moustafa.  Functional magnetic resonance imaging, deep learning, and alzheimer’s disease: A systematic review.  Journal of Neuroimaging, 33(1):5–18, 2023.
  * [20] Junhao Wen, Elina Thibeau-Sutre, Mauricio Diaz-Melo, Jorge Samper-González, Alexandre Routier, Simona Bottani, Didier Dormont, Stanley Durrleman, Ninon Burgos, Olivier Colliot, et al.  Convolutional neural networks for classification of alzheimer’s disease: Overview and reproducible evaluation.  Medical image analysis, 63:101694, 2020.
  * [21] Pengshuai Yang, Zhiwei Hong, Xiaoxu Yin, Chengzhan Zhu, and Rui Jiang.  Self-supervised visual representation learning for histopathological images.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24, pages 47–57. Springer, 2021.
  * [22] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.


# A Framework to Map VMAF with
the Probability of Just Noticeable Difference between Video Encoding Recipes

Jingwen Zhu1§, Suiyi Ling1§, Yoann Baveye1, Patrick Le Callet1  1Nantes
Université, Ecole Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes,
France

###### Abstract

Just Noticeable Difference (JND) model developed based on Human Vision System
(HVS) through subjective studies is valuable for many multimedia use cases. In
the streaming industries, it is commonly applied to reach a good balance
between compression efficiency and perceptual quality when selecting video
encoding recipes. Nevertheless, recent state-of-the-art deep learning based
JND prediction model relies on large-scale JND ground truth that is expensive
and time consuming to collect. Most of the existing JND datasets contain
limited number of contents and are limited to a certain codec (e.g., H264). As
a result, JND prediction models that were trained on such datasets are
normally not agnostic to the codecs. To this end, in order to decouple
encoding recipes and JND estimation, we propose a novel framework to map the
difference of objective Video Quality Assessment (VQA) scores, i.e.,  VMAF,
between two given videos encoded with different encoding recipes from the same
content to the probability of having just noticeable difference between them.
The proposed probability mapping model learns from DCR test data, which is
significantly cheaper compared to standard JND subjective test. As we utilize
objective VQA metric (e.g., VMAF that trained with contents encoded with
different codecs) as proxy to estimate JND, our model is agnostic to codecs
and computationally efficient. Throughout extensive experiments, it is
demonstrated that the proposed model is able to estimate JND values
efficiently.

§§footnotetext: Equal contribution

##  I Introduction

It is well studied in the cognitive community through psychophysical
experiments that our Human Visual System has a limited resolution. Just
Noticeable Difference, defined as the minimum amount by which stimulus
intensity must be adjusted in order to produce a noticeable difference, is
commonly applied to represent this limitation.  JND model developed based on
HVS study has became the model of choice in many multimedia applications,
especially for streaming industries, as it servers as a threshold for video
compression platform. With the JND threshold, one can achieve high coding
efficiency by eliminating perceptual redundancy without sacrificing the
perceived quality to ensure best user experience.

Subjective study: To develop robust JND model, subjective studies need to be
conducted to collect reliable JND data for the target usecase. Recently,
several subjective studies have been conducted. Some JND datasets [1, 2, 3, 4]
were released publicly for Picture Wise JND (PW-JND), and a few for Video Wise
JND (VW-JND) [5, 6]. However, most of the existing subjective JND studies
applied the ‘one-direction’ JND search protocol, i.e., by only
decreasing/increasing the perceived quality of the compressed stimuli.

Objective JND models: Since it is time consuming and costly to conduct
subjective tests, especially for video contents, objective JND models that
predict the JND values automatically are more desirable. One of the most
common ways to predict JND is to first predict the Satisfied User Ratio (SUR)
[7, 8, 9, 10]. For a given image/video content, the individual JND of
different subjects are different. SUR is the Complementary Cumulative
Distribution Function (CCDF) of the group-based JND annotations collected from
subjective test [6]. In the literature, the proxy of JND is usually the
encoding parameters (e.g., QF, QP or the CRF values that decide the encoding
setting). Explicitly, given a certain encoding parameter $P$, SUR is defined
as the percentage of subjects who are satisfied with the the compressed
content. In another word, SUR is the ratio of subjects that cannot perceive
any difference between the reference/anchor and the content encoded with $P$.
JND can be derived from the SUR curve giving a certain threshold, and the most
commonly used JND threshold is $75\%$. Earlier, image JND/SUR models were
developed for streaming contents. Liu et al. [11] has proposed a deep learning
based PW-JND model by formulating the task as a multi-category classification
problem; Later, using also CNN, another PW-JND algorithm was presented in
[12]; To overcome the challenge of training a deep PW-JND model with limited
data, an effective JND model was proposed by Shen [4] using patch-level
structural visibility learning; In [10], a Siamese network was utilized to
predict the SUR curve, which was later improved in [9]. Apart from the JND
models dedicated for images, there are also some models were developed for
videos. Zhang et al. modeled the SUR curve via Gaussian Processes Regression
[7, 13]; A new discrete cosine transform-based energy-reduced JND was
demonstrated in [14]. Similar to image SUR model, a video SUR model was
proposed by Zhang et al. in [15]; Using deep learning, Zhang et al. developed
the Video Wise Spatial SUR method (VW-SSUR) for predicting the SUR value for
compressed video along with the Video Wise Spatial-Temporal SUR (VW-STSUR) to
boost the prediction accuracy. Nevertheless, most of the aforementioned SUR
prediction methods (or JND approaches based on SUR estimation) suffer from 2
drawbacks: (1) the subjective JND datasets is indispensable. Specifically,
large-scale JND datasets are necessary for deep learning-based method for SUR
prediction [10, 9, 8]. (2) these SUR/JND models are not codec agnostic. For
instance, the SUR prediction model proposed by Wang et al. [7] that predicts
the QP value based on H.264 encoding datasets [6] cannot be directly applied
for other codec.

To address these issues in existing works, rather than using encoding
parameter as the JND proxy, which leads to codec dependence, we propose to
exploit objective quality metric, (e.g., VMAF [16]) to estimate the JND.
Intuitively, if an objective quality metric correlates well with human’s
opinions and is able to distinguish non-ambiguous pair confidently, it could
be utilized to tell to which point the difference of a pair of stimuli becomes
noticeable. Therefore, given a content as anchor with a certain perceived
quality, the goal is to estimate the JND by calculating the difference of the
objective quality scores between the anchor and another candidate with
worse/better quality. With ground truth JND data, we can learn to which extend
by increasing/decreasing the objective quality scores, we will notice the
difference. Then, we can use this learned objective scores’ difference (e.g.,
difference in VMAF) as the new proxy for predicting the JND. The anchor is not
limited to the Source sequence (SRC), it can also be a Processed Video
Sequence (PVS) obtained with different encoding recipes. By increasing the
quality of the anchor to a certain point (i.e., the $1_{st}$ JND obtained via
increasing the perceived quality), observer may be able to notice the quality
got improved. Vice versa, we can also get the JND through by decreasing the
quality. In practical application, given the anchor, the proposed model
outputs the difference of objective quality score $\Delta obj$ indicating to
which point the change of quality becomes noticeable. One can then estimate
objective quality score of the increased/decreased JND point $obj_{inc-JND}$
or $obj_{dec-JND}$, by simply adding/subtracting the $\Delta obj$ to/from the
objective quality of the anchor $obj_{anchor}$. With the predicted $obj_{inc-
JND}$ or $obj_{dec-JND}$, one can then map/interpolate the optimized codec
configuration that generates the PVS with an objective quality score that
equals (or close) to $obj_{inc-JND}$ or $obj_{dec-JND}$.

The contribution of this study is two-fold: (1) we collected a JND dataset
with a novel two-direction (with both increased and decreased quality) JND
search subjective protocol, where each stimulus is also annotated with
subjective DCR rating; (2) we presented a framework to map the
difference/residual of an objective quality metric (e.g., VMAF) between the
anchor and a PVS with another encoding recipe by estimating the probability of
the PVS being considered with noticeable quality difference compared to the
anchor with a mapping function. It has to be highlighted that the proposed
framework takes subjective DCR test annotations as input but not the
subjective JND data, as JND subjective test is more resources and time
consuming. Interestingly, it is showcased that the calculated difference of
objective scores for JND estimation depends on the quality range, where the
objective score of the anchor content $obj_{anchor}$ falls in.

##  II Methodologies

###  II-A Subjective studies

####  II-A1 Two-directions JND search

The objective of the two-direction (2-d) JND search is to provide the JND
ground truth for validating the proposed JND mapping function. As shown in
Fig.1, the proposed 2-d JND search not only the JND with worse quality (dec-
JND) compared to the anchor (JND-dec-anchor) via decreasing the perceived
quality, but also the JND with better quality (inc-JND) compared to an anchor
(JND-inc-anchor) by increasing the perceived quality. In our study, given a
content, JND-dec-anchor is defined as the PVS/SRC with best quality, and JND-
inc-anchor is the PVS that with the worse perceived quality (points from the
same Rate-Quality (R-Q) curve). For each direction JND search, the bisection
search process was applied, readers could refer to [6] for more details. By
considering both directions, we can cover more streaming usecases, where JND-
dec-anchor and/or JND-inc-anchor are/is required.

Figure 1: Illustration of the proposed 2-D JND search

20 HD SRCs and 10 UHD SRCs were selected from 229 pristine videos from Amazon
Prime Video streaming platform based on the content selection strategy
proposed in [17] such that the selected contents are with wide-range of
different characteristics and ambiguities in terms of quality. Each SRC was
encoded with 39 $(3\times 13)$ recipes (3 encoding resolution and each
resolution with 13 levels of distortion) using high efficiency video coding
(HEVC). The subjective test of 2-d JND search was conducted in a controlled
lab environment. A 55-inch calibrated ‘UHD Grundig Finearts 55 FLX 9492 SL’
was employed as the display screen. The viewing distance was set as 1.5$H$ for
UHD and 3$H$ for HD contents as recommended in ITU-R BT.2022 [18], where $H$
is the height of the screened video. Five experts, namely the ‘golden eyes’,
participated in the 2-d JND search. They are familiar with encoding artifacts
and thus the collected JND data is more accurate. Furthermore, less resources
(time and observers) were required. The 2-d JND searches were conducted for
each resolution respectively, (including 1080p, 720p and 540p for HD). It is
worth noting that only the JND-dec-anchors for 1080p of HD contents is the
SRC, and the JND-dec-anchors for 720p and 540p of HD contents are the PVS with
best perceived quality. Similar setting was also applied for the UHD set. For
each resolution, the two anchors (i.e., both the anchor for increased and
decreased quality) and their corresponding JNDs were selected, which ended up
to total 12 $(4\times 3)$ videos that are with significantly different
qualities for each content (11 PVS + 1 SRC).

####  II-A2 DCR based on 2-d JND search

The proposed JND mapping model is based on estimating the probability that one
stimulus is significant different/similar to another stimulus or not. Thus,
subjective quality experiments are also required. To this end, we conducted a
subjective quality study utilizing DCR protocol according to ITU-T P.913 [19]
with the same contents selected in the 2-d JND searching test. During the
test, for each observer, the SRC of each content was compared to 11
corresponding PVSs encoded with different encoding recipes. In total 24
observers participated in the test, and all of them passed the pre-experiment
vision check [19] to ensure that they have correct visual acuity.

###  II-B JND mapping framework

Figure 2: Diagram of the proposed framework of the JND mapping function using
DCR datasets

The JND prediction model is depicted in Fig.2 (5 steps): Step 1 - Significant
classification: For a given content ${C_{k}}$, $N$ JNDs (videos encoded from
${C_{k}}$ with different encoding recipes $R_{x}$) were selected via the
proposed 2-D JND search. From the $N$ selected videos (i.e.,
increased/decreased JNDs), we formed pairs that took any two videos among the
$N$ videos. In the end, there were in total $N$:$\frac{{N!}}{{d!(N-d)!}}$
pairs, where $d=2$, $N=12$.  For each content, all the formed pairs were
classified into two classes: (1) significantly different pairs; (2) similar
pairs. This classification was proceeded through a significant test, e.g.,
t-test, by using the individual subjective scores (opinion scores from all the
observers $obs_{1},...,obs_{n}$) from the DCR subjective test described in
Section II-A2. Pairs with significant difference were labeled with $sig=1$,
otherwise $sig=0$.

Step 2 - Objective quality assessment: The quality of each video was evaluated
by an certain objective quality assessment model $obj(\cdot)$ (e.g., $obj=$
VMAF [16]). Afterwards, the residuals/differences of objective quality scores
for each pair were calculated: $\Delta
obj(k,x,y)=\left|{obj({C_{k}}{R_{x}})-obj({C_{k}}{R_{y}})}\right|$.

Step 3 - Sub-quality range decomposition Before constructing the mapping
function, video pairs
${C_{k}}\left\\{{{R_{x}},}\right.\left.{{R_{y}}}\right\\}$ were grouped based
on which sub-quality range the objective quality score of the anchor falls in.
This grouping procedure that divides the entire quality range into a set of
continuous sub-ranges is named as sub-quality range decomposition. The
motivation behind is that the perceptual distances between anchors and JND
points differ along the quality range. There are many possible ways to split
the entire quality range. One of the most straightforward ways is to divide
the whole quality range into equal units, where the size of each bin is 5 in
terms of VMAF score. Nevertheless, in this pilot test, such bins division
strategy may end out into disconnected distribution (where some bins are
empty) or distribution that contains only significantly different/similar
pairs due to the limited numbers of tested videos. Therefore, in this study,
the sub-ranges were split to have same amount of videos (balanced bins).

Step 4 - Mapping function design: For each sub-quality range, the
significantly different pairs that belong to this sub-range, were further
represented by a histogram based on the $\Delta obj$ of the pair, i.e., the
objective score differences of the pairs: $h_{dif}(\Delta
obj)=\\{f_{1}^{dif},\cdots,f_{b}^{dif},\cdots,f_{B}^{dif}\\}$ , where
$f_{b}^{dif}$ is one bin that accumulates the frequency of pairs with
objective quality residual equal to $b$, i.e., $\Delta obj(k,x,y)=b$:

$f_{b}^{dif}=\sum\limits_{k}{\sum\limits_{\\{x,y\\}}{\mathbf{1}}}(\Delta
obj(k,x,y)=b\;\&\;sig(k,x,y)=1),$ (1)

where $\textbf{1}(c)$ is an indicator function that equals to 1 if the
specified binary clause is true; and video pairs $\\{x,y\\}$ belong to the
current objective quality range. Similarly, the distribution of similar pairs
$h_{sim}(\Delta obj)$ could be obtained with:

$f_{b}^{sim}=\sum\limits_{k}{\sum\limits_{\\{x,y\\}}{\mathbf{1}}}(\Delta
obj(k,x,y)=b\;\&\;sig(k,x,y)=0).$ (2)

Afterwards, for each sub-quality range, we can get the two histograms, i.e.,
$h_{dif}(\Delta obj)$ and $h_{sim}(\Delta obj)$, with the same number and
interval of bins of $\Delta obj$, namely, the co-distribution. In this study,
we utilized VMAF, i.e., $obj$ = VMAF. As depicted in Fig.2, the orange bar is
the $h_{sim}(\Delta obj)$ and blue bar is the $h_{dif}(\Delta obj)$. It can be
observed that similar pairs are distributed in regions where the $\Delta$VMAF
is relatively small, and significantly different pairs are distributed in
regions with larger $\Delta$VMAF. For two different encoding recipes $R_{x}$
and $R_{y}$ of the content $C_{k}$, supposing that
$\Delta\textrm{VMAF}(k,x,y)=b$, the probability that these two videos are
Significantly Different (SD) is calculated as:

${P_{SD}}(b)=\frac{{f_{b}^{dif}}}{{f_{b}^{dif}+f_{b}^{sim}}},\vspace{-1.5mm}$
(3)

where $f_{b}^{dif}$ and $f_{b}^{sim}$ were obtained from the co-distribution.

(a) Co-distribution

(b) Original points of MF

(c) Fitted MF

Figure 3: How to obtain the mapping function (MF)

Fig.3 illustrates how to obtain the mapping function from the co-distribution:
for each bin in the co-distribution (Fig.3a), the value of $P_{SD}$ was
obtained through Eq.(3). These $P_{SD}$ along with its corresponding
$\Delta$VMAF are the original points of the Mapping Function (Fig.3b). The
final Mapping Function was then obtained by fitting these original points
(Fig.3c).

Step 5 - From mapping function to JND estimation: After getting the mapping
functions for different quality ranges, we can predict the JND of a given
anchor with the corresponding estimated $\Delta obj$ ( $obj$ = VMAF) by fixing
a threshold. As illustrated in Fig.3c, if the blue curve is the mapping
function of an anchor video $V_{a}$, and the threshold is set to $thr=75\%$,
the corresponding value of $\Delta$VMAF $d$ is then considered JND of $V_{a}$.
In other words, for a PVS $V_{t}$, which was encoded from the same SRC as
$V_{a}$, but encoded with a different recipe, if
$\left|{\textrm{VMAF}({V_{a}})-\textrm{VMAF}({V_{t}})}\right|>b$, we assumed
that observers can perceive difference between them.

##  III Experiment and Results

The proposed JND estimation model was tested on both the JND datasets
collected in Section II-A1 and the publicly available JND VideoSet [6].
According to our best knowledge, we are the first to proposed such codec-
independent JND estimation model using subjective quality data and objective
quality metrics. Thus, as a preliminary study, we did not compared our model
with any existing JND models. Before using the collected DCR annotations for
the significant classification, three methods including (1) VQEG HDTV Annex I
[20], (2) ITU-R BT.1788 [21] and (3) ITU-R BT.500-12 [22] were used to
identify and remove outliers(5 for HD and 3 for UHD). With significant t-test,
we obtained 1075 significantly different pairs for HD and 563 significantly
different pairs for UHD set. We processed the collected HD and UHD datasets
separately, due to page limitation, in this study, we only present the results
of HD for demonstration. After unequal quality range decomposition (to ensure
same number of pairs within each bin), the decomposed sub-quality ranges for
HD include $(30,79]$, $(79,86]$, $(86,90]$, $(90,95]$ and $(95,100]$. For
video pair $\\{x,y\\}$, it was classified to the sub-quality range $A$, if
$\textrm{VMAF}(x)\in A\vee\textrm{VMAF}(y)\in A$.

In our experiments, four fitting functions were considered, with the
constraint that the function is monotonic on the full interval: (1) The
5-parameter logistic curve (5-para) [23]; (2) The 4-parameter cubic polynomial
(4-para) [23]; (3) The 2-parameter logistic curve (2-para) [23]; (4) The
Generalized Linear Model fitting (GLM) (commonly used in psycho-physical
studies [24, 25]). To evaluate the performance of JND prediction, two commonly
used regression error metrics, Mean Average Error (MAE) and Root Mean Square
Error (RMSE) between the ground truth $y$ and the predicted label $\hat{y}$,
were calculated.

The results on our JND dataset (Section II-A1) are summarized in Table I for
increasing JND and Table II for decreasing JND. It can be observed that by
applying the GLM fitting function and a threshold equals to 0.95, our model
achieves best performance in predicting both dec-JND and inc-JND. To validate
the generality of the proposed JND estimation model, the performance of JND
prediction was also evaluated on the large-scale JND VideoSet [6]. It contains
220 5-second SRCs in 4 resolutions. Each SRC was encoded with H.264 codec with
QP values from 1 to 51. The dataset was labeled with the 1st, 2nd and 3rd JND.
The results are summarized in Table III. It can be observed that the best
performances are obtained when the threshold is 0.85 for all the three JNDs.
Furthermore, the prediction errors using the proposed model on the VideoSet
are slightly smaller than ones on our 2-d JND dataset, which further verifies
the fact that the proposed framework is generic and codec agnostic.

TABLE I: Performance of JND prediction ($\Delta$VMAF) for increasing JND for
HD contents considering different thresholds $thr.$ and fitting functions
$fit(\cdot)$

$thr.$ $fit(\cdot)$ 5-para 4-para 2-para GLM Mean Absolute Error (MAE)[1, 100]
0.75 5.1519 4.1997 5.1281 4.8915 0.8 4.1956 3.9765 4.7420 4.4463 0.85 3.6163
3.8181 4.3771 3.9757 0.9 4.2271 3.7447 4.0420 3.6114 0.95 6.7110 3.8777 3.9551
3.4963 Root Mean Square Error (RMSE) [1, 100] 0.75 7.2220 5.9346 7.0309 6.7227
0.8 5.9573 5.7270 6.6600 6.2290 0.85 5.1240 5.5412 6.2459 5.7050 0.9 5.3592
5.3984 5.8011 5.1732 0.95 7.4985 5.4512 5.4493 4.8471

TABLE II: Performance of JND prediction ($\Delta$VMAF) for decreasing JND for
HD contents considering different thresholds $thr.$ and fitting functions
$fit(\cdot)$

$thr$ $fit(\cdot)$ 5-para 4-para 2-para GLM Mean Absolute Error (MAE)[1, 100]
0.75 4.2586 3.4865 4.1996 4.0253 0.8 3.6958 3.3277 3.8736 3.6564 0.85 3.8694
3.2493 3.6509 3.3475 0.9 5.1067 3.3602 3.6679 3.2665 0.95 7.6473 3.6988 4.2052
3.6863 Root Mean Square Error (RMSE) [1, 100] 0.75 7.5868 6.2914 7.2717 7.0435
0.8 6.6783 6.1947 7.0217 6.6774 0.85 6.2428 6.1316 6.7698 6.3115 0.9 6.1184
6.1184 6.5555 5.9811 0.95 6.2585 6.2585 6.5525 5.8850

TABLE III: Performance of JND prediction for the first, second, and third JND
in VideoSet (1080p) considering different thresholds $thr.$ with GLM fitting
functions.

$N\textsuperscript{th}$ $1\textsuperscript{st}$ JND $2\textsuperscript{nd}$
JND $3\textsuperscript{rd}$ JND $thr.$ MAE RMSE MAE RMSE MAE RMSE 0.75 3.5026
4.6771 3.5192 4.5879 2.7160 3.6242 0.8 3.1819 4.3186 3.3584 4.3428 2.5840
3.4218 0.85 3.0491 4.1053 3.3242 4.1961 2.5691 3.3247 0.9 3.2415 4.2470 3.5338
4.2794 2.7681 3.4638 0.95 3.8400 4.7528 3.5192 4.5879 3.6060 4.5246

##  IV Conclusion

In this work, a framework that maps the $\Delta$VMAF values to the probability
of the JND between videos encoded with different encoding recipes is proposed.
Given an anchor video as input, the proposed model takes into account which
sub-quality-range the anchor video falls in, and output the estimated JND in
terms of $\Delta$VMAF. With the $\Delta$VMAF, one can then select the right
encoding recipes.  The current JND prediction model could be improved by
taking the content characteristics into consideration, e.g., using different
mapping functions for different types of contents instead of based on quality
range.

## References

  * [1] L. Jin, J. Y. Lin, S. Hu, H. Wang, P. Wang, I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, “Statistical study on perceived jpeg image quality via mcl-jci dataset construction and analysis,” _Electronic Imaging_ , vol. 2016, no. 13, pp. 1–9, 2016.
  * [2] X. Liu, Z. Chen, X. Wang, J. Jiang, and S. Kowng, “Jnd-pano: Database for just noticeable difference of jpeg compressed panoramic images,” in _Pacific Rim Conference on Multimedia_. Springer, 2018, pp. 458–468.
  * [3] C. Fan, Y. Zhang, H. Zhang, R. Hamzaoui, and Q. Jiang, “Picture-level just noticeable difference for symmetrically and asymmetrically compressed stereoscopic images: Subjective quality assessment study and datasets,” _Journal of Visual Communication and Image Representation_ , vol. 62, pp. 140–151, 2019.
  * [4] X. Shen, Z. Ni, W. Yang, X. Zhang, S. Wang, and S. Kwong, “Just noticeable distortion profile inference: A patch-level structural visibility learning approach,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 26–38, 2020.
  * [5] H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, “Mcl-jcv: a jnd-based h. 264/avc video quality assessment dataset,” in _2016 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2016, pp. 1509–1513.
  * [6] H. Wang, I. Katsavounidis, J. Zhou, J. Park, S. Lei, X. Zhou, M.-O. Pun, X. Jin, R. Wang, X. Wang _et al._ , “Videoset: A large-scale compressed video quality dataset based on jnd measurement,” _Journal of Visual Communication and Image Representation_ , vol. 46, pp. 292–302, 2017.
  * [7] H. Wang, I. Katsavounidis, Q. Huang, X. Zhou, and C.-C. J. Kuo, “Prediction of satisfied user ratio for compressed video,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 6747–6751.
  * [8] Y. Zhang, H. Liu, Y. Yang, X. Fan, S. Kwong, and C. J. Kuo, “Deep learning based just noticeable difference and perceptual quality prediction models for compressed video,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2021.
  * [9] H. Lin, V. Hosu, C. Fan, Y. Zhang, Y. Mu, R. Hamzaoui, and D. Saupe, “Sur-featnet: Predicting the satisfied user ratio curve for image compression with deep feature learning,” _Quality and User Experience_ , vol. 5, no. 1, pp. 1–23, 2020.
  * [10] C. Fan, H. Lin, V. Hosu, Y. Zhang, Q. Jiang, R. Hamzaoui, and D. Saupe, “Sur-net: Predicting the satisfied user ratio curve for image compression with deep learning,” in _2019 eleventh international conference on quality of multimedia experience (QoMEX)_. IEEE, 2019, pp. 1–6.
  * [11] H. Liu, Y. Zhang, H. Zhang, C. Fan, S. Kwong, C.-C. J. Kuo, and X. Fan, “Deep learning-based picture-wise just noticeable distortion prediction model for image compression,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 641–656, 2019.
  * [12] T. Tian, H. Wang, L. Zuo, C.-C. J. Kuo, and S. Kwong, “Just noticeable difference level prediction for perceptual image compression,” _IEEE Transactions on Broadcasting_ , vol. 66, no. 3, pp. 690–700, 2020.
  * [13] H. Wang, X. Zhang, C. Yang, and C.-C. J. Kuo, “Analysis and prediction of jnd-based video quality model,” in _2018 Picture Coding Symposium (PCS)_. IEEE, 2018, pp. 278–282.
  * [14] S. Ki, S.-H. Bae, M. Kim, and H. Ko, “Learning-based just-noticeable-quantization-distortion modeling for perceptual video coding,” _IEEE Transactions on Image Processing_ , vol. 27, no. 7, pp. 3178–3193, 2018.
  * [15] X. Zhang, C. Yang, H. Wang, W. Xu, and C.-C. J. Kuo, “Satisfied-user-ratio modeling for compressed video,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 3777–3789, 2020.
  * [16] Z. Li, A. Aaron, I. Katsavounidis, A. Moorthy, and M. Manohara, “Toward a practical perceptual video quality metric,” _The Netflix Tech Blog_ , vol. 6, no. 2, 2016.
  * [17] S. Ling, Y. Baveye, D. Nandakumar, S. Sethuraman, and P. Le Callet, “Towards better quality assessment of high-quality videos,” in _Proceedings of the 1st Workshop on Quality of Experience (QoE) in Visual Multimedia Applications_ , 2020, pp. 3–9.
  * [18] I. BT, “General viewing conditions for subjective assessment of quality of sdtv and hdtv television pictures on flat panel displays,” _International Telecommunication Union_ , 2012.
  * [19] I. Union, “Methods for the subjective assessment of video quality audio quality and audiovisual quality of internet video and distribution quality television in any environment,” _SERIES P: TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS_ , 2016.
  * [20] VQEG, “Vqeg hdtv group : Test plan for evaluation of video quality models for use with high definition tv content (draft version 3.1),” _https://vqeg.github.io/software-tools/subjective%20test%20software/matlab-screening/_.
  * [21] P. ITU-T RECOMMENDATION, “Subjective video quality assessment methods for multimedia applications,” 1999.
  * [22] R. I.-R. BT, “Methodology for the subjective assessment of the quality of television pictures,” _International Telecommunication Union_ , 2002.
  * [23] I. Tutorial, “Objective perceptual assessment of video quality: Full reference television,” _ITU-T Telecommunication Standardization Bureau_ , 2004.
  * [24] F. A. Wichmann and N. J. Hill, “The psychometric function: I. fitting, sampling, and goodness of fit,” _Perception & psychophysics_, vol. 63, no. 8, pp. 1293–1313, 2001.
  * [25] Y. Lee, J. A. Nelder, and Y. Pawitan, _Generalized linear models with random effects: unified analysis via H-likelihood_. Chapman and Hall/CRC, 2018.


# Production of $T^{+}_{cc}$ in heavy ion collisions

L. M. Abreu  Instituto de Física, Universidade Federal da Bahia,
Campus Universitário de Ondina, 40170-115, Bahia, Brazil
<EMAIL_ADDRESS>F. S. Navarra  Instituto de Física, Universidade de São
Paulo, Rua do Matão, 1371
CEP 05508-090, São Paulo, SP, Brazil<EMAIL_ADDRESS>M. Nielsen
<EMAIL_ADDRESS>H. P. L. Vieira<EMAIL_ADDRESS>
###### Abstract

We study the production of heavy multiquark states in the heavy ion collisions
performed at the LHC. We assume that they are produced at the end of the
quark-gluon plasma phase and then interact with light hadrons during the
hadron gas phase. We use the coalescence model to compute the initial
multiplicities and effective Lagrangians to describe the interactions. We find
that the initial multiplicity of molecules is two orders of magnitude larger
than that of tetraquarks. The interactions in the hadron gas reduce the number
of molecules and increase the number of tetraquarks by a factor two in each
case. At kinetic freeze-out, the difference is still very large.

###### keywords:

Multiquark states, QCD sum rules, Effective Lagrangians, Heavy ion collisions

##  1 Introduction

The number of exotic hadrons keeps continuously growing. The most recently
observed exotic states were announced during this conference [1] 111The
$J/\psi-\Lambda$ bound state was observed in the $B\to J/\psi\Lambda\bar{p}$
decay, as first proposed in [2].  as well as the most recent review article on
the subject [3]. A more complete review can be found in [4]. With this new
experimental knowledge we can expect to deepen our understanding of both the
QCD and hadronic interactions. Since the observation of the first exotic
hadron, the $X(3872)$, in 2003, one of the main questions in the field is:
which is the internal structure [5] and spatial configuration of these
multiquark states ? The two most accepted configurations are loosely bound
hadron molecules and compact tetraquarks. This question gained a new interest
after the recent discovery of the $T_{cc}^{+}$ [6].

We may try to answer this question using the heavy ion collisions collisions
performed at LHC, as suggested in [7]. In these collisions, the number of
produced charm quarks is much larger than in any other type of collision.
Moreover, even if the environment is much less clean than in $e^{+}\,e^{-}$ or
in $p\,p$ collisions, modern techniques allow for the reconstruction of these
states. The $X(3872)$ has been recently observed by the CMS collaboration in
$Pb-Pb$ collisions [8]. This opens a new chapter in heavy hadron spectroscopy.

The striking new feature of heavy ion collisions is the formation of the quark
gluon plasma (QGP). It introduces a new way of formation of hadrons:
coalescence during the back transition to the hadron phase. This is, of
course, a non-perturbative process, which can only be calculated with the help
of models, such as the one presented in [7]. This model yields an interesting
prediction: molecules are much more abundantly produced than tetraquarks! This
is due to the fact that, at the hadronization time, the quark system has a
density which is comparable to normal hadronic matter, in which three quarks
occupy the typical volume of a baryon and a quark-antiquark pair occupies the
volume of a meson. At this point, two formed mesons have a typical separation
of a few fermi and can easily form a molecule. In sharp contrast, the typical
separation between the quarks is much larger than the size of compact
tetraquarks ($\simeq 0.2-0.5$ fm). The quantitave implementation of these
ideas requires several assumptions concerning the quark distribution in the
QGP, the interactions in the bound state and its wave function. The numerical
results confirm the intuitive expectation and the conclusion is that molecules
would be about hundred times more abundant than tetraquarks. This prediction
was first published many years ago in (see references in [7]) but a proper
treatment of the hadronic interactions of the multiquark states was only
developed in the most recent years. We are going to briefly discuss the
subject in the next sections.

##  2 Interactions in a hadron gas

After being formed during the hadronization, the particles have to live for
some time ($\simeq 10$ fm) in a hot hadron gas, where they interact with light
mesons. These interactions can be described by effective Lagrangians. Using
this approach, we have studied the interactions of the $X(3872)$ [9], the
$Z_{b}$ [10], the $J/\psi$ [11], the $\Upsilon$ [12] and, more recently, the
$T_{cc}$ [13]. Even though the Lagrangian may be the same, molecules and
tetraquarks interact in a different way. The difference comes from their
different sizes and it is encoded in the form factors and coupling constants.
For tetraquarks, these quantities can be calculated with QCD Sum Rules (QCDSR)
[14, 15]. For molecules there are models.

In [13] we compared the interactions of a “pure” molecular state with the
interactions of a “pure” tetraquark state in a hadronic medium.

The molecular model was taken from [16]. In this “quasi-free” model of the
$T_{cc}^{+}$, the $D$ and the ${D}^{*}$ are considered to just “fly” together.
Both the binding energy and the interaction between these constituent mesons
are neglected. They are destroyed by any elastic scattering between a $\pi$
and one of the constituent charm mesons. The advantage of this approach is
that the only Lagrangian required is the one of the $D^{*}D\pi$ vertex.
Fortunately the $D^{*}\to D\pi$ decay is measured and the coupling constant is
known. As in all effective Lagrangian approaches, we need a form factor which
contains a free parameter (the cut-off $\Lambda$). This is the main source of
uncertainty.

In [13] we also developed the tetraquark approach. In this case the
$T_{cc}^{+}$ is a degree of freedom, a new field appearing in the Lagrangians.
We have several interaction vertices, described by Lagrangians. All of them
were already known from previous studies, where with the help of QCDSR, we had
computed the form factors and coupling constants [17]. The new ingredient is
the $T_{cc}^{+}\,D\,D^{*}$ vertex. In [13] we performed the QCDSR study of the
relevant three-point function and determined the form factor and the coupling
constant in this vertex.

With these results we could compute the absorption cross sections of
$T_{cc}^{+}$ by pions in both the molecular and tetraquark approach. The
comparison is shown in Fig. 1. As we might expect, the molecular cross section
is much larger. It is then much easier (by almost two orders of magnitude) to
destroy a molecule than a tetraquark. In hadronic interactions it is also easy
to create the $T_{cc}^{+}$ in the inverse processes, such as, for example,
$D^{*}\,D\to T_{cc}^{+}\,\pi$. However, another robust conclusion of [13]
valid both for molecules and tetraquarks, is that the cross section for
$T_{cc}^{+}$ production is ten times smaller than the cross section for
$T_{cc}^{+}$ absorption.

Figure 1: Cross section of the process $T_{cc}^{+}\,\pi\to X$. The “quasi-
free” molecular is represented by the black solid line (the cut-off was chosen
to be $\Lambda=1$ GeV. The other bands show the results obtained with the
tetraquark model. The (wider) blue and red bands show results obtained with
empirical gaussian and monopole form factors (the boundaries of the bands are
defined by the cut-off choices going fom $1$ to $2$ GeV. The central (thinner)
grey band shows the results obtained with form factors derived from QCDSR.

Apart from the numbers, it is interesting to observe how the QCDSR calculation
reduces the uncertainties. Moreover, the uncertainties in QCDSR come from
quantities, such as quark masses and condensates, which can be constrained by
other experiments.

##  3 Hadron multiplicities and system size dependence

Having computed the cross sections, we can compute the corresponding thermal
cross sections and write rate equations for the abundances $N_{T_{cc}}$ and
$N_{X(3872)}$. This was done in [18] and the result is shown in Fig. 2. The
initial conditions are given by the coalescence model [7]. Comparing the two
panels, we see that, not surprisingly, the multiplicities of $T_{cc}$ and
$X(3872)$ evolve with time in a quite similar way and the differences come
mostly from the different initial conditions. It is also natural the fact that
the multiplicities decrease for molecules and increase for tetraquarks, since
the latter are much less abundant. Finally, looking at these curves we can
observe that the particle abundances are approaching constant values. This
would mean that they are reaching the chemical equilibrium. Interestingly, the
approach to equilibrium depends on the configuration and tetraquarks seem to
be already in equilibrium at the end of hadron gas phase in the considered
heavy ion reaction, in contrast to the molecular configuration. This raises
the possibility of discriminating between the two configurations. In the
example shown in the figure, if the observed value of the abundance would be
close to the value predicted by the Statistical Hadronization Model (SHM)[19],
then we could conclude that the particle is a tetraquark.

Figure 2: Time evolution of the $T_{cc}$ (upper panel) and $X(3872)$ (lower
panel) abundances as a function of the proper time in central $Pb-Pb$
collisions at $\sqrt{s_{NN}}=5.02$ TeV. Details can be found in [18].

In order to make predictions for the measurable particle abundances we would
have to combine our microscopic description of the interactions with a full
simulation of the underlying event, i.e., we would need a detailed code able
to generate the whole heavy ion collision. However, using various pieces of
experimental information, it is possible to establish a connection between the
abundance of particles given by the coalescence model and the system size,
represented by the charged particle density at mid-rapidity
$dN/d\eta\,(\eta=0)$. This was done in [18] and the result is shown in Fig. 3.
As it can be seen, as we move to smaller systems (smaller values of
$\mathcal{N}$) the difference of molecule and tetraquark yields increases.
This happens because smaller systems live for shorter times and there is no
time for the absorption-creation dynamics to be effective enough and reduce
the difference in the yields. In this sense Fig. 3 is a consequence of Fig. 2.
The pattern of separating curves for smaller values of $\tau$ translates into
the same pattern for smaller values of $\mathcal{N}$. The predictions of Fig.
3 can be compared with future data. Of course, we have to keep in mind that,
as the system becomes smaller, the whole description based on QGP formation
looses validity.

##  4 Discussion and conclusion

The picture developed above received some criticism, which we review below.

A first criticism was addressed to the coalescence model 222We deeply grateful
to S.H. Lee and to R. Rapp, for formulating this criticism so clearly.. It
assumes that if the overlap between the quark spatial distribution in the
final stage of the QGP and the size of the formed hadron is large, then the
abundance of this hadron will be large. If this was true, the yield of
$\psi(2S)$ ($r\simeq 0.8$ fm) would be larger than the yield of $J/\psi$
($r\simeq 0.4$ fm). Data show the opposite! Although this argument is correct,
it would be necessary to develop a theory for the interaction of the
$\psi(2S)$ with light mesons. It might well be the case that these
interactions would be responsible for the suppression of this particle.
Moreover the sizes are not so different as in the case of multiquark states
and the initial multiplicities given by the coalescence model would certainly
favor the $\psi(2S)$ over the $J/\psi$, but not by a factor 100 !

A second criticism is the following. In heavy ion collisions we also observe
deuterons, which are the prototype of a very weakly bound hadronic system.
Also, the predictions of the SHM for particle yields have been, to a large
extent, confirmed. Even for deuterons, as shown in [19]. This suggests that
the hot system formed in heavy ion collisions reaches chemical equilibrium.
When this happens the production yield is controled solely by the mass,
chemical potential and temperature. The details of the initial conditions and
interactions are “forgotten”. How to reconcile the microscopic transport
dynamics with the success of the SHM, which knows nothing about this dynamics
?

If the multiquark system is a molecule, one may assume that, if it is formed
in quark-hadron transition, it is completely destroyed, absorbed by the
medium. Molecule production (such as the deuteron) happens only at the last
moment of the hadron gas phase by hadron coalescence. Then, if the chemical
and kinetical freeze-out temperatures are close enough, the predictions of the
SHM and the coalescence model may coincide. However, this has to be checked.

If the multiquark system is a tetraquark, it may be formed already inside the
QGP and inherit its thermal properties [20]. Then it would traverse the hadron
gas almost without interacting (in a sort of “color transparency”) and emerge
at the end of the collision remembering its temperature. Again, this almost
non-interacting behavior of tetraquarks has to be proven. Our results, shown
in Fig. 1, suggest that the interaction cross section is sizeable.

To conclude, we believe that heavy ion collisions mark the beginning of a new
era for the study of exotic hadrons. From the theoretical side, it is
necessary to have the best possible description of the interactions of the
exotic mesons in the hadron gas.

Figure 3: Multiplicity as a function of $\mathcal{N}=(d\,N/\,d\eta)^{1/3}$.
Details can be found in [18].

## References

  * [1] Chen Chen and Elisabetta Spadaro [LHCb Collaboration], LHC Seminar (2022), https://indico.cern.ch/event/1176505/.
  * [2] S. J. Brodsky and F. S. Navarra, Phys. Lett. B 411, 152 (1997).
  * [3] L. Maiani and A. Pilloni, arXiv:2207.05141 [hep-ph].
  * [4] A. Esposito, A. Pilloni and A. D. Polosa, Phys. Rept. 668, 1 (2017).
  * [5] S. Narison, F. S. Navarra and M. Nielsen, Phys. Rev. D 83, 016004 (2011).
  * [6] R. Aaij et al. [LHCb], Nature Phys. 18, 751 (2022); F. S. Navarra, M. Nielsen and S. H. Lee, Phys. Lett. B 649, 166 (2007).
  * [7] S. Cho et al. [ExHIC], Prog. Part. Nucl. Phys. 95, 279 (2017).
  * [8] A. M. Sirunyan et al. [CMS], Phys. Rev. Lett. 128, 032001 (2022).
  * [9] A. Martinez Torres, K. P. Khemchandani, F. S. Navarra, M. Nielsen and L. M. Abreu, Phys. Rev. D 90, 114023 (2014); [erratum: Phys. Rev. D 93, 059902 (2016)].
  * [10] L. M. Abreu, F. S. Navarra, M. Nielsen and A. L. Vasconcellos, Eur. Phys. J. C 78, 752 (2018).
  * [11] L. M. Abreu, K. P. Khemchandani, A. Martínez Torres, F. S. Navarra and M. Nielsen, Phys. Rev. C 97, 044902 (2018); F. S. Navarra, M. Nielsen, R. S. Marques de Carvalho and G. Krein, Phys. Lett. B 529, 87 (2002).
  * [12] L. M. Abreu, F. S. Navarra and M. Nielsen, Phys. Rev. C 101, 014906 (2020).
  * [13] L. M. Abreu, F. S. Navarra, M. Nielsen and H. P. L. Vieira, Eur. Phys. J. C 82, 296 (2022).
  * [14] M.A. Shifman, A.I. Vainshtein and V.I. Zakharov, Nucl. Phys. B147, 385 (1979); Nucl. Phys. B147, 448 (1979).
  * [15] S. Narison, QCD spectral sum rules, World Sci. Lect. Notes Phys. 26, 1 (1989); QCD as a theory of hadrons, Cambridge Monogr. Part. Phys. Nucl. Phys. Cosmol. 17, 1 (2004) [hep-ph/0205006]; R. M. Albuquerque, J. M. Dias, K. P. Khemchandani, A. Martínez Torres, F. S. Navarra, M. Nielsen and C. M. Zanetti, J. Phys. G 46, 093002 (2019).
  * [16] J. Hong, S. Cho, T. Song and S. H. Lee, Phys. Rev. C 98, 014913 (2018).
  * [17] M. E. Bracco, M. Chiapparini, F. S. Navarra and M. Nielsen, Prog. Part. Nucl. Phys. 67, 1019 (2012).
  * [18] L. M. Abreu, F. S. Navarra and H. P. L. Vieira, Phys. Rev. D 105, 116029 (2022).
  * [19] A. Andronic, P. Braun-Munzinger, D. Gündüz, Y. Kirchhoff, M. K. Köhler, J. Stachel and M. Winn, Nucl. Phys. A 1010, 122176 (2021).
  * [20] B. Wu, X. Du, M. Sibila and R. Rapp, Eur. Phys. J. A 57, 122 (2021); [erratum: Eur. Phys. J. A 57, 314 (2021)].