# Clinically-Inspired Multi-Agent Transformers
for Disease Trajectory Forecasting
from Multimodal Data

Huy Hoang Nguyen *  A  , Matthew B. Blaschko *  A  , Simo Saarakkala *  A  ,
and Aleksei Tiulpin *  A  Huy Hoang Nguyen is with the Research Unit of Health
Sciences and Technology, University of Oulu, Finland. E-mail:
<EMAIL_ADDRESS>Blaschko is with Center for Processing Speech &
Images, KU Leuven, Belgium. E-mail<EMAIL_ADDRESS>Saarakkala is with the Research Unit of Health Sciences and Technology,
University of Oulu, Finland and Department of Diagnostic Radiology, Oulu
University Hospital, Finland. E-mail<EMAIL_ADDRESS>Tiulpin
is with the Research Unit of Health Sciences and Technology, University of
Oulu, Finland and Neurocenter Oulu, Oulu University Hospital. E-mail:
<EMAIL_ADDRESS>
###### Abstract

Deep neural networks are often applied to medical images to automate the
problem of medical diagnosis. However, a more clinically relevant question
that practitioners usually face is how to predict the future trajectory of a
disease. Current methods for prognosis or disease trajectory forecasting often
require domain knowledge and are complicated to apply. In this paper, we
formulate the prognosis prediction problem as a one-to-many prediction
problem. Inspired by a clinical decision-making process with two agents – a
radiologist and a general practitioner – we predict prognosis with two
transformer-based components that share information with each other. The first
transformer in this framework aims to analyze the imaging data, and the second
one leverages its internal states as inputs, also fusing them with auxiliary
clinical data. The temporal nature of the problem is modeled within the
transformer states, allowing us to treat the forecasting problem as a multi-
task classification, for which we propose a novel loss. We show the
effectiveness of our approach in predicting the development of structural knee
osteoarthritis changes and forecasting Alzheimer’s disease clinical status
directly from raw multi-modal data. The proposed method outperforms multiple
state-of-the-art baselines with respect to performance and calibration, both
of which are needed for real-world applications. An open-source implementation
of our method is made publicly available at https://github.com/Oulu-
IMEDS/CLIMATv2.

###### Index Terms:

Deep Learning, knee, osteoarthritis, prognosis prediction.

##  I Introduction

Recent developments in Machine Learning (ML) suggest that it is soon to be
tightly integrated into many fields, including healthcare [1, 2]. One
particular subfield of ML – Deep Learning (DL) has advanced the most, as it
opened the possibility to make predictions from high-dimensional data. In
medicine, this impacted the field of radiology, in which highly trained human
readers identify pathologies in medical images. The full clinical pipeline,
however, aims to assess the condition of a patient as a whole, and eventually
prescribe the most relevant treatment for a disease [3, 4]. Using DL in this
broad scope by integrating multimodal data has the potential to provide even
further advances in medical applications.

Figure 1: The concept of CLIMATv2 was inspired by a multi-agent decision-
making system with a radiologist and a general practitioner. All types of
imaging data of disease are handled by the radiologist. The general
practitioner then utilizes a report description produced by the radiologist
and the context of clinical variables to forecast a future trajectory of the
disease.

Clinical diagnosis is made by specialized treating physicians or general
practitioners. These doctors are not radiologists and rather use the services
of the latter in decision-making. One of the typical problems that such
doctors face is to make a prognosis [5, 6], which can be formalized as disease
trajectory forecasting (DTF). This is an especially relevant task in
degenerative disorders, often seen in musculoskeletal and nervous systems.
This work studies DTF for knee osteoarthritis (OA) – the most common
musculoskeletal disorder [7], and Alzheimer’s disease (AD) – the leading cause
of dementia [8].

Among all the joints in the body, OA is mostly prevalent in the knee. Knee OA
is characterized by the appearance of osteophytes, and the narrowing of joint
space [9], which in the clinical setting are usually imaged using X-ray
(radiography) [10]. The disease severity is graded according to the Kellgren-
Lawrence system [11] from 0 (no OA) to 4 (end-stage OA), or Osteoarthritis
Research Society International (OARSI) atlas criteria [12]. Unfortunately, OA
is progressive over time (see Figure 2) and no cure has yet been developed for
OA. However, diagnosing OA at an early stage may allow the slowing down of the
disease, for example using behavioral interventions [13].

(a) BL - KL $0$ (b) Y3 - KL $1$ (c) Y$6$ \- KL $3$ (d) Y$8$ \- TKR

Figure 2: Radiographs of a patient with knee OA progressed in $8$ years. The
orange arrow indicates joint space narrowing. The disease progressed from
Kellgren-Lawrence (KL) grade $0$ at the baseline (BL) to $3$ in $6$ years. At
the $8$th year, the patient underwent a total knee replacement (TKR) surgery.

Individuals with AD have difficulties with reading, learning, and even
performing daily activities. AD is fatally progressive and caused more than
$120,000$ deaths in the United States in $2019$; however, no effective cure
for it has been made available [8]. The benefits of early AD diagnosis are
similar to OA – the progression of the disease can be delayed, and patients
may be assigned relevant care in a timely manner [14].

In both of the aforementioned fields – OA and AD, there is a lack of studies
on prognosis prediction. From an ML perspective, a more conventional setup is
to predict whether the patient has the disease at present or a specific point
of time in the future [15, 16, 17, 18, 19, 20, 21]. However, prognosis
prediction aims to answer whether and how the disease would evolve over time.
Furthermore, in a real-life situation, the treating physician makes the
prognosis while interacting with a radiologist or other stakeholders who can
provide information (e.g. blood tests or radiology reports) about the
patient’s condition. This also largely differentiates the diagnostic task from
predicting a prognosis.

In this paper, we present an extended version of our earlier work on automatic
DTF [22], where we proposed a Clinically-Inspired Multi-Agent Transformers
(CLIMAT) framework, aiming to mimic the interaction process between a general
practitioner / treating physician111In the sequel we write general
practitioner, which, however, does not restrict our modeling approach and a
radiologist. In our system, a radiologist module, consisting of a feature
extractor (convolutional neural network; CNN) and a transformer, analyses the
input imaging data and then provides an output state of the transformer
representing a radiology report to the general practitioner – corresponding
module (purely transformer-based). The latter fuses this information with
auxiliary patient data, and makes the prognosis prediction. We graphically
illustrate the described idea in Figure 1.

Compared to the conference version [22], we have enhanced our framework, such
that the module corresponding to the general practitioner does not only
perform prognosis, but is also encouraged to make diagnostic predictions
consistent with a radiologist module. The earlier version of CLIMAT relies on
a simplifying assumption in relation to the independence between the
diagnostic label task and non-imaging data. The introduced update helps the
framework to expand out of the knee osteoarthritis domain, and be more
realistic, thereby allowing our method to be applied in fields where diagnosis
could rely on both imaging and non-imaging data. Moreover, we equip the
framework with a new loss – Calibrated Loss based on Upper Bound (CLUB) – that
aims to maintain the performance while improving the calibration of the
framework’s predictions. Finally, we have also expanded the application of our
framework to the case of AD.

To summarize, our contributions are the following:

  1. 1.

We propose CLIMATv2, a clinically-inspired transformer-based framework that
can learn to forecast disease severity from multi-modal data in an end-to-end
manner. The main novelty of our approach is the incorporation of prior
knowledge of the decision-making process into the model design.

  2. 2.

We derive the CLUB loss, an upper bound on a temperature-scaled cross-entropy
(TCE), and apply it to the DTF problem we have at hand. Experimentally, we
show that CLUB provides better calibration and yields similar or better
balanced accuracy than the competitive baselines.

  3. 3.

From a clinical perspective, our results show the feasibility to perform fine-
grained prognosis of knee OA and AD directly from raw multi-modal 2D and 3D
data.

##  II Related Work

####  II-1 Knee osteoarthritis prognosis

The attention of the literature has gradually been shifting from diagnosing
the current OA severity of a knee to predicting whether degenerative changes
will happen within a specified time frame. While some studies [15, 16, 17]
aimed to predict whether knee OA progresses within a specified duration,
others [18, 19] tried to predict if a patient will undergo a total knee
replacement (TKR) surgery at some point in the future. However, the common
problem of the aforementioned studies is that the scope of knee OA progression
is limited to a single period of time or outcome, which substantially
differentiates our work from the prior art.

(a) Axial

(b) Coronal

(c) Sagittal

Figure 3: The three projections of a 3D FDG-PET scan, which is converted to
the jet colormap for demonstration purposes. The red regions are associated
with high concentrations of the FDG radioactive tracer in the brain.

####  II-2 Alzheimer’s disease prognosis

Compared to the field of OA, a variety of approaches have been proposed to
process longitudinal data in the AD field. Lu et al. [21] utilized a fully-
connected network (FCN) to predict AD progression within a time frame of $3$
years from magnetic resonance imaging (MRI) and fluorodeoxyglucose positron
emission tomography (FDG-PET) scans. Ghazi et al. [23] and Jung et al. [20]
used different long-short-term memory (LSTM)-based models to predict AD
clinical statuses from scalar MRI biomarkers. Albright et al. [24] took into
account various combinations of scalar measures and clinical variables to
predict changes in AD statuses using FCNs and recurrent neural networks (RNN).
In contrast to the prior art relying on either raw imaging data or scalar
measures, our method enables learning from raw imaging scans, imaging-based
measurements, and other scalar variables simultaneously. Additionally, whereas
FCN and sequential networks were widely used in the literature, we propose to
use a transformer-based framework to perform the AD clinical status prognosis
task. Furthermore, we use FCN, two well-known sequential models – gated
recurrent unit (GRU) and LSTM – as our reference approaches.

(a) CLIMATv1 [22]

(b) CLIMATv2

Figure 4: The CLIMAT framework (best viewed in color). There are $N$ and $M$
input imaging and non-imaging feature vectors, respectively. The first feature
vector $\bar{\bm{h}}_{C}^{0}$ of the last layer of the transformer C is
appended to every output vector of $\bar{\bm{h}}_{R}$ to form the input for
the transformer P. All the blue blocks are transformer-based networks. $[CLS]$
and $[POS]$ embeddings are in white and orange, respectively.

####  II-3 Transformers for vision tasks

Although originally developed in the field of natural language processing [25,
26], transformer-based architectures have recently been applied also in vision
tasks. Dosovitskiy et al. [27] pioneered the use of transformer-based
architectures without a CNN for image classification problems. Girdhar et al.
[28] and Arnab et al. [29] studied the same family of architectures to perform
video recognition tasks. However, Hassani et al. [30] pointed out such pure
transformers require a significantly large amount of imaging data to perform
well. The reason is that transformers do not have well-informed inductive
biases, which are strengths of CNNs. Thus, our method relies on [30] due to
medium dataset sizes.

####  II-4 Multimodal data processing with transformers.

Transformers have been empirically robust in learning various categories of
tasks from sequential data such as text or tabular data [25, 31]. However, in
medical imaging, it is common to acquire multiple modalities comprising both
raw images (e.g. plain radiographs, MRI, or PET scans) and tabular data, which
are challenging for a single transformer. Recent work has shown that multiple
transformers are needed to for such multiple modalities [32]. Therefore,
similar to our previous version [22], this study adapts the idea of using
multiple transformers in our framework to perform DTF from multiple
modalities.

##  III Methods

###  III-A The CLIMAT framework: a conceptual overview

As mentioned earlier, we base our framework on multi-agent decision-making
processes in a clinical setting. In many applications, this can be considered
information passing between two agents – a radiologist and a general
practitioner [33]. While the radiologist specializing in imaging diagnosis is
in charge of producing radiology reports, the general practitioner relies on
various modalities including the radiologic findings to forecast the severity
of a certain disease. We model such collaboration by the concept presented in
Figure 1. Specifically, the radiologist analyzes a medical image $\bm{x}_{0}$
(e.g. radiograph or PET image) of a patient to provide an interpretation with
rich visual description and annotations, allowing the diagnosis of the current
stage $y_{0}^{R}$ of the disease. Subsequently, the general practitioner
relies on (i) the clinical data $\bm{m}_{0}$ (e.g. questionnaires or
symptomatic assessments) with a further interpretation if needed, (ii) the
provided radiology report, and (iii) the referenced diagnosis of the
radiologist $y_{0}^{R}$ to predict the course of the disease $y_{0:T}$.

We implement the concept proposed above in the CLIMATv2 framework (see Figure
4 and Section III-B). CLIMATv2 comprises three primary transformer-based
blocks222Hereinafter, we use the terms _block_ and _transformer_
interchangeably. – namely Radiologist (R), Context (C), and General
Practitioner (P). Firstly, assume that we obtain visual features learned from
the imaging data $\bm{x}_{0}$. Then, the block R acts as the radiologist to
perform visual reasoning from the visual features and predict the current
stage $\hat{y}_{0}^{R}$ of a disease. The other two blocks are responsible for
context extracting and prognosis predicting. As such, the block C aims to
extract a context embedding from clinical variables $\bm{m}_{0}$.
Subsequently, the block P utilizes the combination of the context embedding
and the output states of the block R to forecast the disease trajectory
$\hat{y}_{0:T}$.

In this work, we have two major upgrades to CLIMATv1 [22]. Firstly, we do not
assume anymore that $y_{0}$ and $\bm{m}_{0}$ are independent, as this does not
hold in many medical imaging domains, e.g. for OA [34]. Namely, in the current
version of CLIMAT, both the blocks R and P have now been allowed to make
diagnosis predictions simultaneously, making sure that the learned embeddings
contain information on $y_{0}$. Furthermore, we encourage their predictions to
be consistent with the final module of our model. Secondly, besides
performance, in this work, we take into account model calibration, which
allows us to gain better insights into the reliability of models’ predictions
[35]. To facilitate better calibration within our proposed framework, we
propose a novel loss, called CLUB, presented in Section III-C.

###  III-B Technical realization

####  III-B1 Transformer

A transformer encoder comprises a stack of $L$ multi-head self-attention
layers, whose input is a sequence of vectors $\\{\bm{s}_{i}\\}_{i=1}^{N}$
where $\bm{s}_{i}\in\mathbb{R}^{1\times C}$, and $C$ is the feature size. As
such, a transformer is formulated as [25]

$\displaystyle\bm{h}_{0}$
$\displaystyle=[\bm{E}_{[CLS]},\bm{s}_{1},\dots,\bm{s}_{N}]+\bm{E}_{[POS]},$
(1) $\displaystyle\bm{z}_{l-1}$
$\displaystyle=\textsc{MSA}(\textsc{LN}(\bm{h}_{l-1}))+\bm{h}_{l-1},$ (2)
$\displaystyle\bm{h}_{l}$
$\displaystyle=\textsc{MLP}(\textsc{LN}(\bm{z}_{l-1}))+\bm{z}_{l-1},\quad
l=\\{1,\dots,L\\}$ (3) $\displaystyle\bar{\bm{h}}$ $\displaystyle=\bm{h}_{L}$
(4)

where $\bm{E}_{[CLS]}$ is a learnable token, $\bm{E}_{[POS]}$ is a learnable
positional embedding, and $\bar{\bm{h}}$ represents features extracted from
the last layer. MLP is a multi-layer perceptron (i.e. a fully-connected
network), LN is the layer normalization [36], and $\textsc{MSA}(\cdot)$ is a
multi-head self-attention (MSA) layer [25]. The self-attention mechanism
relies on the learning of query, key, value parameter matrices, denoted by
$\bm{W}^{Q}_{l},\bm{W}^{K}_{l}$, and $\bm{W}^{V}_{l}$ with $l=1,\dots,L$,
respectively. Initially, we simultaneously set $\bm{Q}_{0},\bm{K}_{0}$, and
$\bm{V}_{0}$ to $\bm{h}_{0}$ defined in Eq. (1). When iterating through layers
$l=1,\dots,L$, we update the states as follows

$\begin{split}\bm{Q}_{l}&=\bm{Q}_{l-1}\bm{W}^{Q}_{l}\\\
\bm{K}_{l}&=\bm{K}_{l-1}\bm{W}^{K}_{l}\\\
\bm{V}_{l}&=\bm{V}_{l-1}\bm{W}^{V}_{l}\end{split}$ (5)

Finally, the self-attention is established thanks to the _scaled dot-product_
function applied to $\bm{Q}_{l}$, $\bm{K}_{l}$, and $\bm{V}_{l}$, and defined
as

$\mathrm{Attention}(\bm{Q}_{l},\bm{K}_{l},\bm{V}_{l})=\mathrm{Softmax}\left(\frac{\bm{Q}_{l}\bm{K}_{l}^{\intercal}}{\sqrt{d_{k}}}\right)\bm{V}_{l},$
(6)

where $d_{k}$ is the feature dimension of $\bm{Q}_{l}$. In essence,
$\bm{Q}_{l}\bm{K}_{l}^{\intercal}$ represents the association between all
pairs of queries and keys. The normalization based on $d_{k}$ is critical to
address the case where the magnitude of entries in
$\bm{Q}_{l}\bm{K}_{l}^{\intercal}$ is too large. The essential part that
produces the attention is the utilization of softmax, which allows for the
creation of a normalized heatmap over the association of $\bm{Q}_{l}$ and
$\bm{K}_{l}$. Subsequently, by adding more sets of learnable weights
$\bm{W}_{l}^{Q}$, $\bm{W}_{l}^{K}$, and $\bm{W}_{l}^{V}$, we can obtain MSA by
concatenating different output heads of attention. Precisely, the MSA
mechanism is formulated as follows

$\begin{split}\mathrm{head}^{h}_{l}&=\mathrm{Attention}(\bm{Q}_{l},\bm{K}_{l},\bm{V}_{l}),\quad
h=1,\dots,H\\\
\mathrm{MSA}(\cdot)&=\mathrm{Concat}(\mathrm{head}^{1}_{l},\dots,\mathrm{head}^{H}_{l})\bm{W}^{O}_{l},\end{split}$

where $H$ is the number of heads, and $\bm{W}_{l}^{O}$ represents learning
parameters associated with the $H$ output heads.

The three main blocks in our framework are transformer-based networks (see
Figure 4). While the blocks R and C have only $1$ $[CLS]$ token, the block P
can include $K~{}[CLS]$ tokens to allow for multi-target predictions. The
hyperparameter $K$ is introduced in the block P to ensure that there are
enough output heads for multi-task predictions. We typically set $K$ to $1$ or
$T+1$. In the case $K=T+1$, each output head has a corresponding $[CLS]$
token.

####  III-B2 Multimodal feature extraction

Our framework is able to handle multimodal imaging and non-imaging data. As
input data can be clinical variables, raw images (i.e. 2D or 3D images), and
biomarkers extracted by human experts or specialized software, we have
distinct feature extraction modules for different input formats. Specifically,
we use the feed-forward network (FFN), 2D-CNN, and 3D-CNN-based architectures
for scalar or 1D inputs, 2D, and 3D images, respectively. As such, we pre-
define common feature lengths $C_{X}$ and $C_{M}$ for all imaging and non-
imaging embeddings, respectively. Each FFN-based feature extractor consists of
a linear layer, a GELU activation [37], layer normalization [36], and has an
output shape of $1\times C_{X}$ or $1\times C_{M}$ depending on the type of
input data. In the CNN-based modules, we first unroll their output feature
maps into sequences of feature vectors per image super-pixel or super-voxel,
then linearly project them into a $C_{X}$-dimensional space.

####  III-B3 Radiologist module

The Radiologist block is a transformer network with $L_{R}$ layers and is
responsible for processing all imaging features previously extracted in
Section III-B2. For the input data preparation, we concatenate all features of
different imaging modalities to form a sequence of length $N$ that contains
$C_{X}$-dimensional image representations. Subsequently, we propagate this
sequence through the transformer R. To this end, the visual embedding
$\bar{\bm{h}}_{R}\in\mathbb{R}^{(N+1)\times C_{X}}$ produced by its last layer
serves two purposes: representing radiology reports and visual features for
diagnosis predictions. For the former, we subsequently combine
$\bar{\bm{h}}_{R}$ with non-imaging embeddings to constitute inputs for the
General Practitioner block (see Section III-B5). For the latter, following a
common practice in [38, 39, 40], we perform an average pooling onto
$\bar{\bm{h}}_{R}$ to generate a $C_{X}$-dimensional vector. Afterward, we
pass the resulting vector through an FFN comprised of a linear layer, a GELU
activation [37], and a layer normalization [36] to predict the current stage
$y_{0}^{R}$ of the disorder (see Figure 4).

####  III-B4 Clinical context embedding module

Here, we aim to mimic the comprehension of a general practitioner over
different clinical modalities (e.g. questionnaires, extra tests, and risk
factors). As such, we take a single $[CLS]$ embedding followed by $M$ clinical
vector representations extracted in Section III-B2 to form the input sequence
for the Context block (see Figure 4). The underlying architecture of the block
is a transformer-based network. After passing the input sequence through the
transformer C with $L_{C}$ layers, we merely use the first feature vector
$\bar{\bm{h}}_{C}^{0}$ of the last feature maps $\bm{h}_{L_{C}}$ as a common
contextual token representing all the non-imaging modalities.

####  III-B5 General Practitioner module

As soon as the contextual token of length $C_{M}$ is acquired from the Context
block, we concatenate $N+1$ copies of the token $\bar{\bm{h}}_{C}^{0}$ into
the last states $\bar{\bm{h}}_{R}$ of the transformer R to generate a sequence
of $N+1$ mixed feature vectors with a feature size of $C_{X}+C_{M}$. We then
process the obtained sequence using Eq. (1) to have the sequence of $(K+N+1)$
feature vectors. Here, we utilize the third transformer-based module to
simulate the analysis of the general practitioner over all sources of data for
prognosis predictions. Specifically, after passing the input sequence through
the transformer P, we utilize the first $T+1$ vector representations of the
last layer to forecast the disease severity trajectory
$(\hat{y}_{0},\dots,\hat{y}_{T})$. Predicting disease severity at each time
point requires a common or distinct FFN, which comprises a layer normalization
followed by two fully connected layers separated by a GELU activation [37].

###  III-C Calibrated Loss based on Upper Bound for Multi-Task

####  III-C1 Motivations and formulation

Compared to CLIMATv1 [22], we aim to optimize not only the performance but
also the calibration of our model’s predictions. As CLIMATv2 simultaneously
predicts a sequence of $T+1$ targets with different difficulties, we treat it
as a multi-task predictive model. The temporal information here is contained
within the transformer states. Inspired by [41], to harmonize all the tasks,
we propose the CLUB loss (abbreviated from Calibrated Loss Upper Bound).
However, unlike [41], which relies on the _‘not always true’_ assumption that
$\frac{1}{\sigma}\sum_{c^{\prime}}\exp\left(\frac{1}{\sigma^{2}}f_{c^{\prime}}(x)\right)=\left(\sum_{c^{\prime}}\exp\left(f_{c^{\prime}}(x)\right)\right)^{\frac{1}{\sigma^{2}}}$,
where $\sigma$ is a noise factor and $f_{c^{\prime}}(.)$ is the
$c^{\prime}$-th element of the logits produced by a parametric function $f$,
we theoretically derive CLUB as an upper bound of temperature-scaled cross-
entropy (CE) loss.

Consider the $t$-th task with $t\in\\{0\dots T\\}$, let
$\bm{f}_{t}=(f_{t,1},\dots,f_{t,N_{n}^{t}})^{\intercal}\in\mathbb{R}^{N_{c}^{t}}$
denote predicted logits of CLIMATv2 (i.e. an output of the transformer P) on
task $t$, where $N_{c}^{t}$ is the number of classes of the $t$-th target. Let
$\bm{g}_{t}=(g_{t,1},\dots,g_{t,N_{c}^{t}})^{\intercal}=\exp(\bm{f}_{t})$.
Similar to [41], we model the affection of noise $\sigma_{t}$ onto the
prediction of $y_{t}$ in the scaled form
$\textsc{Softmax}\left(\frac{1}{\sigma_{t}^{2}+\varepsilon}\bm{f}_{t}\right)$,
where $\varepsilon\in\mathbb{R}_{+}$ is needed to ensure the scaled softmax to
be valid for all $\sigma_{t}\in\mathbb{R}$. For convenience, we temporarily
eliminate the $t$ index from all notations. By denoting
$\tau=\frac{1}{\sigma^{2}+\varepsilon}\in\mathbb{R}_{+}$, we rewrite the
scaled softmax as

$\textsc{Softmax}\left(\tau\bm{f}\right)=\left(\frac{g_{1}^{\tau}}{\sum_{c^{\prime}}g_{c^{\prime}}^{\tau}},\dots,\frac{g_{N_{c}}^{\tau}}{\sum_{c^{\prime}}g_{c^{\prime}}^{\tau}}\right)^{\intercal}\in[0,1]^{N_{c}},$
(7)

where $c$, $c^{\prime}$ are class indices, and $\tau~{}\in~{}\mathbb{R}_{+}$
is a noise factor.

Without the loss of generality, $c$ is assumed to be the ground truth class of
a certain input $x$. $\tau$ is the inverse temperature that can smoothen
($\tau\leq 1$) or sharpen ($\tau>1$) predicted probabilities. Here, one can
observe that
$\left(\sum_{c^{\prime}}g_{c^{\prime}}^{\tau}\right)^{\frac{1}{\tau}}$ can be
seen as an absolutely homogeneous function or an $\ell_{\tau}$-norm
$\left\|\bm{g}\right\|_{\tau}$ in a Lebesgue space, when $\tau$ belongs to
$(0,1)$ or $[1,\infty)$, respectively. Therefore, a TCE loss can be formulated
as

$\mathcal{L}_{\mathrm{TCE}}=-\log\frac{g_{c}^{\tau}}{\left\|\bm{g}\right\|_{\tau}^{\tau}},$
(8)

where $c$ is the true class. When $\tau=1$, the TCE loss becomes the vanilla
CE loss

$\mathcal{L}_{\mathrm{CE}}=-\log\frac{g_{c}}{\left\|\bm{g}\right\|_{1}}.$ (9)

For the purpose of improving calibration, we are interested in the case of
$\tau\in(0,1]$ [35], allowing us to apply the reverse Hölder’s inequality to
have $\left\|\bm{g}\right\|_{\tau}\leq
N_{c}^{(1-\tau)/\tau}\left\|\bm{g}\right\|_{1}$. Then, we can derive an upper
bound of $\mathcal{L}_{\mathrm{TCE}}$, called the CLUB loss, as

$\displaystyle\mathcal{L}_{\mathrm{CLUB}}$
$\displaystyle\triangleq-\tau\log\frac{g_{c}}{\left\|\bm{g}\right\|_{1}}+(1-\tau)\log
N_{c}$ $\displaystyle=\tau\mathcal{L}_{\mathrm{CE}}+(1-\tau)\log
N_{c},\quad\tau\in[0,1],$

where the equality holds if and only if $\tau=1$. Unlike
$\mathcal{L}_{\mathrm{TCE}}$, our CLUB loss directly depends on
$\left\|\bm{g}\right\|_{1}$ rather than $\left\|\bm{g}\right\|_{\tau}$. Eq.
(III-C1) indicates that $\mathcal{L}_{\mathrm{CLUB}}$ is a convex combination
between the CE loss (9) and $\log N_{c}$, which takes into account the task
complexity in terms of the number of classes.

####  III-C2 Performance and calibration optimization

In our setting, we consider each $\tau_{t}$ associated with task $t$ as a
learnable parameter. As the model’s parameters $\theta$ and $\tau_{t}$’s are
independent, we can respectively derive the gradients of
$\mathcal{L}_{\mathrm{CLUB}}(t)$ w.r.t. $\theta$ and $\tau_{t}$’s as follows

$\displaystyle\frac{\partial\mathcal{L}_{\mathrm{CLUB}}(t)}{\partial\theta}$
$\displaystyle=\tau_{t}\frac{\partial\mathcal{L}_{\mathrm{CE}}(t)}{\partial\theta},\
t=0\dots T,$ (10)
$\displaystyle\frac{\partial\mathcal{L}_{\mathrm{CLUB}}(t)}{\partial\tau_{t}}$
$\displaystyle=\mathcal{L}_{\mathrm{CE}}(t)-\log N_{c},\ t=0\dots T,$ (11)

where $\mathcal{L}_{\mathrm{CE}}(t)$ and $\mathcal{L}_{\mathrm{CLUB}}(t)$ are
the CE and CLUB losses on the $t$-th task, respectively. Whereas the
optimization w.r.t. $\theta$ essentially aims to improve the performance of
our model, learning $\tau_{t}$’s directly impacts its calibration quality.
Eqs. (10) and (11) indicate that $\tau_{t}$’s can be seen as learnable
coefficients of different tasks.

To effectively constrain $\tau_{t}\leq 1$ and avoid a trivial solution where
$\forall t\in\\{0,\dots,T\\},\tau_{t}=1$, we constrain the learnable
parameters $\\{\tau_{t}\\}_{t=0}^{T}$ using Algorithm 1. Specifically, Line
$1$ guarantees that $\rho_{t}$’s are valid for any $\sigma_{t}$’s. Lines $2$
to $4$ prevent all the $\tau_{t}$’s from converging to the obvious value $1$.
Lines $5$ and $6$ re-scales $\tau_{t}$’s such that merely ones with the
maximum values become $1$. This last step is necessary to avoid $\tau_{t}$’s
values being small inversely proportionally to the number of tasks.

###  III-D Multi-Task Learning for Disease Trajectory Forecasting

In practice, it is highly common to have data _not_ fully annotated. Thus, our
framework should allow for handling missing targets by design. As such, our
multi-task loss can tackle such an impaired condition with ease by using an
indicator function to mask out targets without annotation. Formally, we
minimize the following prognosis forecasting loss

$\displaystyle\mathcal{L}_{\mathrm{prog}}=\frac{1}{\sum_{t=0}^{T}\mathbb{I}_{t}}\sum_{t=0}^{T}\mathbb{I}_{t}\mathcal{L}_{\mathrm{CLUB}}(t),$
(12)

where $\mathbb{I}_{t}$ is an indicator function for task $t$.

Input: $T$: the number of future time points

Input: $\\{\sigma_{t}\\}_{t=0}^{T}$: noise parameters

Input: $\varepsilon\in\mathbb{R}_{+}$: hyperparameter

1 $\rho_{t}\xleftarrow{}\frac{1}{\sigma^{2}_{t}+\varepsilon}$ $t=0\dots T$

2 for  _$t=0,\dots,T$_ do

3
$\tilde{\rho}_{t}=\frac{\exp(\rho_{t})}{\sum_{t^{\prime}=0}^{T}\exp(\rho_{t^{\prime}})}$

4

5 end for

6
$\rho_{\max}\xleftarrow{}\textsc{Max}\left(\\{\tilde{\rho}_{t}\\}_{t=0}^{T}\right)$

7 $\tau_{t}\xleftarrow{}\frac{\tilde{\rho}_{t}}{\rho_{\max}}$ $t=0\dots T$

Algorithm 1 Constraint of $\tau_{t}\leq 1$, $t=0\dots T$

While the radiologist has strong expertise in imaging diagnosis, in relation
to prognosis, the general practitioner has more advantages due to the access
to multimodal data, such as the patient’s background. On the other hand,
general practitioners are also able to assess images to some extent. We
incorporate the corresponding prior into our learning framework by enforcing
consistency in predictions between the two agents:

$\mathcal{L}_{\mathrm{cons}}=\left\|\bm{f}_{0}^{R}-\bm{f}_{0}\right\|_{1},$
(13)

where $\bm{f}_{0}^{R}$ and $\bm{f}_{0}$ indicate logits of the blocks R and P
for diagnosis predictions, respectively. It is worth noting that while
$\mathcal{L}_{\mathrm{prog}}$ operates solely on annotated targets,
$\mathcal{L}_{\mathrm{cons}}$ optimizes all targets.

To optimize the whole framework, we minimize the final loss $\mathcal{L}$ as
follows

$\mathcal{L}=\mathcal{L}_{\mathrm{prog}}+\lambda\mathcal{L}_{\mathrm{cons}},$
(14)

where $\lambda\in\mathbb{R}_{+}$ is a consistency regularization coefficient.

##  IV Experiments

###  IV-A Data

In this study, we conducted experiments on two public datasets for knee OA and
AD. The overall description and subject selection of the two datasets and
corresponding tasks can be seen in Figures 5 and I. The details of data pre-
processing and prognosis prediction tasks are presented as follows.

(a) OAI dataset

(b) ADNI dataset

Figure 5: Subject selection in our study.

####  IV-A1 Knee OA structural prognosis prediction

We conducted experiments on the Osteoarthritis Initiative (OAI) cohort,
publicly available at https://nda.nih.gov/oai/. $4,796$ participants from $45$
to $79$ years old participated in the OAI cohort, which consisted of a
baseline, and follow-up visits after $12$, $18$, $24$, $30$, $36$, $48$, $60$,
$72$, $84$, $96$, $108$, $120$, and $132$ months. In the present study, we
used all knee images that were acquired with large imaging cohorts: the
baseline, and the $12$, $24$, $36$, $48$, $72$, and $96$-month follow-ups.

As the OAI dataset includes data from five acquisition centers, we used data
from $4$ centers for training and validation, and considered data from the
left-out one as an independent test set. On the former set, we performed a
$5$-fold cross-validation strategy.

Following [15, 42], we utilized the BoneFinder tool [43] to extract a pair of
knees regions from each bilateral radiograph, and pre-process each of them.
Subsequently, we resized each pre-processed image to $256\times 256$ pixels
(pixel spacing of $0.5mm$), and horizontally flipped it if that image
corresponds to a right knee.

We utilized the Kellgren-Lawrence (KL) as well as OARSI grading systems to
assess knee OA severity. The KL system classifies knee OA into $5$ levels from
$0$ to $4$, proportional to the OA severity increase. The OARSI system
consists of $6$ sub-systems – namely lateral/medial joint space (JSL/JSM),
osteophytes in the lateral/medial side of the femur (OSFL/OSFM), and
osteophytes in the lateral/medial side of the tibia (OSTL/OSTM). And according
to that the furthest targets in KL, JSL, and JSM were $8$ years from the
baseline while it was $4$ years for the other grading aspects.

Regarding the KL grading system, we grouped KL-$0$ and KL-$1$ into the same
class as they are clinically similar, and added TKR knees as the fifth class.
As a result, there were $5$ classes in KL, and there were $4$ severity levels
in each of the OARSI sub-systems. Following [15, 22], we utilized age, sex,
body mass index (BMI), history of injury, history of surgery, and total
Western Ontario and McMaster Universities Arthritis Index (WOMAC) as clinical
variables. We quantized the continuous variables, and presented each of them
by a $4$-element one-hot vector depending on the relative position of its
value in the interval created by the minimum and the maximum.

For clinical relevance, we did not perform knee OA prognosis predictions on
knees that underwent TKR or were diagnosed with the highest grade in any OARSI
sub-system. In addition, we ignored one single entry whose pair of knees were
improperly localized from its lateral radiograph by the BoneFinder tool. To
have more _training_ samples, we generated multiple entries from the
longitudinal record of each participant by considering imaging and non-imaging
data at different follow-up visits (except for the last one) as additional
inputs.

####  IV-A2 AD clinical status prognosis prediction

We applied our framework to forecast the Alzheimer’s disease (AD) clinical
status from multi-modal data on the Alzheimer’s Disease Neuroimaging
Initiative (ADNI) cohort, which is available at https://ida.loni.usc.edu. The
recruitment was done at $57$ sites around America and Canada, and there were
$2,577$ male and female participants from $55$ to $90$ enrolled in the cohort.
The participants underwent a series of tests such as clinical evaluation,
neuropsychological tests, genetic testing, lumbar puncture, MRI, and PET
imaging at a baseline and follow-up visits at $1$, $2$, and $4$-year periods.

TABLE I: Dataset statistics. Subjects are patient knee joints, and patient
brains for OAI, and ADNI, respectively.

Dataset | Task | # subjects
---|---|---
OAI | OA structural prognosis | 8,953
ADNI | AD clinical status prognosis | 1,307

In this study, we used raw FDG-PET scans, MRI measures, cognitive tests,
clinical history, and risk factors as predictor variables. The raw FDG-PET
scans were pre-processed by the dataset owner, and were then standardized to
voxel dimensions of $160\times 160\times 160$ ($1.5\times 1.5\times 1.5mm^{3}$
voxel spacing) using the NiBabel library [44]. To be in line with the OAI
dataset, we applied the same technique to convert scalar inputs to one-hot
encoding vectors with a length of $4$. In querying subjects, while we only
selected entries whose raw FDG-PET scans were available, the other input
variables were allowed to be missing.

Our objective was to forecast the AD clinical statuses of participants’ brains
– cognitively normal (CN), mild cognitive impairment (MCI) or probable AD – in
the next $4$ years. Since the amount of the queried data was substantially
limited (see Table I), we sampled entries from follow-up examinations to
increase the amount of training data, and performed $10$-fold cross-validation
on this task.

###  IV-B Experimental Setup

TABLE II: Input variables for forecasting knee OA severity grades. Group | Variable name | Data type
---|---|---
Raw imaging | Knee X-ray | 2D
Clinical Variables | Age | Numerical
| WOMAC | Numerical
| Sex | Categorical
| Injury | Categorical
| Surgery | Categorical
| BMI | Numerical

####  IV-B1 Implementation details

We trained and evaluated our method and the reference approaches using V100
Nvidia GPUs. Each experimental setting was performed on a single GPU with
$12$GB. We implemented all the methods using the PyTorch framework [45], and
trained each of them with the same set of configurations and hyperparameters.
For each problem, we used the Adam optimizer [46]. The learning rates of
$1e\mathrm{-}4$ and $1e\mathrm{-}5$ were set for the OA and AD-related tasks,
respectively.

To extract visual representations of 2D images, we utilized the ResNet$18$
architecture [47] whose weights were pretrained on the ImageNet dataset [48].
We used a batch size of $128$ for the knee OA experiments. Regarding 3D
images, we chose the 3D-ShuffleNet$2$ architecture because it was well-
balanced between efficiency and performance as shown in [49], which allowed us
to train each model with a batch size of $36$ on a single consumer-level GPU.
We utilized 3D-ShuffleNet$2$’s weights previously pretrained on the
Kinetics-600 dataset [50]. Moreover, we used a common feature extraction
architecture with a linear layer, a ReLU activation, and a layer normalization
[36] for all scalar numerical and categorical inputs. We provide the detailed
description of the input variables in Tables II and III.

TABLE III: Input variables for forecasting clinical statuses of Alzheimer’s Disease. See Sec. IV-E2 for acronyms. Group | Variable name | Data type
---|---|---
Raw imaging | FDG-PET | 3D
MRI measures | Hippocampus | Numerical
| Whole brain | Numerical
| Entorhinal | Numerical
| Fusiform gyrus | Numerical
| Mid. temp. gyrus | Numerical
| Intracranial vol. | Numerical
Clinical variables | Sex | Categorical
| Marriage | Categorical
| Race | Categorical
| Ethnicity | Categorical
| Education | Numerical
Cognitive tests | CDRSB | Numerical
| ADAS11 | Numerical
| MMSE | Numerical
| RAVLT | 1D
Cerebrospinal fluid | A-Beta | Numerical
| Tau | Numerical
| Ptau | Numerical
| Moca | Numerical
| Ecog | 1D
Risk factors | Apolipoprotein E4 | Numerical
| Age | Numerical

####  IV-B2 Baselines

For fair comparisons, our baselines were models that had the same feature
extraction modules for multi-modal data, as described in Section IV-B1, but
utilized different architectures to perform discrete time series forecasting.
As such, we compared our method to baselines with the forecasting module using
fully-connected network (FCN), GRU [51], LSTM [52], multi-modal transformer
(MMTF) [31], Reformer [53], Informer [54], Autoformer [55], or CLIMAT [22].
While FCN, MMTF, Reformer, Informer, Autoformer, and CLIMAT are parallel
models, GRU and LSTM are sequential approaches. Among the transformer-based
methods both versions of CLIMAT have a modular structure of transformers
rather than using a flat structure as in MMTF, Reformer, Informer, and
Autoformer.

####  IV-B3 Metrics

As data from both OAI and ADNI were imbalanced, balanced accuracy (BA) [56]
was a must metric in our experiments. As there were only $3$ classes in the AD
clinical status prognosis prediction task, we also utilized the one-vs-one
multi-class area under the ROC Curve (mAUC-ROC) [57] as another metric. To
quantitatively measure calibration, we used expected calibration error (ECE)
[58, 35]. We reported means and standard errors of each metric computed over
$5$ runs with different random seeds.

To perform analyses of the statistical significance of our results, we
utilized the two-sided Wilcoxon signed-rank test to validate the advantage of
our method compared to each baseline [59]. We equally split the test set into
$20$ subsets without overlapping patients. For such a subset, we computed
metrics averaged over $5$ random seeds per method. The statistical testing was
done patient-wise by comparing our method with every baseline individually. In
the case of the OAI dataset, for all patients, we did two rounds of hypothesis
testing: one for the left and one for the right knee, respectively.
Subsequently, we applied the Bonferroni correction to adjust the significance
thresholds for multiple comparisons ($p=0.025$ due to two knees per patient)
[60].

TABLE IV: Hyperparameter and model selection based on CV performances on the KL-based knee OA prognosis prediction task. BA∗ indicates the averages of BAs of the targets at the baseline and the first $4$ years. Depth | # of [CLS] | FFN0:T | $\lambda$ | Image rep. | BA∗ (%)
---|---|---|---|---|---
2 | 9 | Separate | 0.5 | Average pool | 59.2
4 | 59.4
6 | 59.2
4 | 1 | Common | 0.5 | Average pool | 58.3
9 | Common | 59.8
1 | Separate | 59.7
9 | Separate | 59.4
4 | 9 | Common | 0.0 | Average pool | 59.1
0.25 | 58.9
0.5 | 59.8
0.75 | 56.1
1.0 | 56.7
4 | 9 | Common | 0.5 | Average pool | 59.8
[CLS] head | 58.7
TABLE V: Effect of the consistency term on performance and calibration (K-fold cross-validation). Reported results are averages of BAs and ECEs over the first 4 years. Grading |  | JSL | JSM | AD
---|---|---|---|---
| $\lambda$ | BA | ECE | BA | ECE | BA | ECE
Without $\mathcal{L}_{\mathrm{cons}}$ | 0 | 63.2 | 1.4 | 64.7 | 8.8 | 86.8 | 7.5
With $\mathcal{L}_{\mathrm{cons}}$ | 0.25 | 63.1 | 1.5 | 64.5 | 8.7 | 86.8 | 8.3
0.50 | 64.9 | 1.8 | 65.3 | 9.6 | 87.3 | 7.9
0.75 | 64.8 | 1.8 | 65.1 | 9.5 | 87.3 | 8.1
1 | 64.6 | 1.7 | 65.2 | 9.5 | 86.7 | 9.2

###  IV-C Ablation studies

####  IV-C1 Overview

We conducted a thorough ablation study to investigate the effects of different
components in our CLIMATv2 architecture on the OAI dataset. The empirical
results are presented in Table IV and summarized in the following subsections.

####  IV-C2 Effect of the transformer P’s depth

Firstly, we searched for an optimal depth of the transformer P. The results
show that the transformer P with a depth of $4$ provides the best performance,
yielding $0.2$ % gain in averaged BA compared to depths of $2$ and $4$. The
average BA (over $4$ years) indicates a substantial boost in performance. We,
therefore, use the depth of $4$ for the transformer P in the sequel.

(a) Joint space narrowing (b) Osteophyte in femur (c) Osteophyte in tibia

Figure 6: Performance and calibration comparisons between CLUB and other
baselines. All the measures are on the medial side. The losses can be
categorized into groups: (1) FL and AFL, and (2) CE, MTL, and CLUB, which are
based on cross-entropy and focal loss, respectively.

####  IV-C3 Effect of the number of [CLS] embeddings and FFNs in the
transformer P

Then, we simultaneously validated two components: using single or multiple
[CLS] embeddings, and using common or separate FFN in the transformer P. Of
$4$ combinations of settings, the quantitative results suggest that the
transformer should have $9$ individual [CLS] embeddings, each of which
corresponds to an output head, and merely use one common FFN to make
predictions at different time points.

(a) Kellgren-Lawrence (KL) (b) Lateral joint space (JSL) (c) Medial joint
space (JSM)

(d) Lateral oste. in femur (OSFL) (e) Medial oste. in femur (OSFM) (f) Lateral
oste. in tibia (OSTL) (g) Medial oste. in tibia (OSTM)

Figure 7: Performance comparisons between our CLIMAT models and other
baselines on the knee osteoarthritis prognosis task via different types of
grading (means and standard errors over $5$ random seeds). $*$ and $**$
indicate the statistically significant differences between CLIMATv2 compared
to each baseline via Wilcoxon signed-rank tests ($p<0.05$ and $p<0.001$,
respectively). As the statistical tests were conducted on both knees, p-value
thresholds were adjusted to $0.025$ and $0.0005$, respectively.

####  IV-C4 Effect of the consistency term

To validate the necessity of the $\mathcal{L}_{\mathrm{cons}}$ term, we
conducted an experiment on a set of $\lambda$ values
$\\{0,0.25,0.5,0.75,1\\}$. The empirical evidence in Table IV shows that a
$\lambda$ of $0.5$ resulted in the best performance, which was $0.7\%$ higher
than the setting without $\mathcal{L}_{\mathrm{cons}}$. We further validated
the effects of the consistency term on other knee OA grading criteria as well
as the AD status forecasting task. The empirical results in Table V
consistently demonstrate that the term $\mathcal{L}_{\mathrm{cons}}$ has a
positive impact on performance, albeit with the trade-off of calibration. A
consistency coefficient $\lambda$ of $0.5$ is the most optimal setting in
terms of performance across the tasks. Specifically, we observed BA gains of
$1.7\%$, $0.6\%$, and $0.5\%$ with trade-off ECEs of $0.4\%$, $0.8\%$, and
$0.4\%$ for JSL, JSM, and AD, respectively.

####  IV-C5 Average pooling for image representation

In contrast to the previous version, we adopted a conventional approach used
in prior studies [38, 39, 40], which involves performing an average pooling
over the output sequence of the Radiologist block to constitute an imaging
feature vector for diagnosis prediction $\hat{y}_{0}^{R}$. According to Table
IV, such an approach results in a gain of $1.1\%$ BA compared to the baseline,
which solely utilized the first vector of the sequence generated by the block
R.

Figure 8: Calibration comparisons on the knee OA prognosis predictions. (a)
Averaged ECEs over the first 4 years. (b) Averaged ECEs over 8 years.

####  IV-C6 Multimodal channel-wise concatenation

We conducted an ablation study on the combination of multimodal embeddings. As
such, we compared our channel-wise approach to a sequence-wise baseline that
simply concatenates imaging embeddings and a projected version of non-imaging
ones. For the baseline, we utilized a linear projection layer to ensure that
imaging and non-imaging embeddings are in the same $C_{X}$-dimensional space.
We reported the K-fold cross-validation results in Table VI. On the knee OA-
related tasks, our approach tends to have positive benefits on both
performance and calibration. Specifically, the performance gains were $2.3\%$,
$2.0\%$, and $0.5\%$ for KL, JSL, and JSM, respectively. Except for JSL with
an increase of $0.1\%$ ECE, the approach results in calibration improvements
of $2.5\%$ and $3.2$ for KL and JSM, respectively. On the AD-related task, the
channel-wise approach leads to improvements of $1.1\%$ BA and $0.2\%$ ECE.

####  IV-C7 Effectiveness of CLUB loss

We compared the CLUB loss to CE itself, multi-task loss (MTL) [41], focal loss
(FL) [61], and adaptive focal loss (AFL) [62]. Whereas the first two baselines
and our loss are based on CE loss, the remaining ones are related to FL. In
Figure 6, we graphically visualize the trade-off between performance and
calibration, in which the best in both aspects are expected to locate close to
the top-left corners. We observe that our model trained with FL-related losses
was substantially worse calibrated compared to the settings with any CE-based
loss. Among the losses based on CE, the proposed CLUB helped our model to
achieve the best ECEs in all three OA grading systems with insubstantial drops
in performance.

TABLE VI: Ablation study on imaging and non-imaging combination with k-fold cross-validation ($K=5$ and $K=10$ for OAI and ADNI, respectively). Channel-wise approach (ours) is compared to the sequence-wise approach, concatenating imaging embeddings produced by the block R with projected non-imaging embeddings outputted by the block C. Reported results are averaged BAs and ECEs over the first $4$ years. Grading | Setting | BA (%, $\uparrow$) | ECE (%, $\downarrow$)
---|---|---|---
KL | Channel-wise | 59.8 | 14.7
| Sequence-wise | 57.5 | 17.2
JSL | Channel-wise | 64.9 | 1.8
| Sequence-wise | 62.9 | 1.7
JSM | Channel-wise | 65.3 | 9.6
| Sequence-wise | 64.7 | 12.8
AD | Channel-wise | 87.3 | 7.9
| Sequence-wise | 86.2 | 8.1

###  IV-D Performance and Calibration Comparisons to Competitive Baselines

####  IV-D1 Knee OA structural prognosis prediction

In Figures 7 and 8, we graphically present comparisons between both versions
of CLIMAT and the baselines in the $7$ different knee OA grading scales.

In general, both versions of CLIMAT outperformed the other baselines in
forecasting the knee OA progression within the first $4$ years across the knee
OA grading systems. That is consistent with the observation of [22] in KL. We
observe that LSTM is the most competitive baseline across the grading systems.
Compared to LSTM, on average of the first $4$ years, our model achieved
$1.3\%$, $1.7\%$, $1.9\%$, $2.3\%$, $0.7\%$, $4.4\%$, and $2.4\%$ higher BAs
while having $1.9\%$, $0.1\%$, $0.02\%$, $6.5\%$, $3.6\%$, $1.4\%$, and
$4.1\%$ lower ECEs in KL, JSL, JSM, OSFL, OSFM, OSTL, and OSTM, respectively.
On average, Informer was the most competitive transformer-based baseline.
Compared to Informer, CLIMATv2 achieved BA improvements of
$0.4\%,0.3\%,1.6\%,0.6\%,0.4\%,1.2\%$, and $1.8\%$ in KL, JSL, JSM, OSFL,
OSFM, OSTL, and OSTM, respectively. Except for KL and OSFM, our model had
lower ECEs with differences of $0.3\%,0.1\%,1.5\%,1.2\%$, and $0.4\%$ in JSL,
JSM, OSFL, OSTL, and OSTM, respectively. Moreover, in comparison to CLIMATv1
[22], on average for the first $4$ years, the newer version performed better
in KL, JSM, OSFL, and OSTL with BA improvements of $0.1\%$, $0.4\%$, $0.5\%$,
and $0.2\%$, respectively, whereas it reached $0.1\%$, $0.5\%$, and $0.3\%$
lower BAs in JSL, OSFM, and OSTM, respectively. Regarding the calibration
aspect, CLIMATv2 obtained lower ECEs compared to CLIMATv1 in JSL, JSM, OSFM,
and OSTM with differences of $0.1\%$, $0.3\%$, $0.2\%$, and $0.4\%$,
respectively.

####  IV-D2 Alzheimer’s disease status prognosis prediction

We reported the quantitative results in Table VII. Regarding performance, both
the CLIMAT methods achieved the best performances across the prediction
targets, in which CLIMATv2 was top-$1$ at the first $2$ years in both BA and
mROCAUC. Compared to the transformer-based baseline MMTF, our method
outperformed by $2.2\%$, $1.8\%$, and $2.2\%$ BAs at years $1$, $2$, and $4$,
respectively. In calibration, CLIMATv2 yielded substantially lower ECEs than
all the references at every prediction target. That observation was supported
by the statistical test results in Table VII.

TABLE VII: CV performance and calibration comparisons on the ADNI data (mean
and standard errors over $5$ random seeds). The best performances with and
without substantial differences are indicated by bold and underlined values,
respectively. The substantial improvement is determined by whether the best
performance overlaps with any other method’s. $*$ and $**$ indicate the
statistically significant differences between CLIMATv2 vs. each baseline via
Wilcoxon signed-rank tests (p ¡ 0.05 and p ¡ 0.001, respectively).

Year | Method |  BA (%, $\uparrow$) |  mROCAUC (%, $\uparrow$) |  ECE (%, $\downarrow$)
---|---|---|---|---
1 | FCN | 87.6$\pm$0.2∗∗ | 96.6$\pm$0.1∗∗ | 9.5$\pm$0.3∗∗
GRU | 87.1$\pm$0.3∗∗ | 96.6$\pm$0.1∗∗ | 8.6$\pm$0.4∗
LSTM | 87.9$\pm$0.2∗∗ | 96.8$\pm$0.1∗∗ | 8.7$\pm$0.3∗∗
MMTF | 88.2$\pm$1.0 | 96.4$\pm$0.8∗ | 23.3$\pm$0.4∗∗
Reformer | 80.8$\pm$1.0∗∗ | 93.6$\pm$0.7∗∗ | 9.6$\pm$0.3∗∗
Informer | 78.4$\pm$1.5∗∗ | 93.1$\pm$0.6∗∗ | 10.0$\pm$0.4∗∗
Autoformer | 85.2$\pm$0.3∗∗ | 96.0$\pm$0.1∗∗ | 7.8$\pm$0.2∗
CLIMATv1 | 90.1$\pm$0.1 | 97.6$\pm$0.1∗ | 6.7$\pm$0.2
CLIMATv2 | 90.4$\pm$0.1 | 98.0$\pm$0.1 | 6.5$\pm$0.1
2 | FCN | 85.5$\pm$0.2∗ | 95.7$\pm$0.1∗ | 9.2$\pm$0.2∗
GRU | 85.1$\pm$0.2∗ | 95.6$\pm$0.1∗ | 9.7$\pm$0.2∗
LSTM | 85.7$\pm$0.3∗ | 95.7$\pm$0.2∗ | 9.3$\pm$0.7
MMTF | 85.4$\pm$0.6 | 95.3$\pm$0.7 | 22.3$\pm$0.4∗∗
Reformer | 78.8$\pm$0.9∗∗ | 92.5$\pm$0.6∗∗ | 9.2$\pm$0.2∗
Informer | 78.1$\pm$1.3∗∗ | 92.9$\pm$0.4∗∗ | 8.6$\pm$0.1
Autoformer | 83.7$\pm$0.3∗∗ | 95.2$\pm$0.1∗∗ | 8.6$\pm$0.2
CLIMATv1 | 87.2$\pm$0.0 | 96.6$\pm$0.0 | 9.1$\pm$0.2∗
CLIMATv2 | 87.2$\pm$0.2 | 96.6$\pm$0.1 | 7.9$\pm$0.3
4 | FCN | 80.7$\pm$0.3∗ | 93.7$\pm$0.1 | 9.6$\pm$0.3∗
GRU | 81.7$\pm$0.1 | 93.8$\pm$0.1 | 12.6$\pm$0.4∗∗
LSTM | 81.0$\pm$0.7∗ | 93.5$\pm$0.2 | 12.4$\pm$0.6∗∗
MMTF | 80.6$\pm$0.7 | 93.0$\pm$0.6 | 17.9$\pm$0.3∗∗
Reformer | 76.7$\pm$0.9∗∗ | 90.5$\pm$0.5∗∗ | 11.1$\pm$0.5∗
Informer | 71.6$\pm$0.9∗∗ | 87.4$\pm$0.3∗∗ | 12.6$\pm$0.3∗∗
Autoformer | 80.5$\pm$0.3∗∗ | 93.3$\pm$0.1∗∗ | 9.5$\pm$0.2
CLIMATv1 | 83.0$\pm$0.2 | 94.5$\pm$0.1 | 9.6$\pm$0.3∗
CLIMATv2 | 82.8$\pm$0.1 | 94.2$\pm$0.1 | 9.2$\pm$0.2

###  IV-E Attention maps over multiple modalities

The self-attention mechanism of the transformers in CLIMATv2 allowed us to
visualize attention maps over imaging and non-imaging modalities when our
model made a prediction at a specific target. Specifically, we used
$\mathrm{Softmax}\left(\bm{Q}_{L}\bm{K}_{L}^{\intercal}/\sqrt{d_{k}}\right)$,
where $\bm{Q}_{L},\bm{K}_{L}$ are query and key matrices of the last layer
$L$, respectively, and $d_{k}$ is the feature dimension of the key matrix, as
attention maps [25]. While we utilized the softmax output corresponding to
$\bar{h}_{C}^{0}$ in the transformer F for clinical variables, we took the
softmax output in computing $\bar{h}_{P}^{t}$ with $t=0,\dots,T$ in the
transformer P to visualize attention maps on imaging modalities. Here, we set
$t=1$, corresponding to the forecast of a disease severity $1$ year from the
baseline.

####  IV-E1 Knee OA structural prognosis

In Figure 9, we visualized attention maps over different input modalities
across $7$ grading criteria. As such, in Figure 9a, we displayed a healthy
knee at the baseline overlaid by $7$ corresponding saliency maps. For
differentiation, we also provided colored ellipses. Figure 9b shows the
heatmap over the $6$ clinical variables on each grading criterion. Values on
each row sum up to $1$. In this particular case, we observe that the model has
paid the most attention to the intercondylar notch, together with BMI and
WOMAC [63].

####  IV-E2 AD clinical status prognosis

As imaging data consisted of 3D FDG-PET scans as well as the other imaging
measurements, we had to separate them into Figures 10a and 10b. We can observe
that an attention sphere locates around the posterior cingulate cortex, the
inferior frontal gyrus, and the middle gyrus [64]. Figure 10b shows
accumulated attention weights corresponding to the FDG-PET feature vectors
alongside ones of the other imaging measurements. The reason that imaging
variables were assigned a substantially higher importance is that the number
of the 3D visual embeddings was dominant compared to the others (i.e. $125$
versus $6$). In Figure 10c, high attention can be observed on the percent
forgetting score of the Rey Auditory Verbal Learning Test (RAVLT), RAVLT
immediate, the AD assessment score 11-item (ADAS11), Clinical Dementia Rating
Scale–Sum of Boxes (CDRSB), Mini-Mental State Exam (MMSE), and Functional
Activities Questionnaire (FAQ).

##  V Conclusions

In this paper, we proposed a novel general-purpose transformer-based method to
forecast the trajectory of a disease’s stage from multimodel data. We applied
our method to two real-world applications, that are related to OA and AD. Our
framework provides tools to integrate multi-modal data and has interpretation
capabilities through self-attention.

In comparison with the prior version, CLIMATv2 has two primary upgrades.
First, we have eliminated the assumption of independence between non-imaging
data $m_{0}$ and diagnostic predictions $y_{0}$ used in CLIMATv1 [22] since it
does not hold not only in OA and AD, but also in other diseases. Specifically,
Liu et al. [34] provided empirical evidence of the benefit of the inclusion of
non-imaging data in the knee OA grading task. The study conducted by Bird et
al. [65] indicated a link between human genes and AD while Li et al. [66]
showed that a blood test can detect the existence of amyloid-beta plaques in
the human brain, which is strongly associated with AD status. Second, we have
proposed the CLUB loss, which allowed us to optimize for both performance and
calibration.

There are some limitations in this study, which are worth mentioning. First,
we used common DL architectures as imaging and non-imaging feature extractors.
While such a standardized procedure resulted in fair comparisons, better
results could have been obtained with e.g. Neural Architecture Search methods
[67]. Furthermore, a wider range of DL modules could have been considered, but
this could substantially increase the use of computing resources.
Specifically, to obtain results in this work, it required roughly 400 GPU
hours for experiments in Table VII and 525 GPU hours in Figure 7 for every
method, respectively.

The second limitation of the present study, is that attention maps produced by
transformers act as human-friendly signals of our model, and should be
carefully used in practice with expert knowledge in the domain. Transformers
may highlight areas not associated with the body part, which can be seen in
Figure 11 as well as in other studies [68, 69, 70].

Lastly, we primarily utilized the transformer proposed by [27]. More efficient
and advanced transformers such as [53, 54, 55, 71] could be further
investigated to integrate into the framework.

To conclude, to our knowledge, this is not only the first study in the realm
of OA, but also the first work on AD clinical status prognosis prediction from
the multi-modal setup that includes raw 3D scans and scalar variables. The
developed method can be of interest to other fields, where forecasting of
calibrated disease trajectory is of interest. An implementation of our method
is made publicly available at https://github.com/Oulu-IMEDS/CLIMATv2.

##  VI Acknowledgement

The OAI is a public-private partnership comprised of five contracts (N01-
AR-2-2258; N01-AR-2-2259; N01-AR-2- 2260; N01-AR-2-2261; N01-AR-2-2262) funded
by the National Institutes of Health, a branch of the Department of Health and
Human Services, and conducted by the OAI Study Investigators. Private funding
partners include Merck Research Laboratories; Novartis Pharmaceuticals
Corporation, GlaxoSmithKline; and Pfizer, Inc. Private sector funding for the
OAI is managed by the Foundation for the National Institutes of Health.

Data collection and sharing for this project was funded by the Alzheimer’s
Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant
U01 AG024904) and DOD ADNI (Department of Defense award number
W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the
National Institute of Biomedical Imaging and Bioengineering, and through
generous contributions from the following: AbbVie, Alzheimer’s Association;
Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.;
Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.;
Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La
Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE
Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research &
Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development
LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.;
NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals
Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical
Company; and Transition Therapeutics. The Canadian Institutes of Health
Research is providing funds to support ADNI clinical sites in Canada. Private
sector contributions are facilitated by the Foundation for the National
Institutes of Health (www.fnih.org). The grantee organization is the Northern
California Institute for Research and Education, and the study is coordinated
by the Alzheimer’s Therapeutic Research Institute at the University of
Southern California. ADNI data are disseminated by the Laboratory for Neuro
Imaging at the University of Southern California.

The authors wish to acknowledge CSC – IT Center for Science, Finland, for
generous computational resources.

We would like to acknowledge the strategic funding of the University of Oulu,
Infotech Oulu Focus Institute, and Sigrid Juselius Foundation, Finland. The
publication was also supported by funding from the Academy of Finland (Profi6
336449 funding program). We also acknowledge funding from the Alfred
Kordelinin foundation (220211).

We acknowledge funding from the Flemish Government under the
“Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

Dr. Claudia Lindner is acknowledged for providing BoneFinder. Phuoc Dat Nguyen
is acknowledged for discussions about transformers.

(a) A knee with attention maps

(b) Contributions of the variables

Figure 9: An example of progression from a healthy knee at baseline to early
osteoarthritis. Our model identified the changes in the intercondylar notch,
sex, and symptomatic status.

(a) Attention maps on an axial FDG-PET slice

(b) FDG-PET and imaging variables

(c) Importances of other variables

Figure 10: Interpretability of our method’s prediction on a selective sample
from the ADNI dataset.

(a) Cases with attention maps primarily overlapping with brain regions

(b) Cases with attention maps overlapping with both brain and background
regions

Figure 11: Attention maps on axial FDG-PET slices. Three axial slices on each
row belong to the same PET scan.

## References

  * [1] R. J. Desai, S. V. Wang, M. Vaduganathan, T. Evers, and S. Schneeweiss, “Comparison of machine learning methods with traditional models for use of administrative claims with electronic medical records to predict heart failure outcomes,” _JAMA network open_ , vol. 3, no. 1, pp. e1 918 962–e1 918 962, 2020.
  * [2] A. Boutet, R. Madhavan, G. J. Elias, S. E. Joel, R. Gramer, M. Ranjan, V. Paramanandam, D. Xu, J. Germann, A. Loh _et al._ , “Predicting optimal deep brain stimulation parameters for parkinson’s disease using functional mri and machine learning,” _Nature communications_ , vol. 12, no. 1, pp. 1–13, 2021.
  * [3] A. Cheerla and O. Gevaert, “Deep learning with multimodal representation for pancancer prognosis prediction,” _Bioinformatics_ , vol. 35, no. 14, pp. i446–i454, 2019.
  * [4] K. A. Tran, O. Kondrashova, A. Bradley, E. D. Williams, J. V. Pearson, and N. Waddell, “Deep learning in cancer diagnosis, prognosis and treatment selection,” _Genome Medicine_ , vol. 13, no. 1, pp. 1–17, 2021.
  * [5] P. Milanez-Almeida, A. J. Martins, R. N. Germain, and J. S. Tsang, “Cancer prognosis with shallow tumor rna sequencing,” _Nature Medicine_ , vol. 26, no. 2, pp. 188–192, 2020.
  * [6] X. Mei, H.-C. Lee, K.-y. Diao, M. Huang, B. Lin, C. Liu, Z. Xie, Y. Ma, P. M. Robson, M. Chung _et al._ , “Artificial intelligence–enabled rapid diagnosis of patients with covid-19,” _Nature medicine_ , vol. 26, no. 8, pp. 1224–1228, 2020.
  * [7] S. Glyn-Jones, A. Palmer, R. Agricola, A. Price, T. Vincent, H. Weinans, and A. Carr, “Osteoarthritis,” _The Lancet_ , vol. 386, no. 9991, pp. 376–387, 2015.
  * [8] “2021 alzheimer’s disease facts and figures,” _Alzheimer’s & Dementia_, vol. 17, no. 3, pp. 327–406, 2021.
  * [9] B. Heidari, “Knee osteoarthritis prevalence, risk factors, pathogenesis and features: Part i,” _Caspian journal of internal medicine_ , vol. 2, no. 2, p. 205, 2011.
  * [10] L. S. Lee, P. K. Chan, W. C. Fung, V. W. K. Chan, C. H. Yan, and K. Y. Chiu, “Imaging of knee osteoarthritis: A review of current evidence and clinical guidelines,” _Musculoskeletal Care_ , vol. 19, no. 3, pp. 363–374, 2021\.
  * [11] J. Kellgren and J. Lawrence, “Radiological assessment of osteo-arthrosis,” _Annals of the rheumatic diseases_ , vol. 16, no. 4, p. 494, 1957.
  * [12] R. D. Altman and G. Gold, “Atlas of individual radiographic features in osteoarthritis, revised,” _Osteoarthritis and cartilage_ , vol. 15, pp. A1–A56, 2007.
  * [13] E. Rezuş, A. Burlui, A. Cardoneanu, L. A. Macovei, B. I. Tamba, and C. Rezuş, “From pathogenesis to therapy in knee osteoarthritis: bench-to-bedside,” _International Journal of Molecular Sciences_ , vol. 22, no. 5, p. 2697, 2021.
  * [14] J. Rasmussen and H. Langerman, “Alzheimer’s disease–why we need early diagnosis,” _Degenerative neurological and neuromuscular disease_ , vol. 9, p. 123, 2019.
  * [15] A. Tiulpin, S. Klein, S. M. Bierma-Zeinstra, J. Thevenot, E. Rahtu, J. van Meurs, E. H. Oei, and S. Saarakkala, “Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data,” _Scientific reports_ , vol. 9, no. 1, pp. 1–11, 2019.
  * [16] P. Widera, P. M. Welsing, C. Ladel, J. Loughlin, F. P. Lafeber, F. P. Dop, J. Larkin, H. Weinans, A. Mobasheri, and J. Bacardit, “Multi-classifier prediction of knee osteoarthritis progression from incomplete imbalanced longitudinal data,” _Scientific Reports_ , vol. 10, no. 1, pp. 1–15, 2020\.
  * [17] B. Guan, F. Liu, A. Haj-Mirzaian, S. Demehri, A. Samsonov, T. Neogi, A. Guermazi, and R. Kijowski, “Deep learning risk assessment models for predicting progression of radiographic medial joint space loss over a 48-month follow-up period,” _Osteoarthritis and cartilage_ , vol. 28, no. 4, pp. 428–437, 2020.
  * [18] K. Leung, B. Zhang, J. Tan, Y. Shen, K. J. Geras, J. S. Babb, K. Cho, G. Chang, and C. M. Deniz, “Prediction of total knee replacement and diagnosis of osteoarthritis by using deep learning on knee radiographs: data from the osteoarthritis initiative,” _Radiology_ , vol. 296, no. 3, pp. 584–593, 2020\.
  * [19] A. A. Tolpadi, J. J. Lee, V. Pedoia, and S. Majumdar, “Deep learning predicts total knee replacement from magnetic resonance images,” _Scientific reports_ , vol. 10, no. 1, pp. 1–12, 2020.
  * [20] W. Jung, A. W. Mulyadi, and H.-I. Suk, “Unified modeling of imputation, forecasting, and prediction for ad progression,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 168–176.
  * [21] D. Lu, K. Popuri, G. W. Ding, R. Balachandar, and M. F. Beg, “Multimodal and multiscale deep neural networks for the early diagnosis of alzheimer’s disease using structural mr and fdg-pet images,” _Scientific reports_ , vol. 8, no. 1, pp. 1–13, 2018.
  * [22] H. H. Nguyen, S. Saarakkala, M. B. Blaschko, and A. Tiulpin, “Climat: Clinically-inspired multi-agent transformers for knee osteoarthritis trajectory forecasting,” in _2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2022, pp. 1–5.
  * [23] M. M. Ghazi, M. Nielsen, A. Pai, M. J. Cardoso, M. Modat, S. Ourselin, L. Sørensen, A. D. N. Initiative _et al._ , “Training recurrent neural networks robust to incomplete data: application to alzheimer’s disease progression modeling,” _Medical image analysis_ , vol. 53, pp. 39–46, 2019.
  * [24] J. Albright, A. D. N. Initiative _et al._ , “Forecasting the progression of alzheimer’s disease using neural networks and a novel preprocessing algorithm,” _Alzheimer’s & Dementia: Translational Research & Clinical Interventions_, vol. 5, pp. 483–491, 2019.
  * [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [27] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , vol. 1, 2020.
  * [28] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, “Video action transformer network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 244–253.
  * [29] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, “Vivit: A video vision transformer,” _arXiv preprint arXiv:2103.15691_ , 2021.
  * [30] A. Hassani, S. Walton, N. Shah, A. Abuduweili, J. Li, and H. Shi, “Escaping the big data paradigm with compact transformers,” _arXiv preprint arXiv:2104.05704_ , 2021.
  * [31] S. Hu, E. Fridgeirsson, G. van Wingen, and M. Welling, “Transformer-based deep survival analysis,” in _Survival Prediction-Algorithms, Challenges and Applications_. PMLR, 2021, pp. 132–148.
  * [32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [33] L. Jans, J. Bosmans, K. Verstraete, and R. Achten, “Optimizing communication between the radiologist and the general practitioner,” _JBR-BTR_ , vol. 96, no. 6, pp. 388–390, 2013.
  * [34] L. Liu, J. Chang, P. Zhang, Q. Ma, H. Zhang, T. Sun, and H. Qiao, “A joint multi-modal learning method for early-stage knee osteoarthritis disease classification,” _Heliyon_ , vol. 9, no. 4, 2023.
  * [35] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1321–1330.
  * [36] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [37] D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” 2020, arXiv:1606.08415.
  * [38] X. Chu, Z. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen, “Conditional positional encodings for vision transformers,” _arXiv preprint arXiv:2102.10882_ , 2021.
  * [39] Z. Pan, B. Zhuang, J. Liu, H. He, and J. Cai, “Scalable vision transformers with hierarchical pooling,” in _Proceedings of the IEEE/cvf international conference on computer vision_ , 2021, pp. 377–386.
  * [40] N. Park and S. Kim, “How do vision transformers work?” _arXiv preprint arXiv:2202.06709_ , 2022.
  * [41] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7482–7491.
  * [42] A. Tiulpin and S. Saarakkala, “Automatic grading of individual knee osteoarthritis features in plain radiographs using deep convolutional neural networks,” _Diagnostics_ , vol. 10, no. 11, p. 932, 2020.
  * [43] C. Lindner, S. Thiagarajah, J. M. Wilkinson, G. A. Wallis, T. F. Cootes, arcOGEN Consortium _et al._ , “Fully automatic segmentation of the proximal femur using random forest regression voting,” _IEEE transactions on medical imaging_ , vol. 32, no. 8, pp. 1462–1472, 2013.
  * [44] M. Brett, C. J. Markiewicz, M. Hanke, M.-A. Côté, B. Cipollini, P. McCarthy, D. Jarecka, C. Cheng, Y. Halchenko, M. Cottaar _et al._ , “nipy/nibabel: 3.2. 1,” _Zenodo_ , 2020.
  * [45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _arXiv preprint arXiv:1912.01703_ , 2019.
  * [46] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [47] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [48] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [49] O. Kopuklu, N. Kose, A. Gunduz, and G. Rigoll, “Resource efficient 3d convolutional neural networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [50] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6299–6308.
  * [51] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” _arXiv preprint arXiv:1409.1259_ , 2014.
  * [52] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [53] N. Kitaev, Ł. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” _arXiv preprint arXiv:2001.04451_ , 2020.
  * [54] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 35, no. 12, 2021, pp. 11 106–11 115.
  * [55] M. Chen, H. Peng, J. Fu, and H. Ling, “Autoformer: Searching transformers for visual recognition,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 12 270–12 280.
  * [56] K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann, “The balanced accuracy and its posterior distribution,” in _2010 20th international conference on pattern recognition_. IEEE, 2010, pp. 3121–3124.
  * [57] D. J. Hand and R. J. Till, “A simple generalisation of the area under the roc curve for multiple class classification problems,” _Machine learning_ , vol. 45, no. 2, pp. 171–186, 2001.
  * [58] M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in _Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015.
  * [59] F. Wilcoxon, “Individual comparisons by ranking methods,” in _Breakthroughs in statistics_. Springer, 1992, pp. 196–202.
  * [60] O. J. Dunn, “Multiple comparisons among means,” _Journal of the American statistical association_ , vol. 56, no. 293, pp. 52–64, 1961.
  * [61] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [62] J. Mukhoti, V. Kulharia, A. Sanyal, S. Golodetz, P. H. Torr, and P. K. Dokania, “Calibrating deep neural networks using focal loss,” _arXiv preprint arXiv:2002.09437_ , 2020.
  * [63] H. O. León, C. E. R. Blanco, T. B. Guthrie, and O. J. N. Martínez, “Intercondylar notch stenosis in degenerative arthritis of the knee,” _Arthroscopy: The Journal of Arthroscopic & Related Surgery_, vol. 21, no. 3, pp. 294–302, 2005.
  * [64] B. Hallam, J. Chan, S. G. Costafreda, R. Bhome, and J. Huntley, “What are the neural correlates of meta-cognition and anosognosia in alzheimer’s disease? a systematic review,” _Neurobiology of aging_ , vol. 94, pp. 250–264, 2020\.
  * [65] T. D. Bird, “Genetic factors in alzheimer’s disease,” _New England Journal of Medicine_ , vol. 352, no. 9, pp. 862–864, 2005.
  * [66] Y. Li, S. E. Schindler, J. G. Bollinger, V. Ovod, K. G. Mawuenyega, M. W. Weiner, L. M. Shaw, C. L. Masters, C. J. Fowler, J. Q. Trojanowski _et al._ , “Validation of plasma amyloid-$\beta$ 42/40 for detecting alzheimer disease amyloid plaques,” _Neurology_ , vol. 98, no. 7, pp. e688–e699, 2022.
  * [67] T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” _The Journal of Machine Learning Research_ , vol. 20, no. 1, pp. 1997–2017, 2019.
  * [68] O. Petit, N. Thome, C. Rambour, L. Themyr, T. Collins, and L. Soler, “U-net transformer: Self and cross attention for medical image segmentation,” in _International Workshop on Machine Learning in Medical Imaging_. Springer, 2021, pp. 267–276.
  * [69] M. Odusami, R. Maskeliūnas, and R. Damaševičius, “An intelligent system for early recognition of alzheimer’s disease using neuroimaging,” _Sensors_ , vol. 22, no. 3, p. 740, 2022.
  * [70] Y. Rao, G. Chen, J. Lu, and J. Zhou, “Counterfactual attention learning for fine-grained visual categorization and re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1025–1034.
  * [71] X. Yang, H. Zhang, G. Qi, and J. Cai, “Causal attention for vision-language tasks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 9847–9857.