# Real-Time Systems Optimization with Black-box Constraints and Hybrid
Variables

Sen Wang1, Dong Li1, Shao-Yu Huang2, Xuanliang Deng1, Ashrarul H. Sifat1,
Changhee Jung2, Ryan Williams1, Haibo Zeng1 For any inquiries or questions,
please contact $\\{$swang666<EMAIL_ADDRESS>1Virginia Tech, 2Purdue
University

###### Abstract

When optimizing real-time systems, designers often face a challenging problem
where the schedulability constraints are non-convex, non-continuous, or lack
an analytical form to understand their properties. Although the optimization
framework NORTH proposed in [1] is general (it works with arbitrary
schedulability analysis) and scalable, it can only handle problems with
continuous variables, which limits its application. In this paper, we extend
the applications of the framework NORTH to problems with a hybrid of
continuous and discrete variables. This is achieved in a coordinate-descent
method, where the continuous and discrete variables are optimized separately
during iterations. The new framework, NORTH+, improves around 20% solution
quality than NORTH in experiments.

##  I Introduction

Over the years, the real-time systems community has developed an impressive
list of scheduling algorithms and schedulability analysis techniques. However,
many schedulability analyses are not friendly for optimization purposes. Some
analysis methods are complicated and do not provide closed-form expressions,
such as those based on demand bound functions [2], real-time calculus [3],
abstract event model interfaces [4], or timed automata [5, 6].

Another challenge in optimizing real-time systems is their ever-increasing
complexity. The functionality of real-time systems such as automotive and
aircraft is growing fast, especially given the recent trend of introducing
autonomous features and system-to-system connectivity [7, 8]. The existing
optimization algorithms do not scale to the hundreds of software tasks that
modern automotive systems contain [9].

The existing approaches for optimizing real-time systems do not adequately
address the above two challenges as they lack applicability or scalability.
Besides, many standard mathematical optimization frameworks cannot efficiently
handle complicated schedulability analysis constraints. Therefore, in recent
work [1], Wang et al. introduced NORTH (Numerical Optimizer with Real-Time
Highlight), an optimization framework for optimizing systems with black-box
schedulability constraints based on numerical optimization algorithms.
However, the application of NORTH is limited to systems with only continuous
variables.

In this paper, we extend NORTH [1] to handle problems with hybrid (i.e., a
mixture of continuous and discrete) variables. The new framework, named
NORTH+, achieves this extension based on an iterative algorithm. During the
iterations, the continuous and discrete variables are optimized independently.
The continuous optimization component remains the same as NORTH, while the
discrete optimization can leverage heuristics to ensure algorithm scalability.
Simulated experiments indicate that NORTH+ brings around 20$\%$ performance
improvements compared with NORTH [1].

##  II Problem description

Generally speaking, real-time systems design can be mathematically described
as an optimization problem:

$\displaystyle\min\limits_{\textbf{x}}\ \ \ \ \ \ $
$\displaystyle\mathcal{F}(\textbf{x})$ (1) subject to
$\displaystyle\text{Sched}(\textbf{x})=0$ (2)

where the vector of design variables
$\textbf{x}=\\{\textbf{x}^{\mathcal{C}},\textbf{x}^{\mathcal{D}}\\}$ includes
continuous variables $\textbf{x}^{\mathcal{C}}$ and discrete variables
$\textbf{x}^{\mathcal{D}}$. It represents the design choices, such as the task
execution times, periods, and priority assignments. $\mathcal{F}(\textbf{x})$
maps the design variables x into the design objective (a scalar). In addition,
real-time system design is subject to the schedulability analysis constraint:

$\text{Sched}(\textbf{x})=\begin{cases}0,&\text{system is schedulable}\\\
1,&\text{otherwise}\end{cases}$ (3)

Here, we treat the schedulability analysis as a black-box that only returns
binary results, i.e., schedulable or not.

##  III Methdology

The overview of NORTH+ is shown in Fig. 1.

Figure 1: NORTH+ is an extension of the NORTH optimization framework [1]. The
NMBO and VE components are shown in Fig. 2 and explained more in Section
III-A, the discrete optimization is introduced in Section III-B. The
continuous and discrete variables in NORTH+ are optimized separately: when
optimizing continuous variables, the discrete variables are treated as
constants, and vice versa.  Figure 2: Optimization framework NORTH from [1]
that optimizes only continuous variables.

###  III-A Performing continuous optimization

We briefly review [1] in this section, which forms the foundation of this
paper. The framework of [1] is shown in Fig. 2, which has two major steps
during iterations:

  * •

Numerical Method-Based Optimization (NMBO): an optimization method based on
trust-region optimizers [10]. Given an initial solution x, the algorithm first
performs unconstrained optimization to find a descent direction
$\boldsymbol{\Delta}$ and then adjusts $\boldsymbol{\Delta}$ to keep
$\textbf{x}+\boldsymbol{\Delta}$ feasible. It terminates when
$\|\boldsymbol{\Delta}\|_{2}$ becomes smaller than a given threshold.

  * •

Variable Elimination (VE): After NMBO terminates, we try to find a feasible
descent direction (i.e., we can take a big step along this direction without
violating the schedulability constraints) by searching within the sub-space of
the $\boldsymbol{\Delta}$ above. When $\textbf{x}+\boldsymbol{\Delta}$
violates the schedulability constraints, it is often the case that only a
small subset of tasks miss their deadline. Therefore, excluding these tasks
from further optimization frees the numerical optimizers from schedulability
constraints and improves performance in future iterations.

  * •

Formulate new initial solution: Eliminated variables will be considered
constant in subsequent iterations, and a new initial solution will be
formulated based solely on the remaining variables.

###  III-B Performing discrete optimization

Although the primary goal of NORTH+ is optimizing the objective function (1),
enhancing system schedulability can be regarded as a secondary goal. When
NORTH optimizes the continuous variables, it tends to push them towards the
boundaries of schedulable regions. An example is shown in Fig. 3. Thus, some
variables are often close to violating the schedulability constraints when the
numerical optimizer terminates. This situation prevents further optimization
opportunities.

Figure 3:  From [1], an example that visualizes how NORTH optimizes the
continuous variables C in the figure. Expanding the schedulable region at
$\textbf{C}^{(k)}$ gives NORTH more chances of optimization.

Therefore, we can broaden the system’s schedulability region to improve
performance. For example, when the discrete variables are priority
assignments, many classical algorithms can be utilized to assign priorities,
such as Rate Monotonic (RM) [11, 12] or DkC [13]. These algorithms should be
executed within each iteration, as illustrated in Fig. 1, as optimal priority
assignments may change as the continuous variables are optimized.

((a)) Relative performance gap

((b)) Run-time speed

Figure 4: Performance comparison between NORTH and NORTH+RM.

##  IV Experiments

In the experiments, we consider an example application where the objective
function is derived from control performance optimization [14]:

$\displaystyle\min_{\textbf{T},\textbf{P}}\sum_{i=0}^{N-1}(\alpha_{i}\textbf{T}_{i}+\beta_{i}r_{i}(\textbf{T},\textbf{P}))$
(4)

where the variables $\textbf{T}_{i}$ are the task $\tau_{i}$’s periods; $N$
dentoes the number of tasks; $\alpha_{i}$ and $\beta_{i}$ are control weight
parameters; P are the priority assignments for all the tasks;
$r_{i}(\textbf{T},\textbf{P})$ is the task $\tau_{i}$’s response time.

We evaluate the framework on simulated task sets. Ten thousand task sets were
generated randomly following [14, 1]. The task sets consisted of periodic
tasks $\boldsymbol{\tau}$ with implicit deadlines scheduled by fixed-task
priority schedulers on a uni-processor platform. The worst-case execution time
$C_{i}$ of each task $\tau_{i}$ was generated randomly from [1, 100]; the task
periods were upper-bounded by $T_{i}\leq 5\sum_{i=1}^{n}C_{i}$. In the
objective function (4), $\alpha_{i}$ was randomly generated in the range [1,
1000]; $\beta_{i}$ was generated in the range [1, 10000].

In the experiments, the black-box schedulability analysis constraint follows
the classical response time analysis [15]:

$r_{i}=C_{i}+\sum_{j\in\text{hp}(i)}\lceil\frac{r_{i}}{T_{j}}\rceil{C_{j}}$
(5)

where $\text{hp}(i)$ denotes the tasks with higher priority than the task
$\tau_{i}$. Let $D_{i}$ denote the deadline of the task $\tau_{i}$, we have:

$\text{Sched}(\textbf{T},\textbf{P})=\begin{cases}0,&\forall
i,r_{i}(\textbf{T},\textbf{P})\leq D_{i}\\\ 1,&\text{otherwise}\end{cases}$
(6)

We compared the performance between NORTH [1] and NORTH+RM because they work
with black-box schedulability constraints. In NORTH+RM, task priorities are
adjusted with RM each time after NORTH optimizes the period variables, as
illustrated in Fig. 1. Fig. 4 shows experiment results, where the relative gap
is defined as follows:

$\frac{\mathcal{F}_{\text{NORTH+RM}}-\mathcal{F}_{\text{NORTH}}}{\mathcal{F}_{\text{NORTH}}}\times
100\%$ (7)

##  V Discussion, Open Challenges and Future Work

Experiments show that adding an extra discrete optimization step to NORTH
improves applicability and performance. Although finding the optimal solutions
for the discrete optimization step in NORTH+ could be difficult, simple
heuristics may work reasonably well in practice. In simulated experiments,
assigning priorities based on RM brought around 20$\%$ performance
improvements.

###  V-A Challenge: Optimality vs Applicability and Efficiency

Fig. 4(a) shows that NORTH+RM does not consistently outperform NORTH. NORTH+
is not designed to provide optimal solutions. Given a real-time system
optimization problem with possibly black-box constraints and many variables,
NORTH+ can provide a solution quickly with reasonably good quality, albeit
without a guarantee of optimality.

Providing an optimality guarantee for problems with black-box constraints is
challenging. These black-box constraints could be non-convex and non-
continuous, and cause the optimal solutions, denoted as $\textbf{x}^{*}$, to
potentially exist anywhere. In worst-case scenarios, many solution candidates
we searched for may be infeasible or non-optimal until we find
$\textbf{x}^{*}$, regardless of the optimization algorithms. In this case,
enumerating all the possible solution candidates may be necessary to provide
the optimality guarantee. This detrimentally impacts algorithm scalability.

If special properties of the black-box constraints are known, such as
continuity or convexity, we can efficiently narrow down the search space.
Although the problem is becoming less “black-box”, domain-specific observation
and techniques may be available to improve efficiency and potentially provide
an optimality guarantee, such as sustainable schedulability analysis [14],
convexity [16], dynamic programming [17]. However, the availability of these
domain-specific observations also implies a more limited range of
applicability.

Another promising direction that aligns with the black-box assumption (i.e.,
without reducing applicability) is machine learning (ML) methods [18, 19, 20].
ML can construct mathematical models to approximate the black-box constraints
and infer desirable properties. Once such a model is established, classical
optimization algorithms can be employed to find good solutions. However, a
significant challenge with these methods is ensuring feasibility, i.e.,
guaranteeing that the solutions found are always feasible.

###  V-B Future Work in Methdology Generalization

####  V-B1 Multi-objective optimization

Classical techniques from multi-objective optimization can be utilized with
NORTH+ if there are multiple objective functions. These techniques include
assigning weight parameters to combine multiple objectives into a scalar value
or transforming specific objective functions into constraints. For example, an
objective function aiming to minimize response time could be transformed into
a constraint that requires the response time to be smaller than a threshold.
NORTH+’s general applicability works with both strategies mentioned above.

####  V-B2 Discrete optimization

In this paper, we find that improving the system schedulability can indirectly
improve the overall performance of NORTH. This observation encourages
combining discrete optimization methods and schedulability analysis to achieve
better performance. We leave this part as the future work.

##  VI ACKNOWLEDGMENT

This work is partially supported by NSF Grants No. 1812963 and 1932074.

## References

  * [1] S. Wang, R. K. Williams, and H. Zeng, “A general and scalable method for optimizing real-time systems with continuous variables,” IEEE Real-Time and Embedded Technology and Applications Symposium, pp. 119–132, 2023.
  * [2] S. Baruah, A. K. Mok, and L. E. Rosier, “Preemptively scheduling hard-real-time sporadic tasks on one processor,” IEEE Real-Time Systems Symposium, pp. 182–190, 1990.
  * [3] L. Thiele, S. Chakraborty, and M. Naedele, “Real-time calculus for scheduling hard real-time systems,” in IEEE International Symposium on Circuits and Systems, pp. 101–104, 2000.
  * [4] R. Henia, A. Hamann, M. Jersak, R. Racu, K. Richter, and R. Ernst, “System level performance analysis–the symta/s approach,” IEE Proceedings-Computers and Digital Techniques, vol. 152, no. 2, pp. 148–166, 2005\.
  * [5] K. G. Larsen, P. Pettersson, and W. Yi, “Uppaal in a nutshell,” International Journal on Software Tools for Technology Transfer, vol. 1, pp. 134–152, 1997.
  * [6] M. Nasri, G. Nelissen, and B. B. Brandenburg, “Response-time analysis of limited-preemptive parallel dag tasks under global scheduling,” in ECRTS, 2019.
  * [7] L. Heintzman, A. Hashimoto, N. Abaid, and R. K. Williams, “Anticipatory planning and dynamic lost person models for Human-Robot search and rescue,” in IEEE International Conference on Robotics and Automation, pp. 8252–8258, May 2021.
  * [8] PerceptIn, “2021 rtss industry challenge.” http://2021.rtss.org/industry-session/, 2021.
  * [9] A. H. Simon Kramer, Dirk Ziegenbein, “Real world automotive benchmarks for free,” 2015.
  * [10] Nocedal and J. Wright, Numerical Optimization, 2nd edition.  Springer New York, NY, 2006.
  * [11] P. Pillai and K. Shin, “Real-time dynamic voltage scaling for low-power embedded operating systems,” Proceedings of the eighteenth ACM symposium on Operating systems principles, 2001.
  * [12] E. Bini, G. Buttazzo, and G. Lipari, “Minimizing cpu energy in real-time systems with discrete speed management,” ACM Trans. Embed. Comput. Syst., vol. 8, pp. 31:1–31:23, 2009.
  * [13] R. I. Davis and A. Burns, “Priority assignment for global fixed priority pre-emptive scheduling in multiprocessor real-time systems,” 2009 30th IEEE Real-Time Systems Symposium, pp. 398–409, 2009.
  * [14] Y. Zhao, R. Zhou, and H. Zeng, “An optimization framework for real-time systems with sustainable schedulability analysis,” 2020 IEEE Real-Time Systems Symposium (RTSS), pp. 333–344, 2020.
  * [15] M. Joseph and P. K. Pandya, “Finding response times in a real-time system,” Comput. J., vol. 29, pp. 390–395, 1986.
  * [16] H. Aydin, V. Devadas, and D. Zhu, “System-level energy management for periodic real-time tasks,” 2006 27th IEEE International Real-Time Systems Symposium (RTSS’06), pp. 313–322, 2006.
  * [17] S.-Y. Huang, J. Zeng, X. Deng, S. Wang, A. Sifat, B. Bharmal, J.-B. Huang, R. Williams, H. Zeng, and C. Jung, “Rtailor: Parameterizing soft error resilience for mixed-criticality real-time systems,” in 2023 IEEE Real-Time Systems Symposium (RTSS), IEEE, 2023.
  * [18] S. Lee, H. Baek, H. Woo, K. G. Shin, and J. Lee, “Ml for rt: Priority assignment using machine learning,” 2021 IEEE 27th Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 118–130, 2021.
  * [19] Z. Bo, Y. Qiao, C. Leng, H. Wang, C. Guo, and S. Zhang, “Developing real-time scheduling policy by deep reinforcement learning,” 2021 IEEE 27th Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 131–142, 2021.
  * [20] S. Wang, J. Chen, X. Deng, S. A. Hutchinson, and F. Dellaert, “Robot calligraphy using pseudospectral optimal control in conjunction with a novel dynamic brush model,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6696–6703, 2020.


11institutetext: Mohamed bin Zayed University of Artificial Intelligence, Abu
Dhabi, UAE 22institutetext: Sheikh Shakbout Medical City, Abu Dhabi, UAE
22email: {santosh.sanjeev,salwa.khatib,mai.kassem,ibrahim.almakky,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
#  PECon: Contrastive Pretraining to Enhance Feature Alignment between CT and
EHR Data for Improved Pulmonary Embolism Diagnosis

Santosh Sanjeev✉ 11 0000-0003-3664-3844 Salwa K. Al Khatib 11
0000-0002-9588-8473 Mai A. Shaaban 11 0000-0003-1454-6090 Ibrahim Almakky 11
0009-0008-8802-7107 Vijay Ram Papineni 22 0000-0002-6162-3290 Mohammad Yaqub
11 0000-0001-6896-1105

###### Abstract

Previous deep learning efforts have focused on improving the performance of
Pulmonary Embolism (PE) diagnosis from Computed Tomography (CT) scans using
Convolutional Neural Networks (CNN). However, the features from CT scans alone
are not always sufficient for the diagnosis of PE. CT scans along with
electronic heath records (EHR) can provide a better insight into the patient’s
condition and can lead to more accurate PE diagnosis. In this paper, we
propose Pulmonary Embolism Detection using Contrastive Learning (PECon), a
supervised contrastive pretraining strategy that employs both the patient’s CT
scans as well as the EHR data, aiming to enhance the alignment of feature
representations between the two modalities and leverage information to improve
the PE diagnosis. In order to achieve this, we make use of the class labels
and pull the sample features of the same class together, while pushing away
those of the other class. Results show that the proposed work outperforms the
existing techniques and achieves state-of-the-art performance on the RadFusion
dataset with an F1-score of 0.913, accuracy of 0.90 and an AUROC of 0.943.
Furthermore, we also explore the explainability of our approach in comparison
to other methods. Our code is publicly available at
https://github.com/BioMedIA-MBZUAI/PECon.

###### Keywords:

Contrastive learning Multimodal data Pulmonary Embolism CT scans.

##  1 Introduction

Pulmonary Embolism (PE) is an acute cardiovascular disorder considered the
third most common cause of cardiovascular death after coronary artery disease
and stroke. Despite advances in diagnosis and treatment over the past $30$
years, PE has high early mortality rates [1], with nearly $100k$ to $200k$
deaths in the US each year [2]. Unfortunately, individuals diagnosed with PE
frequently encounter a long delay before receiving a diagnosis, and
approximately $25\%$ of patients are initially misdiagnosed [3, 4]. Hence, it
is crucial to provide radiologists and clinicians with tools that can help
them with the diagnosis.

Prior to deep learning, many efforts focused on using traditional feature
extraction methods for PE diagnosis from Computed Tomography (CT) scans [5,
6]. However, more recently research efforts have investigated improving the
performance of PE diagnosis using deep Convolutional Neural Networks (CNNs)
[7, 8] and attention mechanisms [9]. More specifically, PENet [10] was
introduced as a 3D CNN designed to detect PE using multiple CT slices. Such
use of 3D convolutions allows the network to consider information from
multiple slices when making predictions, which is important for diagnosing PE,
as its presence is not limited to a single CT slice.

Features from CT scans alone could be insufficient for PE diagnosis.
Therefore, CT scans along with the patient’s Electronic Health Records (EHR)
data can provide a better insight into the patient’s condition and can help
improve the diagnosis of PE. Previous studies have shown improved performance
when combining demographic and clinical data with medical imaging data for
various medical conditions such as Alzheimer’s disease and skin cancer [11,
12]. However, multimodal fusion is a non-trivial task, which can have varying
results based on the fusion approach. Therefore, Huang et al. [13] compared
different multimodal fusion approaches that combine inputs from both CT scans
and EHR data to diagnose the presence of PE and explore optimal data selection
and fusion strategies. Their results show that multimodal end-to-end deep
learning models combining imaging and EHR provide better discrimination of
abnormalities than using either modality independently. To this effect, Zhou
et al. [14] released RadFusion, a multimodal dataset containing both EHR data
and high-resolution CT scans labeled for PE. They assess the fairness
properties across different subgroups and results suggest that integrating EHR
data with medical images can improve classification performance and robustness
without introducing large disparities between population groups.

Contrastive learning has been previously employed in medical imaging.
Recently, Zhang et al. [15] made use of available pairs of images and text
usually in medical contexts, and proposed Contrastive VIsual Representation
Learning from Text (ConVIRT) [15]. The approach involves pretraining medical
image encoders with paired text via a bidirectional contrastive objective.
Nevertheless, their text encoder employs a general-purpose lexicon, resulting
in issues with unknown words when processing a medical text. Although the
division of terms into word pieces helps alleviate this problem, it results in
the fragmentation of common biomedical terminology, leading to suboptimal
results [16, 17]. Furthermore, [18] has shown that constrastive learning
between two images (same modality) in a fully-supervised setting can
outperform the self-supervised contrastive pretraining by leveraging the label
information. In this paper, we propose Pulmonary Embolism Detection using
Contrastive Learning (PECon), a novel fully supervised contrastive pretraining
framework between 3D CT scans and EHR data. PECon uses a supervised
contrastive learning objective to train a neural network to encode CT scans
and EHR into a joint embedding space. In this space, we establish cross-modal
alignment, through a fully supervised contrastive (SC) pretraining aiming to
maximise the feature alignment between CT and EHR data. To the best of our
knowledge we are the first to introduce a SC pretraining using different
modalities (EHR,CT) in medical domain to maximize alignment between the
feature representations. Unlike [18], which uses augmentations to get 2 views
of the same image(single modality), we use CT and EHR as 2 views(multimodal)
of the same patient. We demonstrate the effectiveness of our approach on a
real-world dataset of PE patients, where we show that our pretrained model
significantly outperforms state-of-the-art baselines. The main contributions
of this work can be summarized as follows:

  * •

We propose a supervised contrastive pretraining method that better aligns the
image and EHR embeddings thereby achieving enhanced alignment between the
features of both modalities.

  * •

We demonstrate how the proposed method enhances the performance of multimodal
PE detection and achieves state-of-the-art results on the RadFusion [14]
dataset.

Figure 1: PECon: a self-supervised contrastive pretraining method for improved
PE diagnosis. The framework consists of a visual branch with a frozen image
encoder backbone and a trainable MLP (top part) and an EHR branch with a
trainable MLP (bottom left part). The output representations of both MLPs are
later used for supervised contrastive loss (bottom right part).
$(x_{c1},x_{e1})$, $(x_{c2},x_{e2})$, $(x_{c3},x_{e3})$ are the CT and EHR
embeddings of 3 patients in a given batch.

##  2 Methodology

In this section, we discuss the details of PECon, the proposed pretraining
method illustrated in Fig. 1, and the fine-tuning stage that follows. The
pretraining stage involves a supervised contrastive learning step, where, for
each anchor feature extracted from a CT scan volume or an EHR record, all
other features having the same label as this feature in the batch are pulled
to it, and all those having the opposite label are pushed away. As such, given
a pair of a CT scan and corresponding EHR data $(x_{c},x_{e})$ belonging to
the same patient, our goal is to learn a function
$M(x_{c},x_{e})=\hat{y}_{x_{c},x_{e}}$ that uses this paired input to
approximate $y$, the ground truth PE diagnosis of patient. We set a pre-text
task whose objective is to learn an image projection head $f_{c}$ and an EHR
projection head $f_{e}$ that respectively map $x_{c}$ and $x_{e}$ to
$d^{{}^{\prime}}$-dimensional vectors $z$. More precisely, projection head
$f_{c}$ takes as input a $d$-dimensional embedding, which is the result of
averaging $N\times d$-dimensional embeddings from an image encoder. The image
encoder is used to generate a $d$-dimensional embedding for each of the $N$
sub-volumes belonging to a single CT scan ($x_{c}^{(1)}$, $x_{c}^{(2)}$, ..
,$x_{c}^{(N)}$ as shown in Fig. 1). For a given input batch of data, [18]
applies data augmentation twice and obtains two copies of the batch.
Alternatively, in PECon, considering a batch containing $B$ pairs of
$(x_{c},x_{e})$, the training objective of PECon is a supervised contrastive
loss with the following expression:

$\mathcal{L}_{PECon}=\sum_{i=1}^{2B}\frac{-1}{|P(i)|}\sum_{p\in
P(i)}log\frac{exp(\langle z_{i}.z_{p}\rangle/\tau)}{\sum_{a\in
A(i)}exp(\langle z_{i}.z_{a}\rangle/\tau)}$ (1)

where $A(i)$ is the set of all $z$ features extracted from $x_{c}$ and $x_{e}$
samples in the batch distinct from feature $i$, $P(i)\equiv\\{p\in
A(i):y_{p}=y_{i}\\}$ is the set of features within the batch having the same
ground truth label as feature $i$, $|.|$ is the set cardinality, and
$\tau\in\mathcal{R}^{+}$ is a temperature parameter.

The visual projection head $f_{c}$ and the EHR projection head $f_{e}$ are
then separately fine-tuned for classification by minimizing the cross-entropy
loss. For this, the multimodal prediction is given by:

$\hat{y}_{x_{c},x_{e}}=\lambda\hat{y}_{x_{c}}+(1-\lambda)\hat{y}_{x_{e}}$ (2)

where $\lambda$ is a modality weighting hyper-parameter $\in[0,1]$, which can
be used to set the influence each modality model will have on the overall
multimodal classification.

##  3 Experimental Setup

###  3.1 Dataset

In this work, we use the RadFusion dataset [14], which consists of both CT
scans and EHR data of the patients with and without PE. It consists of $1,837$
axial CTPA exams from $1794$ patients captured with $1.25mm$ spacing. The
number of negative PE cases is $1,111$, while the number of positive PE cases
is $726$. The EHR data consists of demographic features, vitals such as
systolic and diastolic blood pressure, body temperature, $641$ unique
inpatient and outpatient medications, $141$ unique diagnosis groupings, and
laboratory tests of around $22$ categories. This information is represented
with a binary presence/absence as well as the latest value of the test. EHR in
the dataset is structured (categorical, numerical) without text. PE cases are
mainly of 3 types: central, segmental and subsegmental based on the location
of PE within the arterial branches. The number of cases of central, segmental
and subsegmental PE are 257, 387 and 52 respectively. For a fair comparison,
we followed the standard split of Radfusion[14] which was, training,
validation, and testing splits of $80\%$, $10\%$, $10\%$ respectively. The
scans are normalised between $[-1,1]$ and the EHR data is normalised between
$[0,1]$.

###  3.2 Implementation Details

Pretraining stage. Pretraining is carried out in an end-to-end fashion with an
SGD optimizer, using an initial learning rate (LR) of $0.1$, $\tau=0.8$, a
batch size of $128$, and for $100$ epochs on a single NVIDIA RTX A6000 GPU. We
select the weights of the epoch with the lowest validation loss for later
fine-tuning. As for data augmentation, the 3D scans are randomly (1) cropped
along their widths and heights to $192\times 192$, (2) jittered up to 8 slices
along the depth axis, and (3) rotated up to $15^{\circ}$, as is done in [10].
The image encoder architecture we adopt is PeNet [10]. The PENet we use is an
end to end 3D CNN model consisting of 4 encoder blocks primarily constructed
from 3D convolutions with skip connections and squeeze-and-excitation blocks.
The PENet backbone is initialized with weights trained on RadFusion,
pretrained on Kinetics dataset [19] and it remains frozen during the
pretraining. Input to the PENet models are subvolumes consisting of $24$
slices, where PENet outputs a $2048$ vector for each subvolume. The feature
vector input to the visual projection head is an average of the $2048$
embeddings generated for each subvolume. The image projection head is an MLP
consisting of $2$ hidden layers with $512$ and $256$ neurons respectively
outputting a feature vector of size $128$. Similarly, the EHR projection head
consists of $1$ hidden layer with $128$ neurons. Image size and number of
slices were chosen based on the PENet[10], which provided the best results and
allowed for a fair comparison. The modality weighting parameter was selected
empirically by testing with different values and choosing the best performing
on validation set. We explored different embedding sizes (128,256,512) with
performance stagnation after 128. Similarly other hyper-parameters have been
experimentally selected. Fine-tuning stage. In the fine-tuning stage, the
visual projection head and the EHR projection head are separately fine-tuned
on the RadFusion dataset. The imaging model is fine-tuned for $25$ epochs with
an SGD optimizer, using a LR of $0.01$ and a stepLR scheduler with a step size
of 20. The EHR model is fine-tuned for $25$ epochs using SGD optimizer with a
LR of $0.1$ and a stepLR scheduler with a step size of 10. We report results
for a $\lambda$ of $0.375$ and the ablation is shown in the Supplementary.

##  4 Results and Discussion

Table 1: Performance Comparison of the imaging-only, EHR-only and multimodal fusion models with the state-of-the-art Modality |  | Include
---
subsegmental
Models | Accuracy | F1 score | AUROC
CT | $\checkmark$ | PENet [10] | 0.689 | 0.677 | 0.796
PECon Visual model | 0.726 | 0.752 | 0.775
EHR | ElasticNet [14] | 0.837 | 0.850 | 0.922
PECon EHR model | 0.858 | 0.872 | 0.922
Multimodal | Radfusion [14] | 0.890 | 0.902 | 0.946
PECon Multimodal | 0.900 | 0.913 | 0.943
CT | $\times$ | PENet [10] | 0.759 | 0.735 | 0.842
PECon Visual model | 0.759 | 0.766 | 0.817
EHR | ElasticNet [14] | 0.877 | 0.877 | 0.932
PECon EHR model | 0.883 | 0.883 | 0.930
Multimodal | Radfusion [14] | 0.895 | 0.895 | 0.962
PECon Multimodal | 0.914 | 0.918 | 0.961
Figure 2: Illustration of TSNE plots. (a) Train set embeddings of Radfusion,
(b) Train set embeddings of InfoNCE, (c) Train set embeddings of PECon (Ours),
(d) Val set embeddings of Radfusion, (e) Val set embeddings of InfoNCE and (f)
Val set of embeddings of PECon

We consistently use the RadFusion test set to evaluate our approach and
compare it with existing approaches. Table 1 compares the performance of our
approach with the benchmark models. We observe that PECon achieves state-of-
the-art performance on CT, EHR, as well as the multimodal setting. The boost
in multimodal performance demonstrates the effectiveness of the approach in
improving class separability in the latent space. Further, the improvement on
the individual modalities shows a good separation even at the modality level,
where samples belonging to the same class are pulled together and samples
belonging to different classes are pushed apart. Although, we observe a small
dip in the AUROC score in the visual and multimodal settings, our focus is on
the F1-score as it provides a better measure of performance considering the
slight class imbalance. Due to the questionable clinical value of subsegmental
PE [20], we evaluate and compare the results with the benchmark models
including and excluding subsegmental PE from the test set. We observe similar
performance gains when including or excluding the subsegmental cases from the
test set. However, excluding subsegmental cases from the training data for
PECon during both pretraining and fine-tuning stages results in the best
performance. We test the effect of excluding the sub-segmental cases in either
the pretraining or the fine-tuning stages, while still including them in the
evaluation. Results in Table 2 show that excluding samples from both stages
achieves the best results.

Table 2: Ablation studies on the inclusion of subsegmental cases in the pre-training and finetuning stages. Pre-training | Finetuning | Accuracy | F1 score | AUROC
---|---|---|---|---
$\checkmark$ | $\checkmark$ | 0.889 | 0.905 | 0.931
$\checkmark$ | $\times$ | 0.878 | 0.893 | 0.941
$\times$ | $\checkmark$ | 0.873 | 0.887 | 0.920
$\times$ | $\times$ | 0.900 | 0.913 | 0.943
Figure 3: Class activation map representations of PENet and PECon on sample CT
slices from the RadFusion dataset.

In Figure 2, we visualize and compare the TSNE plots of Radfusion and PECon
(Ours) for the train and validation sets. We also visualize results of PECon
with InfoNCE [21]. We observe that pretraining using a contrastive loss
enhances the feature alignment between CT scan and EHR features, thereby
making the data more separable and forming clusters. On the other hand, the
RadFusion train and validation do not have a good separation. Furthermore, the
use of supervision in PECon pulled together sample features of the same class
while pushing away the features of different classes. This is lacking in the
case of InfoNCE since it is a fully self-supervised method that is agnostic to
the class labels of the features. To enhance the explainability of our
approach, we use Grad-CAM [22] to visualize class activation maps of three CT
slices pertaining to positive scans from the RadFusion dataset using PENet and
PECon in Figure 3. In the case of PENet, some attention is put on irrelevant
regions around the lungs, whereas PECon imaging model consistently
demonstrates better and more precise localization of the PE. This shows that
combining the two modalities using our approach enables the visual model to
learn more precise features from the input volumes. We further compare our
pretraining stage with other contrastive learning methods. Results in Table 3
show that PECon achieves better performance. This demonstrates that the
inclusion of supervision with contrastive learning is more effective for PE
classification compared to self-supervised pretraining methods. The hyper-
parameters for each method are reported in the Supplementary.

Table 3: Ablation studies on the contrastive learning strategy | Contrastive Learning
---
strategy
Accuracy | F1 score | AUROC
ConViRT (InfoNCE) [15] | 0.892±0.003 | 0.907±0.002 | 0.938±0.001
Barlow Twins [23] | 0.889±0.005 | 0.904±0.003 | 0.930±0.004
PECon | 0.898±0.003 | 0.912±0.002 | 0.939±0.003

##  5 Conclusion and Future Work

In this work, we present (PECon), a supervised contrastive pretraining
framework for multimodal prediction of PE to enhance the alignment between the
CT and EHR latent representations. To accomplish this, we start from an anchor
feature extracted from CT scans and EHR data and pull features from the same
class to it and push away those from the other class. We trained and validated
our approach on the RadFusion dataset, and report state-of-the-art results.
Additionally, we show that our model attends to more precise and localized
regions in the CT scans through class activation maps.

The limitations of this work include testing on other datasets to assess the
generalizability of our approach. Furthermore, our study was limited to the
impact of the contrastive pretraining loss on the late fusion multimodal
approach, where in the future it would be important to study this impact on
early and joint fusion approaches. Further, attention techniques will be
explored for feature aggregation.

## References

  * [1] Bělohlávek, J., Dytrych, V., Linhart, A.: Pulmonary embolism, part i: Epidemiology, risk factors and risk stratification, pathophysiology, clinical presentation, diagnosis and nonthrombotic pulmonary embolism. Experimental and clinical cardiology 18 2, 129–38 (2013)
  * [2] Tarbox, A., Swaroop, M.: Pulmonary embolism. International Journal of Critical Illness and Injury Science 3(1), 69–72 (2013). https://doi.org/10.4103/2229-5151.109427, https://www.ijciis.org/text.asp?2013/3/1/69/109427
  * [3] Hendriksen, J.M.T., Koster-van Ree, M., Morgenstern, M.J., Oudega, R., Schutgens, R.E.G., Moons, K.G.M., Geersing, G.J.: Clinical characteristics associated with diagnostic delay of pulmonary embolism in primary care: a retrospective observational study. BMJ open 7(3), e012789 (mar 2017). https://doi.org/10.1136/bmjopen-2016-012789
  * [4] Alonso-Martínez, J.L., Sánchez, F.J., Echezarreta, M.A.: Delay and misdiagnosis in sub-massive and non-massive acute pulmonary embolism. European journal of internal medicine 21, 278–282 (8 2010). https://doi.org/10.1016/J.EJIM.2010.04.005, https://pubmed.ncbi.nlm.nih.gov/20603035/
  * [5] Masutani, Y., MacMahon, H., Doi, K.: Computerized detection of pulmonary embolism in spiral ct angiography based on volumetric image analysis. IEEE Transactions on Medical Imaging 21(12), 1517–1523 (2002). https://doi.org/10.1109/TMI.2002.806586
  * [6] Liang, J., Bi, J.: Computer aided detection of pulmonary embolism with tobogganing and mutiple instance classification in ct pulmonary angiography. In: Karssemeijer, N., Lelieveldt, B. (eds.) Information Processing in Medical Imaging. pp. 630–641. Springer Berlin Heidelberg, Berlin, Heidelberg (2007)
  * [7] Lin, Y., Su, J., Wang, X., Li, X., Liu, J., Cheng, K.T., Yang, X.: Automated pulmonary embolism detection from ctpa images using an end-to-end convolutional neural network. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. pp. 280–288. Springer International Publishing, Cham (2019)
  * [8] Khachnaoui, H., Agrébi, M., Halouani, S., Khlifa, N.: Deep learning for automatic pulmonary embolism identification using cta images. In: 2022 6th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP). pp. 1–6 (2022). https://doi.org/10.1109/ATSIP55956.2022.9805929
  * [9] Suman, S., Singh, G., Sakla, N., Gattu, R., Green, J., Phatak, T., Samaras, D., Prasanna, P.: Attention based cnn-lstm network for pulmonary embolism prediction on chest computed tomography pulmonary angiograms (2021). https://doi.org/10.48550/ARXIV.2107.06276, https://arxiv.org/abs/2107.06276
  * [10] Huang, S.C., Kothari, T., Banerjee, I., Chute, C., Ball, R., Borus, N., Huang, A., Patel, B., Rajpurkar, P., Irvin, J., Dunnmon, J., Bledsoe, J., Shpanskaya, K., Dhaliwal, A., Zamanian, R., Ng, A., Lungren, M.: Penet–a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric ct imaging. npj Digital Medicine 3 (12 2020). https://doi.org/10.1038/s41746-020-0266-y
  * [11] Li, H., Fan, Y.: EARLY PREDICTION OF ALZHEIMER’S DISEASE DEMENTIA BASED ON BASELINE HIPPOCAMPAL MRI AND 1-YEAR FOLLOW-UP COGNITIVE MEASURES USING DEEP RECURRENT NEURAL NETWORKS. Proceedings. IEEE International Symposium on Biomedical Imaging 2019, 368–371 (apr 2019). https://doi.org/10.1109/ISBI.2019.8759397
  * [12] Kawahara, J., Daneshvar, S., Argenziano, G., Hamarneh, G.: Seven-Point Checklist and Skin Lesion Classification Using Multitask Multimodal Neural Nets. IEEE Journal of Biomedical and Health Informatics 23(2), 538–546 (2019). https://doi.org/10.1109/JBHI.2018.2824327
  * [13] Huang, S.C., Pareek, A., Zamanian, R., Banerjee, I., Lungren, M.P.: Multimodal fusion with deep neural networks for leveraging ct imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific reports 10(1), 1–9 (2020)
  * [14] Zhou, Y., Huang, S.C., Fries, J.A., Youssef, A., Amrhein, T.J., Chang, M., Banerjee, I., Rubin, D., Xing, L., Shah, N., Lungren, M.P.: Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detection from ct and ehr (2021). https://doi.org/10.48550/ARXIV.2111.11665, https://arxiv.org/abs/2111.11665
  * [15] Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference. pp. 2–25. PMLR (2022)
  * [16] Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et al.: Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 (2023)
  * [17] Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3(1), 1–23 (2021)
  * [18] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems 33, 18661–18673 (2020)
  * [19] Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., Zisserman, A.: A short note about kinetics-600 (2018). https://doi.org/10.48550/ARXIV.1808.01340, https://arxiv.org/abs/1808.01340
  * [20] Albrecht, M.H., Bickford, M.W., Nance, Jr., J.W., Zhang, L., De Cecco, C.N., Wichmann, J.L., Vogl, T.J., Schoepf, U.J.: State-of-the-Art Pulmonary CT Angiography for Acute Pulmonary Embolism. Am. J. Roentgenol. (Nov 2016). https://doi.org/10.2214/AJR.16.17202
  * [21] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [22] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [23] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction (2021). https://doi.org/10.48550/ARXIV.2103.03230, https://arxiv.org/abs/2103.03230