# Structured Pruning for Multi-Task Deep Neural Networks

Siddhant Garg
University of Massachusetts Amherst
<EMAIL_ADDRESS>Lijun Zhang
University of Massachusetts Amherst
<EMAIL_ADDRESS>Hui Guan
University of Massachusetts Amherst
<EMAIL_ADDRESS>
###### Abstract

Although multi-task deep neural network (DNN) models have computation and
storage benefits over individual single-task DNN models, they can be further
optimized via model compression. Numerous structured pruning methods are
already developed that can readily achieve speedups in single-task models, but
the pruning of multi-task networks has not yet been extensively studied. In
this work, we investigate the effectiveness of structured pruning on multi-
task models. We use an existing single-task filter pruning criterion and also
introduce an MTL-based filter pruning criterion for estimating the filter
importance scores. We prune the model using an iterative pruning strategy with
both pruning methods. We show that, with careful hyper-parameter tuning,
architectures obtained from different pruning methods do not have significant
differences in their performances across tasks when the number of parameters
is similar. We also show that iterative structure pruning may not be the best
way to achieve a well-performing pruned model because, at extreme pruning
levels, there is a high drop in performance across all tasks. But when the
same models are randomly initialized and re-trained, they show better results.

##  1 Introduction

Multi-task learning (MTL) addresses multiple machine learning tasks
simultaneously by creating a single multi-task deep neural network (DNN) [39].
Due to parameter sharing, a multi-task DNN model is more computation and
memory efficient compared to multiple single-task models. The MTL framework is
extremely useful for resource-constrained devices like smartphones, wearables,
and self-driving cars that host AI-powered applications but have low memory
resources and strict latency requirements. For example, in self-driving cars,
the model needs to recognize traffic lights, objects, and lanes based on the
input signal [37].

On the other hand, network pruning [24, 11, 26] is a long-standing and
effective method to compress DNNs. It aims to detect the importance of model
parameters and remove the ones that tend to have the least significance on the
performance. Approaches for network pruning can be classified as unstructured
pruning methods [11, 24] that mask individual weights in the network, and
structured pruning methods [26, 36] that remove complete filters and directly
lead to efficient deep neural models without requiring specialized hardware
for sparse structures. Network pruning is studied rigorously in literature for
single-task models and there exist many pruning criteria based on weight
magnitude [11, 26], connection sensitivity [25, 10], and even learning-based
pruning methods [7, 19].

However, the study on the effectiveness of structured pruning methods on
multi-task models is sparse. A few works like [4, 17] propose weight sharing
and merging strategies for constructing a multi-task model from multiple
single-task models such that there is a minimum conflict between tasks. Then
the constructed multi-task model could be effectively pruned with single-task
pruning methods. But these works show very similar results when comparing
their proposed pipeline with the baseline single-task pruning methods. Another
work [43], proposed a method to directly prune MTL networks but while pruning
those models with single-task pruning baselines, they did not try different
hyperparameter settings to retrain/fine-tune the pruned models. It could be
possible that different hyperparameter settings would work better because
different pruning methods lead to different architectures.

In this work, we investigate the effectiveness of structured pruning on multi-
task DNNs. We apply two structured pruning methods to prune multi-task models
and show that regardless of the method used, we can obtain similar results
from the pruned models with the same number of parameters. Specifically, we
use an existing single-task pruning method as well as introduce another MTL-
based pruning criterion. The proposed criterion is called CosPrune and it
identifies and prunes the convolutional filters that have conflicts between
tasks. It uses pairwise cosine similarity between the task-specific gradients
that flow through the filter during back-propagation. We accumulate this
similarity score for some training iterations for every filter in the multi-
task model. The filters with the least accumulated scores are pruned away. In
contrast, the single-task pruning method is called Taylor Pruning [36] which
is a popular gradient-based pruning method. It determines the importance of
the filter by looking at the increase in the loss function if that filter is
removed. The Taylor pruning importance score is also accumulated over some
training iterations before pruning.

We start our analyses by using the iterative pruning and fine-tuning strategy
which repeatedly prunes a small proportion of filters and fine-tunes the
multi-task model to gain back the lost performance [11, 30]. Using this
strategy we get consistently better results with CosPrune against Taylor
pruning across all the tasks. The multi-task model achieved higher
GFLOPs/parameter reduction with CosPrune without performance loss across all
the tasks. This shows that the proposed CosPrune criterion coupled with
iterative pruning is a reasonable method for pruning multi-task models.

However, when we re-train the pruned models independently with random
initialization, we observe that they can give relatively better results on all
the tasks when compared to the corresponding fine-tuning stage of iterative
pruning. The key is to determine good learning rates for each of the pruned
models. Using the same learning rates is not the best strategy to compare
different pruned architectures. This is because every model has different
layer-wise configurations and so, an optimal hyper-parameter setting for one
model may not be best for the other models with different architectural
designs. This type of analysis is generally not done in the existing
literature – the same training settings are used for the dense model as well
as the pruned models. A study on single-task structured pruning [30] also
shows that re-training the pruned models from random initialization can lead
to better results than fine-tuning the pruned architectures in the iterative
pruning setting.

Furthermore, the pruned architectures from different pruning methods give
similar results to each other at the same parameter level after re-training.
There are no consistent winners with respect to different pruning criteria,
which is contrary to what we observe in the case of iterative pruning.  There
are also some recent works [27, 29] in the context of single-task neural
networks where random channel pruning is able to match the results of the
dense model under appropriate settings. Similarly, another work [30] used
different pruning methods on various architectures to show that randomly re-
initializing the pruned models can match the performance of the respective
unpruned models but they did not compare those different pruning strategies on
the same model.

We go beyond existing works and apply different structured pruning methods to
the same multi-task model. We emphasize that the pruned MTL model obtained
from any reasonable pruning method can perform well if it is trained from
random initialization with its optimal learning rate. And we would like
researchers to pay attention to the re-training comparisons with sufficient
hyper-parameter tuning when they try to propose a new pruning method.

##  2 Related Works

Multi-Task Learning: Deep MTL networks afford numerous benefits in terms of
computation (storage and latency) [22], knowledge sharing between tasks [1]
and improving generalization in the learned representations [35] due to which
they have applications across many domains like Computer Vision [8, 20, 14,
13], Robotics [46, 50], and Reinforcement Learning [45, 42, 47, 48]. The most
common MTL framework is hard parameter sharing [2, 39] where a backbone
network is shared among all tasks and with individual task-specific heads.
Soft parameter sharing, on the other hand, has different sets of parameters
for individual tasks [35, 44] and various methods are used to effectively
combine information from them [34, 21, 38] to make predictions.

Deep Neural Network Pruning: It has long been established that the deep neural
networks are highly over-parameterized [24, 16, 26, 33, 15], and more than
$90\%$ of the connections can be pruned away without losing accuracy.
Unstructured pruning methods for single-task models [11, 12] can achieve high
sparsity with latency improvements with accelerated hardware for sparse neural
networks [51]. On the other hand, structured pruning can directly lead to
computational benefits by removing whole filters in the case of convolutional
layers [26]. Gradient-based metrics for estimating the importance scores of
the filters have recently become popular over magnitude-based criteria [26].
For example, Taylor Pruning [36] uses the dot product between the filter
weights with its gradient to approximate the change in loss function that
could occur if that filter is masked. Another example is SNIP [25], which
determines the connection sensitivity using a similar importance metric as
Taylor Pruning but it is done only once at initialization to apply single-shot
pruning. Furthermore, random channel pruning methods [27, 29] are also shown
to perform well under appropriate training conditions.

Pruning Multi-task Neural Networks: There is very limited study on pruning MTL
neural networks but nonetheless it is an important problem because of its
tremendous potential in deriving efficient deep learning models. At the same
time, it is also a difficult problem because of the coexistence of complex
task relationships in the shared parameter space and redundancy in the neural
network. A recent work on MTL pruning is DiSparse [43] which aims to
disentangle task relationships to find the unanimously least important filters
across all the tasks and prune them. Another work called PAM [17] propose a
method to merge single-task models into an MTL network such that the resulting
MTL network can be safely pruned while considering the tasks’ relatedness.
Similarly, other works like [5, 18] propose different merging strategies for
the construction of computationally efficient MTL networks. All of these
existing works do baseline comparisons by using single-task pruning methods to
prune their MTL networks. However, they use only the hyperparameters that are
specified in the original study on single-task pruning methods irrespective of
the different model architectures and sparsity ratios. But in our work, we do
extensive analyses and hyperparameter tuning for both the pruning methods
involved to show their true effectiveness in pruning deep MTL networks.

##  3 Pruning Methods

In this section, we will define our Multi-Task Learning model and the proposed
CosPrune importance score estimation method as well as review the Taylor
Pruning importance score.

Multi-task framework: Given a set of $T$ tasks
$\mathcal{T}=\\{\tau_{1},\dots,\tau_{T}\\}$, an MTL dataset with inputs,
$\mathcal{X}$, and the corresponding labels
$\mathcal{Y}=\\{Y_{1},\dots,Y_{T}\\}$ where $Y_{t}$ is set of labels for task
$\tau_{t}\in\mathcal{T}$, we want to learn the mapping
$f_{\Theta}:\mathcal{X}\rightarrow\mathcal{Y}$, where
$\Theta=\theta_{s}\cup\\{\theta_{t}\\}_{t=1}^{T}$. Here $\theta_{s}$ is the
set of model parameters that are shared by all the tasks and $\theta_{t}$ is
the set of parameters only for task $\tau_{t}$. From now, we will denote
$\\{\theta_{t}\\}$ as the set of all the task-specific parameters for all the
tasks collectively and omit values of $t$ for abbreviation. The parameters
$\Theta$ are trained by minimizing the multi-task loss function given by

$\displaystyle\mathcal{L}(\mathcal{X},\mathcal{Y},\Theta)$
$\displaystyle=\sum_{t=1}^{T}\ell_{t}\left(\mathcal{X},\mathcal{Y},\theta_{s},\theta_{t}\right)$
(1)

where $\ell_{t}\left(\mathcal{X},\mathcal{Y},\theta_{s},\theta_{t}\right)$ is
the loss function for the task $\tau_{t}$.

CosPrune Importance Score Estimation: Let $\mathbf{W}^{k\times k\times
c}\in\Theta$ be a convolutional filter with kernel size $k$, and $c$ channels,
then the weights of this filter will be optimized using the gradient descent
given by equation 2,

$\displaystyle\mathbf{W}\leftarrow\mathbf{W}-\eta\nabla_{\mathbf{W}}\mathcal{L}(\mathcal{X},\mathcal{Y},\Theta)$
(2)

where $\eta$ is the learning rate and
$\nabla_{\mathbf{W}}\mathcal{L}(\mathcal{X},\mathcal{Y},\Theta)$ is the
gradient of the total loss function with respect to $\mathbf{W}$. Let
$\mathbf{g}_{\mathcal{T}}^{\mathbf{W}}=\nabla_{\mathbf{W}}\mathcal{L}(\mathcal{X},\mathcal{Y},\Theta)$,
then

$\displaystyle\mathbf{g}^{\mathbf{W}}_{\mathcal{T}}$
$\displaystyle=\sum_{t=1}^{T}\mathbf{g}^{\mathbf{W}}_{t}$ (3)

where
$\mathbf{g}^{\mathbf{W}}_{t}=\nabla_{\mathbf{W}}\ell_{t}\left(\mathcal{X},\mathcal{Y},\theta_{s},\theta_{t}\right)$.
$\mathbf{g}^{\mathbf{W}}_{\mathcal{T}}$ and $\mathbf{g}^{\mathbf{W}}_{t}$ can
be denoted as the total gradient and task-$\tau_{t}$ gradient respectively
with respect to the parameter $\mathbf{W}$. Note that the total gradient is
the vector sum of all the individual task gradients. These individual
gradients can have high differences in magnitudes as well as they can point to
different directions with negative cosine similarity between them. This can
lead to conflicts between different tasks and that could be detrimental to the
optimization process [49]. There have been several works like PCGrad [49],
MGDA [6], CAGrad [28] that apply various methods to remove the conflicts in
the optimization process. In our work, we prune the filters that have the
highest degree of accumulated conflicts but optimize our MTL model with the
total gradient. This helps us to achieve the highest degree of optimization in
terms of computation and performance.

Algorithm 1 MTLCosPrune

Input: Dataset: $\mathcal{X}$, $\mathcal{Y}$ with $T$ tasks; Model Parameters:
$\mathbf{W}\in\Theta$; Fine-tune Iterations – $I$; $\\#$Filters to prune at
each pruning step – $P$,

Initialize $\Theta$,

while $\\#$Unpruned Filters $>threshold$ do Set
$\mathcal{A}\mathcal{C}(\mathbf{W})\leftarrow 0$ $\forall\mathbf{W}\in\Theta$
(accumulated scores)

for $i=1,2,\dots,I$ do $\mathbf{x}_{i}\leftarrow$Batch Inputs
$\mathbf{y}_{i}\leftarrow$Batch Labels
$\mathbf{g}^{\mathbf{W}}_{t}=\nabla_{\mathbf{W}}\ell_{t}\left(\mathbf{x}_{i},\mathbf{y}_{i},\theta_{s}^{i},\theta_{t}^{i}\right)$
$\forall t$, $\forall\mathbf{W}\in\theta_{s}^{i}$ Get
$\mathcal{C}(\mathbf{W},\\{\mathbf{g}^{\mathbf{W}}_{t}\\})$ using 4 $\forall
t$, $\forall\mathbf{W}\in\theta_{s}^{i}$ Update
$\mathcal{A}\mathcal{C}(\mathbf{W})\leftarrow\mathcal{A}\mathcal{C}(\mathbf{W})+\mathcal{C}(\mathbf{W},\\{\mathbf{g}^{\mathbf{W}}_{t}\\})$
$\forall t$, $\forall\mathbf{W}\in\theta_{s}^{i}$ Compute
$\mathbf{g}^{\mathbf{W}}_{\mathcal{T}}=\sum_{t=1}^{T}\mathbf{g}^{\mathbf{W}}_{t}$
Update
$\mathbf{W}\leftarrow\mathbf{W}-\eta\mathbf{g}^{\mathbf{W}}_{\mathcal{T}}$,
$\forall t$, $\forall\mathbf{W}\in\theta_{s}^{i}$

end forPrune $P$ filters $\mathbf{W}$ with lowest
$\mathcal{A}\mathcal{C}(\mathbf{W})$ scores

end while

To calculate the degree of conflict, we define the CosPrune Importance score,
as the sum of pairwise cosine similarity between all the tasks for the
parameter $\mathbf{W}$. Let
$\tilde{\mathbf{g}}^{\mathbf{W}}_{t}=\frac{\mathbf{g}^{\mathbf{W}}_{t}}{\left\lVert\mathbf{g}^{\mathbf{W}}_{t}\right\rVert}$
be the normalized task gradient, and $\\{\mathbf{g}^{\mathbf{W}}_{t}\\}$ be
the set of all the task gradients with respect to $\mathbf{W}$, then
$\mathcal{C}\left(\mathbf{W},\\{\mathbf{g}^{\mathbf{W}}_{t}\\}\right)$ can be
calculated using:

$\displaystyle\mathcal{C}\left(\mathbf{W},\\{\mathbf{g}^{\mathbf{W}}_{t}\\}\right)$
$\displaystyle=\sum_{(\tau_{i},\tau_{j})\in\mathcal{T}\times\mathcal{T}}\tilde{\mathbf{g}}^{\mathbf{W}}_{i}.\tilde{\mathbf{g}}^{\mathbf{W}}_{j}$
(4)

We only prune the shared parameters, $\mathbf{W}\ \in\theta_{s}$, therefore
$\mathbf{g}^{\mathbf{W}}_{t}\neq\mathbf{0}$ for all $t=1,\dots,T$. The value
of $\mathcal{C}(.,.)$ will be higher when the pairwise gradients are in
agreement and it will lower if tasks are in conflict. For calculating the
final importance score of a filter, we keep on adding the CosPrune scores for
all the filters for a fixed number of training iterations. After that, we rank
the filters according to their accumulated importance scores and prune the
lowest-scoring filters. After pruning, we reset the accumulated scores to zero
and fine-tune the model again along with collecting the CosPrune scores of the
remaining unpruned filters. The details are provided in Algorithm 1.

Taylor Pruning [36]: It is a structured pruning method for single-task
convolution neural networks which serves as a popular baseline for modern
gradient-based pruning methods. In this method, the importance score of a
filter is defined by the squared change in the loss function induced by
removing the filter. To make the computation efficient, the squared change is
approximated by the first-order Taylor expansion which simplifies to a dot
product between the filter weights and its gradient. Let the importance score
for the filter $\mathbf{W}$ be $\mathbf{I}$, then it is given by

$\displaystyle\mathbf{I}=\mathbf{W}.\mathbf{g}^{W}_{\mathcal{T}}$ (5)

where $\mathbf{g}^{\mathbf{W}}_{\mathcal{T}}$ is the total gradient passing
through filter $\mathbf{W}$ during backpropagation as define in equation 3.

We experiment with both CosPrune and Taylor pruning methods and compare their
performance in effectively pruning the MTL models.

##  4 Experiments and Results

In this section, we provide the details of all the experiments and provide
quantitative evidence of our claims.

###  4.1 Setup

Model Architecture: For our multi-task model, we use a hard parameter-sharing
paradigm where the backbone parameters are shared across all the tasks and
individual classification heads are used for each task. MTL backbone comprises
of VGG-16 [41] model without the last fully-connected layers and MTL
classification heads use Artrous Spatial Pyramid Pooling (ASPP) heads [3] for
each task. The backbone contains approximately $71M$ parameters and each ASPP
head has approximately $13.4M$ parameters. In all of our experiments, we only
prune the backbone but all the results are reported with the total model
parameters (backbone + all task heads) which are approximately $84.12M$
parameters.

Multi-Task Learning (MTL) Dataset: In this work, NYUv2 [40] dataset is used
for all of the experiments. It is a popular Multi-Task Learning MTL dataset
with densely labeled images recorded from RGB and Depth cameras of Microsoft
Kinect. It contains three tasks – Semantic Segmentation, Depth Estimation, and
Surface Normal Prediction whose ground truth labels are defined in [8].

Evaluation Metrics: Semantic segmentation is evaluated using mean Intersection
over Union (mIoU) and the Pixel accuracy (both the higher the better). Depth
Estimation is evaluated using absolute and relative errors calculated using
the L1 loss between the ground truth and predictions (both the lower the
better). For this task, we also report the relative difference between the
prediction and ground truth via the percentage of
$\delta=\max\left(\frac{y_{pred}}{y_{gt}},\frac{y_{gt}}{y_{pred}}\right)$
within threshold $1.25,1.25^{2}$, and $1.25^{3}$ [9] (the higher the better).
Surface Normal Prediction is evaluated using the Mean and Median Angle errors
calculated by cosine similarity loss (the lower the better). We also report
the percentage of pixels whose predictions are within $11.25^{\circ}$,
$22.5^{\circ}$, and $30^{\circ}$ to the ground truth (higher the better)[8].
Due to space constraints and a high number of pruned model configurations, we
report Pixel Accuracy for Semantic Segmentation, relative error for Depth
Estimation, and Angle Mean for Surface Normal Prediction as the main
evaluation metrics, for both pruning methods. All other metrics are reported
in the supplementary material

Loss Functions and Model Training: For training on the NYUv2 dataset, the
model minimizes the sum of losses from the individual tasks with equal
weightage. Semantic segmentation uses cross-entropy loss, Surface Normal
Prediction uses cosine similarity loss and Depth Estimation uses L1 loss as
its training signals. We start with a randomly initialized MTL model with
VGG-16 backbone and ASPP heads and train the model for 500 epochs. We use a
batch size of $16$, and $0.0001$ learning rate with cosine scheduling [31]. We
use AdamW [32] optimizer for the iterative pruning experiments.

$\\#$Params |  Semantic Segmentation |  |  |  |  |  |
---|---|---|---|---|---|---|---
Pixel Accuracy $(\%)$ $\uparrow$ |  Depth Estimation |  |  |  |  |  |  |
Relative Error $\downarrow$ |  Surface Normal |  |  |  |  |  |  |
Angle Mean $\downarrow$ |  |  |  |  |  |  |  |  |  |
$\\#(M)$ |  $\%_{R}$ |  Taylor |  CosPrune |  $\%\Delta\uparrow$ |  Taylor |  CosPrune |  $\%\Delta\downarrow$ |  Taylor |  CosPrune |  $\%\Delta(\downarrow)$
$59.9$ |  $28.7$ |  $\mathbf{45.96}$ |  $45.91$ |  $-\phantom{1}0.11$ |  $0.2616$ |  $\mathbf{0.2614}$ |  $\mathbf{-\phantom{1}0.08}$ |  $18.54$ |  $\mathbf{18.37}$ |  $\mathbf{-0.92}$
$42.7$ |  $49.2$ |  $45.66$ |  $\mathbf{45.70}$ |  $\mathbf{+\phantom{1}0.09}$ |  $\mathbf{0.2666}$ |  $0.2670$ |  $+\phantom{1}0.17$ |  $18.35$ |  $\mathbf{18.22}$ |  $\mathbf{-0.73}$
$21.2$ |  $74.8$ |  $43.02$ |  $\mathbf{44.86}$ |  $\mathbf{+\phantom{1}4.27}$ |  $0.2862$ |  $\mathbf{0.2777}$ |  $\mathbf{-\phantom{1}2.98}$ |  $18.19$ |  $\mathbf{17.19}$ |  $\mathbf{-1.53}$
$17.6$ |  $79.1$ |  $41.56$ |  $\mathbf{42.98}$ |  $\mathbf{+\phantom{1}3.41}$ |  $0.3384$ |  $\mathbf{0.2829}$ |  $\mathbf{-16.42}$ |  $18.16$ |  $\mathbf{18.03}$ |  $\mathbf{-0.71}$
$14.3$ |  $83.0$ |  $35.83$ |  $\mathbf{41.70}$ |  $\mathbf{+16.39}$ |  $0.3896$ |  $\mathbf{0.3276}$ |  $\mathbf{-15.93}$ |  $18.65$ |  $\mathbf{18.00}$ |  $\mathbf{-3.45}$
$13.4$ |  $84.0$ |  $34.88$ |  $\mathbf{36.08}$ |  $\mathbf{+\phantom{1}3.43}$ |  $0.3928$ |  $\mathbf{0.3747}$ |  $\mathbf{-\phantom{1}4.59}$ |  $18.82$ |  $\mathbf{18.39}$ |  $\mathbf{-2.26}$

Table 1: Iterative pruning results at selected iterations. $\\#(M)$: _Number
of model parameters in millions_. $\%_{R}$: _$\%$ parameter reduction with
respect to full model_. $\%\Delta\uparrow(\downarrow)$: _$\%$ increase
(decrease) with respect to Taylor pruning_. $\uparrow$: _Higher the better_
$\downarrow$: _Lower the better_.

(a) Pixel Accuracy

(b) Relative Error

(c) Angle Mean

Figure 1: Iterative pruning performance trends for different tasks for the
Taylor and CosPrune methods.

Model Pruning: After the model is trained on the NYUv2 dataset, we apply
iterative pruning with different filter pruning criteria (CosPrune and Taylor
pruning) to get pruned MTL models. We initialize the model with the NYUv2
trained weights and set the initial learning rate as $10^{-5}$ with cosine
scheduling for $500$ epochs and start the pruning process. For every iteration
(batch of inputs to the model), we get the individual task gradients by
applying backward propagation on task losses. Then we calculate the importance
score for every filter in the model. The importance scores are accumulated for
10 epochs after which 100 filters that have the lowest accumulated scores, are
pruned. This completes a single pruning iteration and after it, the
accumulated filter importance scores are reset to $0$. The learning rate and
scheduler are also rewinded to the initial settings and the pruning process
continues. Note that the parameters of the model keep on updating using the
total loss function after every iteration which marks the fine-tuning phase of
the model.

###  4.2 Results: Multi-Task Structure Pruning

Iterative Pruning Results: We applied the iterative pruning strategy with both
the Taylor Pruning criterion and the CosPrune importance criterion separately
on the model initialized with the trained NYUv2 weights and ran $6$ sets of
experiments to accommodate for the variance in each method. We present a
subset of evaluation metrics in Figure 1, where we show the best values for
different task metrics ($y$-axis) against the number of parameters ($x$-axis)
for both methods. We have shown plots for Pixel Accuracy (Semantic
Segmentation), relative error (Depth Estimation), and Angle Mean (Normal
prediction). The plots for the remaining metrics are reported in the
supplementary material.

From the plots in Figure 1, we can see that the CosPrune method is
consistently better than the Taylor pruning method across all the tasks, i.e.,
it has a relatively lower Pixel Accuracy drop in the case of Semantic
Segmentation, and lower error increment in case of Depth Estimation and Normal
prediction tasks. We also report numerical values in Table 4.1 where we
compare both the methods for the pruned models with a similar number of
parameters. We can see that there is the highest performance gap at $14.3M$
model parameters which is $83\%$ parameter reduction, and Pixel Accuracy is
$\mathbf{16.39\%}$ better, Depth Estimation is $\mathbf{15.93\%}$ better and
Normal prediction is $\mathbf{3.45\%}$ better than the Taylor pruning method.
However, when with extreme pruning at $13.4M$ parameters, the gap is
significantly lesser but nonetheless still better. For $59.9M$ parameters, the
Pixel Accuracy slightly drops for CosPrune but all the metrics are very close
as can also be seen from the plots in Figure 1.

$\\#$Param |  $lr$ setting |  Semantic Segmentation |  |  |  |
---|---|---|---|---|---|---
Pixel Accuracy $(\%)$ $\uparrow$ |  Depth Estimation |  |  |  |  |  |
Relative Error $\downarrow$ |  Surface Normal |  |  |  |  |  |  |
Angle Mean $\downarrow$ |  |  |  |  |  |  |  |  |
$\\#(M)$ |  $\%_{R}$ |  Taylor |  CosPrune |  Taylor |  CosPrune |  Taylor |  CosPrune |  Taylor |  CosPrune
$60.3$ |  $28.3$ |  $0.0005$ |  $0.0001$ |  47.82 |  $46.52$ |  $0.2688$ |  0.2525 |  17.97 |  $18.20$
$37.1$ |  $55.8$ |  $0.001\phantom{1}$ |  $0.0001$ |  46.51 |  $46.18$ |  $0.2761$ |  0.2614 |  17.91 |  $18.21$
$29.7$ |  $64.6$ |  $0.001\phantom{1}$ |  $0.0001$ |  46.44 |  $45.34$ |  $0.2835$ |  0.2761 |  17.85 |  $18.09$
$25.9$ |  $69.2$ |  $0.001\phantom{1}$ |  $0.0005$ |  $46.86$ |  $\mathbf{47.84}$ |  $0.2853$ |  $\mathbf{0.2604}$ |  $\mathbf{17.78}$ |  $17.97$
$19.1$ |  $77.3$ |  $0.001\phantom{1}$ |  $0.0001$ |  $\mathbf{45.54}$ |  $43.42$ |  $0.2874$ |  $\mathbf{0.2570}$ |  $\mathbf{17.86}$ |  $17.92$
$15.3$ |  $81.8$ |  $0.001\phantom{1}$ |  $0.0001$ |  $42.86$ |  $\mathbf{44.67}$ |  $0.3362$ |  $\mathbf{0.2957}$ |  $18.08$ |  $\mathbf{17.99}$
$14.4$ |  $82.9$ |  $0.001\phantom{1}$ |  $0.001\phantom{1}$ |  $41.82$ |  $\mathbf{44.40}$ |  $0.3381$ |  $\mathbf{0.3109}$ |  $18.10$ |  $\mathbf{18.06}$

Table 2: Results of the pruned models trained from scratch across different
tasks. $\\#(M)$: _Number of model parameters in millions_. $\%_{R}$: _$\%$
parameter reduction with respect to full model_. $lr$ setting: _Optimal
learning rate for the respective model_. $\uparrow$: _Higher the better_
$\downarrow$: _Lower the better_.

(a) Pixel Accuracy

(b) Relative Error

(c) Angle Mean

Figure 2: Results of the pruned models, trained individually from scratch,
across all the tasks for various model parameters. Solid plot lines correspond
to primary results. Dashed plots correspond to the iterative pruning trends
from Figure 1.

Re-training the Pruned models: So far we have observed that the CosPrune can
efficiently prune the MTL model to achieve high parameter reduction within a
few iterations while maintaining its performance over the baseline method.
Both pruning criteria lead to pruned models with different architectural
designs in terms of the number of filters in each layer of the backbone. Since
there is a difference in the performance, we want to figure out the role of
these different layer configurations in the model’s performance for all the
tasks. To do this, we take the pruned models at various iterations, randomly
initialize the weights and train them to converge on the NYUv2 dataset.
Initially, we were using the same hyperparameters for training all the models
but saw that some models were severely affected. For example, a model with
$15M$ parameters, obtained from CosPrune could achieve only $25.5\%$ Pixel
Accuracy even after training for $500$ epochs. But when the learning rate of
$0.0001$ was used it reached over $42\%$, and the performance of the other two
tasks was also improved. In another case, a model obtained from Taylor Pruning
with $14M$ parameters resulted in $38\%$ Pixel Accuracy with $0.0001$ learning
rate, whereas the same model reached over $44\%$ with $0.001$ learning rate.
Moreover, some pruned models gave their best results for $0.0005$ learning
rate.

Validation Metric |  Semantic Segmentation |  Depth Estimation |  Surface Normal
---|---|---|---
Pix.Acc.$(\%)$ $\uparrow$ |  mIoU $\uparrow$ |  Abs.Err. $\downarrow$ |  Rel.Err $\downarrow$ |  Mean $\downarrow$ |  Median $\downarrow$
Total Validation Loss |  $42.81$ |  $0.1526$ |  $0.6627$ |  $0.2764$ |  $\mathbf{18.08}$ |  $16.15$
Pixel Accuracy |  $\mathbf{46.56}$ |  $\mathbf{0.1654}$ |  $\mathbf{0.6239}$ |  $\mathbf{0.2527}$ |  $18.51$ |  $\mathbf{15.55}$
Table 3: Results at the best training epoch when different validation metrics
are used.

Therefore, to get the best possible performance from every pruned model, we
trained them on $3$ sets of learning rates which are $0.001$, $0.0005$, and
$0.0001$. We ran all the experiments for $500$ epochs with cosine scheduling
and used Adam [23] optimizer for training. We summarize results for a subset
of models in Table 4.2. Each row shows the performance comparison between
pruned models obtained from Taylor pruning and CosPrune which have the same
number of parameters. We have also given the desired learning rate for each
model. It seems that Taylor pruned models favor $0.001$ learning rate with a
few favoring $0.0001$, and $0.0005$. On the other hand, CosPruned models favor
smaller learning rates of $0.0001$, and $0.0005$. The corresponding plots for
the 3 tasks are shown in Figure 2.

From the results in Table 4.2, we can see that the models obtained from
different pruning methods lead to very similar performance across all the
tasks when the number of parameters is roughly equal. Figure 2, also shows
this similar trend where the winning method keeps on fluctuating with changing
model parameters. In some cases, Taylor pruned models perform better than
CosPruned models and in other cases, it is vice-versa but the results are
still very close for both methods across all tasks. We have also compared the
results of the individually pruned models with the results from the
corresponding iterative pruning iterations in Figure 2. It can be seen that at
very low parameters, the iterative pruning results become worse than the
results of the models that were trained solely from scratch except in the case
of normal estimation where the Angle Mean is slightly better in the case of
the iterative CosPrune method.

Analyzing the Results: We observed that if we consider the pruned models
obtained from two different pruning methods, randomly initialize them and re-
train them with their optimal learning rate, they give similar results for the
same number of parameters. At some parameter levels, the pruned model obtained
from Taylor pruning performed better but at some other parameter levels, the
pruned model obtained from CosPrune gave better results. But there is only a
slight difference in the results which could be induced by the randomness of
the training process. By looking at these results, we can conclude that
different architecture configurations with a similar number of parameters do
not play a major role in the final performance across tasks as long as the
models are trained with their optimal training settings.

Furthermore, we saw that the structured iterative pruning was biased towards
one pruning criterion over the other as Figure 1 shows that the CosPrune
models outperformed Taylor pruned models at the same parameter level. But when
the pruned models were re-trained from scratch, their final performances
became similar and even closer to the unpruned model as seen in Figure 2. But
for both methods, there was a steep drop in the performance at extreme pruning
ratios, which was again mitigated by random initialization and re-training
that led to better performance even at those extreme pruning ratios. This
might be happening because, in the case of iterative structured pruning, the
inherited trained weights from previous pruning iterations may act as bad
initialization for the pruned models at successive iterations [30]. Even if
those weights were optimal for the earlier unpruned model versions, they might
not be good for the current version because of changes in the architecture
after pruning and it can be difficult for the optimizer to find another good
local minimum.

###  4.3 Additional Observation for NYUv2 Dataset

Best Validation Metric: In the case of multi-task learning, total validation
loss is traditionally considered a good validation metric that implies that
the model is performing well on all the tasks collectively. However, in our
experiments on the NYUv2 dataset, we observed that most of the time the least
total validation loss failed to achieve the best model performance. In many
instances, the Pixel Accuracy was low at the epoch with the best validation
loss but it reached higher at some other epochs but with a slightly higher
loss. However, the results for Depth Estimation and Surface Normal Prediction
were similar at these different epochs. To verify this further, we tracked the
best Pixel Accuracy in a training run and also tracked the best total
validation loss in a separate training run but done with the same settings of
hyperparameters. We trained the model 3 times to accommodate for the variance
and show these results in Table 3. We can see that all the metric values
except the Angle Mean are better when the model at the highest Pixel Accuracy
is considered as compared to the model at least validation loss. This is also
quite reasonable because the Semantic Segmentation task is the most difficult
task and the scale of its loss function is higher than the other two tasks.
Therefore, for getting the best model, we considered the best Pixel Accuracy
score instead of the total validation loss.

##  5 Conclusions

In this work, we used two different structured pruning methods to compress
deep multi-task neural networks. We are first to comprehensively analyze their
effectiveness in pruning such networks. Specifically, we showed that different
architectural configurations of the pruned models can give similar results
regardless of the pruning method used if the number of parameters is the same.
We also showed that iterative structured pruning may not be the best strategy
to compress deep multi-task models. They might favor one pruning strategy over
another. But regardless of the pruning strategy used, the performance of the
pruned models can take a steep drop after certain iterations. However, we also
showed that after random initialization and re-training those models with
their respective optimal learning rates, they can give much higher performance
across all the tasks.

## References

  * [1] Milad Abdollahzadeh, Touba Malekzadeh, and Ngai-Man Man Cheung.  Revisit multimodal meta-learning through the lens of multi-task learning.  Advances in Neural Information Processing Systems, 34:14632–14644, 2021.
  * [2] R Caruana.  Multitask learning: A knowledge-based source of inductive bias1.  In Proceedings of the Tenth International Conference on Machine Learning, pages 41–48. Citeseer, 1993.
  * [3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Semantic image segmentation with deep convolutional nets and fully connected crfs.  arXiv preprint arXiv:1412.7062, 2014.
  * [4] Hanjing Cheng, Zidong Wang, Lifeng Ma, Xiaohui Liu, and Zhihui Wei.  Multi-task pruning via filter index sharing: A many-objective optimization approach.  Cognitive Computation, 13:1070–1084, 2021.
  * [5] Hanjing Cheng, Zidong Wang, Lifeng Ma, Xiaohui Liu, and Zhihui Wei.  Multi-task pruning via filter index sharing: A many-objective optimization approach.  Cognitive Computation, 13:1070–1084, 2021.
  * [6] Jean-Antoine Désidéri.  Multiple-gradient descent algorithm (mgda) for multiobjective optimization.  Comptes Rendus Mathematique, 350(5-6):313–318, 2012.
  * [7] Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding.  Resrep: Lossless cnn pruning via decoupling remembering and forgetting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4510–4520, 2021.
  * [8] David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In Proceedings of the IEEE international conference on computer vision, pages 2650–2658, 2015.
  * [9] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  Advances in neural information processing systems, 27, 2014.
  * [10] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen.  Rigging the lottery: Making all tickets winners.  In International Conference on Machine Learning, pages 2943–2952. PMLR, 2020.
  * [11] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  arXiv preprint arXiv:1803.03635, 2018.
  * [12] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin.  The lottery ticket hypothesis at scale.  arXiv preprint arXiv:1903.01611, 2020.
  * [13] Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung-Yi Lin.  Multi-task self-training for learning general representations.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8856–8865, 2021.
  * [14] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei.  Dynamic task prioritization for multitask learning.  In Proceedings of the European conference on computer vision (ECCV), pages 270–287, 2018.
  * [15] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  Advances in neural information processing systems, 28, 2015.
  * [16] Babak Hassibi and David Stork.  Second order derivatives for network pruning: Optimal brain surgeon.  Advances in neural information processing systems, 5, 1992.
  * [17] Xiaoxi He, Dawei Gao, Zimu Zhou, Yongxin Tong, and Lothar Thiele.  Pruning-aware merging for efficient multitask inference.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 585–595, 2021.
  * [18] Xiaoxi He, Zimu Zhou, and Lothar Thiele.  Multi-task zipping via layer-wise neuron sharing.  Advances in Neural Information Processing Systems, 31, 2018.
  * [19] Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun-Yuan Kung.  Chex: channel exploration for cnn model compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287–12298, 2022.
  * [20] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439–1449, 2021.
  * [21] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482–7491, 2018.
  * [22] Donghyun Kim, Tian Lan, Chuhang Zou, Ning Xu, Bryan A Plummer, Stan Sclaroff, Jayan Eledath, and Gerard Medioni.  Mila: Multi-task learning from videos via efficient inter-frame attention.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2219–2229, 2021.
  * [23] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [24] Yann LeCun, John Denker, and Sara Solla.  Optimal brain damage.  Advances in neural information processing systems, 2, 1989.
  * [25] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr.  Snip: Single-shot network pruning based on connection sensitivity.  arXiv preprint arXiv:1810.02340, 2018.
  * [26] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.  Pruning filters for efficient convnets.  arXiv preprint arXiv:1608.08710, 2016.
  * [27] Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, and Luc Van Gool.  Revisiting random channel pruning for neural network compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 191–201, 2022.
  * [28] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu.  Conflict-averse gradient descent for multi-task learning.  Advances in Neural Information Processing Systems, 34:18878–18890, 2021.
  * [29] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy.  The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.  arXiv preprint arXiv:2202.02643, 2022.
  * [30] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell.  Rethinking the value of network pruning.  arXiv preprint arXiv:1810.05270, 2018.
  * [31] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [32] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [33] Christos Louizos, Karen Ullrich, and Max Welling.  Bayesian compression for deep learning.  Advances in neural information processing systems, 30, 2017.
  * [34] Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris.  Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5334–5343, 2017.
  * [35] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994–4003, 2016.
  * [36] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz.  Importance estimation for neural network pruning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272, 2019.
  * [37] John Phillips, Julieta Martinez, Ioan Andrei Bârsan, Sergio Casas, Abbas Sadat, and Raquel Urtasun.  Deep multi-task learning for joint localization, perception, and prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4679–4689, 2021.
  * [38] Sebastian Ruder12, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard.  Learning what to share between loosely related tasks.  arXiv preprint arXiv:1705.08142, 2017.
  * [39] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  Advances in neural information processing systems, 31, 2018.
  * [40] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  ECCV (5), 7576:746–760, 2012.
  * [41] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [42] Shagun Sodhani, Amy Zhang, and Joelle Pineau.  Multi-task reinforcement learning with context-based representations.  In International Conference on Machine Learning, pages 9767–9779. PMLR, 2021.
  * [43] Xinglong Sun, Ali Hassani, Zhangyang Wang, Gao Huang, and Humphrey Shi.  Disparse: Disentangled sparsification for multitask model compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12382–12392, 2022.
  * [44] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko.  Adashare: Learning what to share for efficient deep multi-task learning.  Advances in Neural Information Processing Systems, 33:8728–8740, 2020.
  * [45] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu.  Distral: Robust multitask reinforcement learning.  Advances in neural information processing systems, 30, 2017.
  * [46] Christopher Williams, Stefan Klanke, Sethu Vijayakumar, and Kian Chai.  Multi-task gaussian process learning of robot inverse dynamics.  Advances in neural information processing systems, 21, 2008.
  * [47] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli.  Multi-task reinforcement learning: a hierarchical bayesian approach.  In Proceedings of the 24th international conference on Machine learning, pages 1015–1022, 2007.
  * [48] Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang.  Multi-task reinforcement learning with soft modularization.  Advances in Neural Information Processing Systems, 33:4767–4777, 2020.
  * [49] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning.  Advances in Neural Information Processing Systems, 33:5824–5836, 2020.
  * [50] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In Conference on robot learning, pages 1094–1100. PMLR, 2020.
  * [51] Maohua Zhu, Tao Zhang, Zhenyu Gu, and Yuan Xie.  Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus.  In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pages 359–371, 2019.


11institutetext: Department of Engineering Sciences, University of Oxford, OX3
7DQ Oxford, UK 22institutetext: Oxford Suzhou Centre for Advanced Research,
University of Oxford, Suzhou 215123, Jiangsu, China

# Masks and Manuscripts: Advancing Medical Pre-training with End-to-End
Masking and Narrative Structuring

Shreyank N Gowda 11 David A. Clifton 1122

###### Abstract

Contemporary medical contrastive learning faces challenges from inconsistent
semantics and sample pair morphology, leading to dispersed and converging
semantic shifts. The variability in text reports, due to multiple authors,
complicates semantic consistency. To tackle these issues, we propose a two-
step approach. Initially, text reports are converted into a standardized
triplet format, laying the groundwork for our novel concept of “observations”
and “verdicts”. This approach refines the Entity, Position, Exist triplet into
binary questions, guiding towards a clear “verdict”. We also innovate in
visual pre-training with a Meijering-based masking, focusing on features
representative of medical images’ local context. By integrating this with our
text conversion method, our model advances cross-modal representation in a
multimodal contrastive learning framework, setting new benchmarks in medical
image analysis.

##  1 Introduction

Deep learning has seen significant achievements in image classification [14,
17] and action recognition [13, 21, 10], driving forward the capabilities of
computer vision applications. With the swift advancements in deep learning,
computer-aided diagnosis in the medical field has made notable progress
through various models. However, these models, predominantly trained on
specific anatomical or disease categories, demand costly data annotation and
re-training when a new disease emerges [22], limiting their practical utility.
While deep learning has flourished due to large-scale labeled natural image
datasets [7], the annotation of medical images remains time-consuming and
expensive. A common strategy involves pre-training on large datasets like
ImageNet [7] and then fine-tuning on specialized medical datasets [37]. While
research has shown excellent performance in the natural image [11, 31, 39]
using this strategy, it often falls short in terms of generalized performance
due to the domain differences between natural and medical images.

Medical image analysis stands at the crossroads of technological advancements
and clinical applications. The rapid growth of medical imaging modalities and
the increasing volume of patient data offer unprecedented opportunities to
leverage sophisticated machine learning techniques. Yet, the complexity of
this data—rich in variability, granularity, and intricacies—poses unique
challenges for the development of effective and reliable models. Adding to the
challenge, the associated textual data, which often accompanies these images
in the form of clinical notes and radiological reports, can introduce
inconsistencies due to varied semantics and writing styles across different
authors [9], potentially leading to significant noise in the learning process
and compromising model reliability.

Self-supervised learning has emerged as a powerful paradigm for pre-training
deep neural networks without relying on human annotations, leveraging inherent
structure in the data to learn useful representations in a scalable manner
despite their associated costs [30, 12]. Drawing on these advancements, this
paper introduces a novel approach that combines self-supervised learning with
the nuanced requirements of medical data processing. We propose a two-step
conversion process for textual data into a standardized triplet format to
refine representations and an innovative visual pre-training strategy that
employs Meijering-based masking for improved feature extraction [27].

Furthermore, vision-language pre-training (VLP) models like CLIP [30] have
shown success in learning joint multimodal representations but struggle with
the unique challenges presented by medical images and unstructured reports
[38]. We address these challenges by employing a filtering-based approach for
preprocessing medical images, enhancing the performance of deep learning
models through classical techniques like Meijering’s vesselness filter [27,
32] and adapting these for novel masking strategies to facilitate cleaner
reconstructions and improved model performance.

Incorporating external knowledge into medical models has shown promise in
mimicking medical decision-making processes and enhancing model training with
domain-specific clinical knowledge [36]. Building on the concept of MedKLIP
[38], we extend the extraction of key data from radiology reports into
triplets and encode it with domain knowledge to improve masked generative pre-
training [38].

This paper aims to bridge the gap between the raw, multifaceted medical data
and the refined, consistent representations needed for effective machine
learning applications, pushing the boundaries of medical pre-training by
tailoring advanced techniques specifically for the medical domain. Through an
amalgamation of self-supervised learning, VLP models, preprocessing
techniques, and the incorporation of medical knowledge, we propose a
comprehensive framework for enhancing the performance and reliability of
machine learning models in medical image analysis and report generation.

##  2 Methodology

Figure 1: An overview of the architecture for integrated modeling of masked
medical visual and linguistic data. The blue and green pathways represent the
flow of information for the reconstruction of images and text, respectively.
The dashed lines show the intermodal contribution of exposed signals for the
creation of attention.

In Figure 1, we outline our approach, adopting MaskVLM’s [25] architecture.
Our innovations primarily lie in visual and textual data pre-processing,
rather than the architecture itself, which serves as a foundation for our
training methodology. We employ transformer-based encoders [8, 34] for
concurrent image and text analysis. The image encoder extracts features as
vector sets, including a class token, while the text encoder processes text
with a start token. Through self-attention and cross-modality encoders
featuring cross-attention, we enhance inter-modal integration, culminating in
accurate image descriptions. Further details and the rationale for our methods
are elaborated in subsequent sections.

###  2.1 Masked Image Modeling

Masked Image Modeling (MIM) [15] revolutionizes self-supervised learning by
enabling models to infer visual contexts without extensive labeled data,
thereby enhancing efficiency and generalization. However, for fine-grained
medical data, random masking leads to unclear reconstructions. Despite being
superior to random initialization [41], improvements are needed due to the
reliance on local features in medical images. Utilizing ridge filters, like
the Meijering filter, converts X-ray images into intermediate forms for
reconstruction, addressing the fine-grained nature of medical data [27, 32].
This approach, supported by qualitative analysis in Figure 2 and further in
Table 7 and Table 7, empirically outperforms random masking as evidenced by
our ablation study.

Figure 2: Comparing masking strategies: (a) Masking random patches for
reconstruction often results in blurry outputs lacking in detail. (b)
Filtering the image before reconstruction preserves fine-grained details,
leading to higher resolution outcomes.

####  2.1.1 Meijering Filter Overview

The Meijering filter enhances tubular structures in images via the Hessian
matrix’s eigenvalues [27]. For an image $I(x,y)$, the Hessian matrix
$\mathbf{H}$ and its eigenvalues $\lambda_{1},\lambda_{2}$ guide the filter
response $R$, which is zero for $\lambda_{2}>0$ and
$\sqrt{\lambda_{1}^{2}+\lambda_{2}^{2}}$ otherwise. By adjusting the scale
$\sigma$, the filter adapts to features of various sizes, optimizing the
response $R_{\sigma}$ across scales to highlight relevant structures, thus
aiding in medical image analysis.

###  2.2 Manuscript Generation

Radiology reports in publicly available datasets vary due to individual
writing styles, affecting the consistency of sentence embeddings and their use
in contrastive learning. We adopt triplet extraction from MedKLIP [38] for
uniformity, converting extracted triplets into new reports. This approach
ensures semantic consistency, facilitating the extraction of positive and
negative pairs, and allows for binary answers in observations and verdicts,
useful for masked pre-training. Figure 3 provides an overview of our
methodology.

Figure 3: Our report generation process starts with the triplet extraction
method from MedKLIP, as outlined in part (a). Instead of adopting the
Knowledge-enhanced Triplet Encoding, we transform these into a textual report,
embedding it sentence by sentence. This approach facilitates masked pre-
training by providing binary labels for each observation and the verdict.

####  2.2.1 Report Pre-processing

Following MedKLIP [38], we apply RadGraph [20], a medical Named Entity
Recognition (NER) tool, to streamline radiology reports into triplets. The NER
identifies and classifies medical terms as "entity" (clinical findings) or
"position" (body location), and assigns an "exist" status to indicate the
presence, absence, or uncertainty of clinical observations. This process,
generating triplets like entity, position, exist, simplifies sentence
structures in reports while retaining crucial information, leveraging the
Triplet Extraction method in medical Visual Language Processing (VLP).

###  2.3 Conditional Reconstruction

Our method aims to reconstruct an original signal in one modality (image or
text) from its masked version using unmasked information from the other
modality. We follow notations used by MaskVLM [25] as we build on them.
Specifically, for a medical image $I$ and its masked form $I_{m}$, with
accompanying unmasked text $T$, the goal is to retrieve $I$ utilizing
($I_{m}$, $T$). In reverse, with an unmasked image $I$ and masked text
$T_{m}$, the objective is to recover $T$ with $(I,T_{m})$. We employ image and
text encoders $f_{im}$ and $f_{txt}$ for feature extraction. The cross-
modality encoders $g_{im}$ and $g_{txt}$, leveraging context from unmasked
modalities, reconstruct the signals through decoders $d_{im}$ and $d_{txt}$
for images and texts, respectively. This reconstruction process is optimized
by a multimodal joint modeling loss $\mathcal{L}_{MVLM}$, promoting focus on
unmasked modality features for accurate reconstruction:

$\displaystyle\mathcal{L}_{\text{MVLM}}=$
$\displaystyle\mathbb{E}_{(I,T)\sim\mathcal{D}}\left[\mathcal{H}\left(y^{M}_{T},\phi_{\text{txt}}^{M}(I,T|m)\right)+\left\|\frac{1}{\Omega(I^{M})}\left(I^{M}-\phi^{M}_{\text{im}}(I_{m},T^{\prime})\right)\right\|_{1}\right]$
(1)

where $\phi_{\text{txt}}$ and $\phi_{\text{im}}$ represent the decoding
processes for text and image modalities, respectively, focusing on masked
regions. This process enables learning from cross-modal interactions to
enhance reconstruction accuracy.

###  2.4 Multi-Modal Alignment

To enhance masked signal reconstruction, we incorporate cross-modal alignment
through two additional objectives. The first, image-text contrastive (ITC)
learning [30], projects image and text encoder outputs into a shared space,
using separate fully-connected layers. This process is formulated as
minimizing the contrastive loss $\mathcal{L}_{ITC}$, encouraging agreement
between matched image-text pairs in embedding space. However, variability in
radiology report styles may affect the identification of positive and negative
pairs, as observed in our ablation study.

The ITC loss for the k-th image-text pair, with batch size $N$ and temperature
$\tau$, is given by:

$\displaystyle\mathcal{L}_{ITC}=-\frac{1}{N}\sum_{k=1}^{N}\left[\log\frac{\exp(z^{k}_{\text{im}}\cdot
z^{k}_{\text{txt}}/\tau)}{\sum_{n=1}^{N}\exp(z^{n}_{\text{im}}\cdot
z^{k}_{\text{txt}}/\tau)}+\log\frac{\exp(z^{k}_{\text{im}}\cdot
z^{k}_{\text{txt}}/\tau)}{\sum_{n=1}^{N}\exp(z^{k}_{\text{im}}\cdot
z^{n}_{\text{txt}}/\tau)}\right],$ (2)

The second objective, image-text matching (ITM) classification [5], predicts
the alignment of image-text pairs by element-wise multiplication of cross-
modality encoder outputs, followed by classification. The ITM loss
$\mathcal{L}_{ITM}$ ensures supervised alignment across modalities:

$\mathcal{L}_{ITM}=\mathbb{E}_{(I,T)\sim
D}\left[\mathcal{H}(y_{ITM},g^{itm}_{\text{cross}}(z^{cross}_{\text{im}},z^{cross}_{\text{txt}}))\right].$
(3)

The total pre-training loss $\mathcal{L}$ combines all losses:

$\mathcal{L}=\mathcal{L}_{MVML}+\mathcal{L}_{ITC}+\mathcal{L}_{ITM}.$ (4)

Model fine-tuning is performed for each downstream task over a few epochs,
alongside zero-shot evaluations following prior work [6, 38].

##  3 Experimental Analysis

###  3.1 Datasets

We use datasets that have been used in recent approaches [18, 6, 38] namely:
MIMIC-CXR v2 [23], RSNA Pneumonia Detection [33], SIIM-ACR Pneumothorax [26],
NIH Chest X-Ray Dataset [35], CheXpert [19], COVIDx CXR [29] and Edema
Severity [2]. We follow the evaluation protocol of MedKlip [38].

###  3.2 Implementation Details

The model utilizes a ViT-B pre-trained on ImageNet for image processing and
ClinicalBERT for text, resizing images to 224x224 and setting a common latent
dimension of 768. Training is conducted on 4 NVIDIA Tesla V100 GPUs with a
batch size of 128, using AdamW optimizer with a 0.05 weight decay. Unlike
fixed masking ratios, the model adapts the mask ratio based on the input,
using a 15% ratio for text that increases to 30% when paired with images. The
model is pre-trained for 100 epochs, considerably less than the 800 epochs in
similar studies, and fine-tuned for 10 epochs with a specific learning rate
strategy, including a warm-up to 3e-4 followed by a cosine scheduler for the
pre-training phase, and fixed learning rates of 1e-5 during fine-tuning.
Extending training to 300, 500, or 800 epochs did not improve accuracy.

###  3.3 Classification

####  3.3.1 Semi and Fully Supervised

We conduct semi and fully supervised classification on RSNA Pneumonia, SIIM-
ACR, and CheXpert datasets, varying labeled data from 1% to 100%. Results,
including mean and standard deviation over 5 runs using PRIOR [6], are in
Table 1. M&M surpasses SOTA by up to 2.64%.

| RSNA Pneumonia | SIIM-ACR | CheXpert
---|---|---|---
Methods | $1\%$ | $10\%$ | $100\%$ | $1\%$ | $10\%$ | $100\%$ | $1\%$ | $10\%$ | $100\%$
MoCo [16] | $82.33$ | $85.22$ | $87.90$ | $75.49$ | $81.01$ | $88.43$ | $78.00$ | $86.27$ | $87.24$
SimCLR [4] | $80.18$ | $84.60$ | $88.07$ | $74.97$ | $83.21$ | $88.72$ | $67.41$ | $86.74$ | $87.97$
ConVIRT [40] | $83.98$ | $85.62$ | $87.61$ | $84.17$ | $85.66$ | $91.50$ | $85.02$ | $87.58$ | $88.21$
GLoRIA [18] | $84.12$ | $86.83$ | $89.13$ | $85.05$ | $88.51$ | $92.11$ | $83.61$ | $87.40$ | $88.34$
BioViL [1] | $81.95$ | $85.37$ | $88.62$ | $79.89$ | $81.62$ | $90.48$ | $80.77$ | $87.56$ | $88.41$
LoVT [28] | $85.51$ | $86.53$ | $89.27$ | $85.47$ | $88.50$ | $92.16$ | $85.13$ | $88.05$ | $88.27$
PRIOR [6] | $85.74$ | $87.08$ | $89.22$ | $87.27$ | $89.13$ | $92.39$ | $86.16$ | $88.31$ | $88.61$
MedKLIP [38] | $87.31$ | $87.99$ | $89.31$ | $85.27$ | $90.71$ | $91.88$ | $86.24$ | $88.14$ | $88.68$
M&M (Ours) | 88.11 | 89.44 | 91.91 | 88.81 | 91.15 | 93.88 | 88.45 | 90.02 | 90.88

Table 1: Comparing supervised classification results obtained by fine-tuning
on RSNA Pneumonia Detection [33], SIIM [26] and CheXpert [19]. All methods are
trained on different portions of the training set from 1% to 100% and
evaluated using AUC-ROC. Following PRIOR [6] each reported value is the
average of five runs along with the standard deviation.

| RSNA Pneumonia | SIIM-ACR | NIH Chest X-Ray
---|---|---|---
Methods | AUC↑ | F1↑ | ACC↑ | AUC↑ | F1↑ | ACC↑ | AUC↑ | F1↑ | ACC↑
ConVIRT [40] | 80.42 | 58.42 | 76.11 | 64.31 | 43.29 | 57.00 | 61.01 | 16.28 | 71.02
GLoRIA [18] | 71.45 | 49.01 | 71.29 | 53.42 | 38.23 | 40.47 | 66.10 | 17.32 | 77.00
BioViL [1] | 82.80 | 58.33 | 76.69 | 70.79 | 48.55 | 69.09 | 69.12 | 19.31 | 79.16
PRIOR [6] | 85.58 | 62.91 | 77.85 | 86.62 | 70.11 | 84.44 | 74.51 | 23.29 | 84.41
MedKLIP [38] | 86.94 | 63.42 | 80.02 | 89.24 | 68.33 | 84.28 | 76.76 | 25.25 | 86.19
M&M (Ours) | 88.91 | 66.58 | 83.14 | 91.15 | 71.58 | 86.15 | 77.92 | 27.55 | 88.52

Table 2:  Comparing recent state-of-the-art methods on zero-shot
classification task. We use AUC, F1 and ACC scores for comparison. Following
MedKLIP [38] for evaluation on NIH Chest X-Ray, the metrics all refer to the
macro average on the 14 diseases.

####  3.3.2 Zero-Shot

We assess zero-shot classification on RSNA Pneumonia, SIIM-ACR, and NIH Chest
X-Ray datasets with state-of-the-art models, showcasing our model’s
generalization across various clinical sources. Our approach achieves
consistent performance enhancements, attributed to domain adaptation as
discussed in MedKLIP [38]. Results in Table 2 show our M&M model outperforming
existing methods by up to 3.16%, following pre-training on MIMIC-CXR and up to
a 3.29% improvement on the novel COVID-19 disease in Table 4.

Methods | AUC↑ | F1↑ | ACC↑
---|---|---|---
ConVIRT [40] | 52.08 | 69.02 | 52.66
GloRIA [18] | 66.59 | 70.07 | 60.83
BioViL [1] | 53.82 | 69.10 | 53.75
MedKLIP [38] | 73.96 | 76.70 | 70.06
M&M (Ours) | 75.15 | 77.89 | 73.35
Table 3: Performance on Covid-19 CXR.

Methods | AUC↑ | F1↑ | ACC↑
---|---|---|---
ConVIRT [40] | 77.00 | 56.76 | 69.19
GLoRIA [18] | 77.74 | 57.98 | 71.45
BioViL [1] | 75.40 | 55.72 | 69.14
MedKLIP [38] | 78.98 | 58.26 | 72.80
M&M (Ours) | 80.71 | 60.18 | 73.91
Table 4: Performance comparison on Edema severity grading multi-class task.

###  3.4 Grading

In addition to diagnosis, determining disease severity is essential. We refine
pre-trained features for a multi-category classification task using the Edema
Severity dataset, which includes classes 0 to 3, each indicating a different
severity stage. The average scores across all severity levels are shown in
Table 4.

###  3.5 Segmentation

Table 5 presents our fine-tuning experiments for segmenting three distinct
diseases, where we utilize 1%, 10%, and 100% of the available data. Regardless
of the varying image distributions associated with each disease, our
techniques consistently outperform current leading methods. We see significant
gains in particular when data is scarce outperforming previous works by up to
2.96%.

Methods | RSNA Pneumonia | SIIM-ACR | Covid-19
---|---|---|---
1% | 10% | 100% | 1% | 10% | 100% | 1% | 10% | 100%
Scratch | 43.47 | 60.47 | 70.68 | 21.33 | 33.23 | 74.47 | 14.81 | 23.67 | 32.28
ConVIRT [40] | 57.06 | 64.91 | 72.01 | 54.06 | 61.21 | 73.52 | 19.95 | 27.24 | 37.37
GLoRIA [18] | 65.55 | 69.07 | 73.28 | 56.73 | 57.78 | 76.94 | 18.89 | 28.09 | 38.69
BioVil [1] | 68.24 | 70.38 | 72.49 | 62.67 | 69.98 | 78.49 | 21.13 | 32.39 | 41.62
PRIOR [6] | 70.11 | 70.88 | 74.43 | 66.14 | 71.24 | 78.85 | 23.66 | 34.72 | 43.01
MedKLIP [38] | 70.64 | 71.62 | 75.79 | 66.59 | 72.10 | 79.37 | 24.45 | 35.39 | 43.99
M&M (Ours) | 72.28 | 73.11 | 76.68 | 69.55 | 73.47 | 80.28 | 28.25 | 37.32 | 45.04
Table 5: Evaluating Dice scores against leading methods for segmentation
tasks, we analyze diseases with 1%, 10%, 100% labeled data, improving across
all scenarios.

###  3.6 Ablation Study

We assess the effects of different image masking techniques—no masking, random
patch masking [15], attention-guided masking (AttMask)[24], and salient patch
selection (AutoMAE)[3]—and validate our manuscript generation approach. Our
method boosts performance by as much as 8.06%, as Table 7 illustrates. For
report generation, we examine four methods: original reports, extracted
triplets, knowledge-enhanced triplets (KE-Triplet) via MedKLIP, and converting
triplets into manuscripts, using the same masking for all (Table 7). KE-
Triplet leads, surpassing even MedKLIP by up to 1.97%, highlighting our
model’s efficacy on the NIH Chest X-Ray dataset.

Methods | AUC↑ | F1↑ | ACC↑
---|---|---|---
MaskVLM [25] | 58.87 | 14.96 | 66.69
No Masking | 61.48 | 16.33 | 70.54
MAE [15] | 68.84 | 18.85 | 75.59
AttMask [24] | 71.85 | 20.84 | 77.81
AutoMAE [3] | 73.18 | 22.81 | 80.46
M&M (Ours) | 77.92 | 27.55 | 88.52
Table 6: Ablation study on masking strategy, focusing on zero-shot setting to
directly assess learned features.

Methods | AUC↑ | F1↑ | ACC↑
---|---|---|---
Original Report | 69.95 | 20.04 | 77.71
Triplet | 73.48 | 24.42 | 82.89
KE-Triplet | 76.84 | 26.11 | 86.55
M&M (Ours) | 77.92 | 27.55 | 88.52
Table 7: Ablation study on manuscript generation method selection, focusing
solely on the zero-shot setting to evaluate the effectiveness of learned
features.

##  4 Conclusion

Medical contrastive learning is challenged by inconsistent semantics and
morphology in sample text pairs, causing semantic drifts. Report variability
from multiple authors further complicates semantic interpretation. To address
this, we propose a two-step approach. First, standardize reports into
triplets. Then, convert triplets into binary “observations” that guide towards
“verdicts”. For images, we use Meijering-based masking for pre-training
instead of random masking to capture local context critical for medical
images. Our multimodal contrastive learning framework, combining the
standardized text and tailored visual representations, advances medical image
analysis, achieving new state-of-the-art performances on multiple different
downstream tasks. While the filter improves the performance in the discussed
X-ray data, future work would be to test this on other modalities, such as
MRI, which is noisier and more heterogeneous.

####  4.0.1

David A. Clifton was supported by the Pandemic Sciences Institute at the
University of Oxford; the National Institute for Health Research (NIHR) Oxford
Biomedical Research Centre (BRC); an NIHR Research Professorship; a Royal
Academy of Engineering Research Chair; the Wellcome Trust funded VITAL project
(grant 204904/Z/16/Z); the EPSRC (grant EP/W031744/1); and the InnoHK Hong
Kong Centre for Cerebro-cardiovascular Engineering (COCHE).

## References

  * [1] Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: Eur. Conf. Comput. Vis. pp. 1–21. Springer (2022)
  * [2] Chauhan, G., Liao, R., Wells, W., Andreas, J., Wang, X., Berkowitz, S., Horng, S., Szolovits, P., Golland, P.: Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment. In: International Conference on Medical Image Computing and Computer Assisted Intervention. pp. 529–539. Springer (2020)
  * [3] Chen, H., Zhang, W., Wang, Y., Yang, X.: Improving masked autoencoders by learning where to mask. arXiv:2303.06583 (2023)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Int. Conf. Mach. Learn. pp. 1597–1607. PMLR (2020)
  * [5] Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Universal image-text representation learning. In: Eur. Conf. Comput. Vis. pp. 104–120. Springer (2020)
  * [6] Cheng, P., Lin, L., Lyu, J., Huang, Y., Luo, W., Tang, X.: Prior: Prototype representation joint learning from medical images and reports. In: Int. Conf. Comput. Vis. pp. 21361–21371 (2023)
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 248–255. Ieee (2009)
  * [8] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition scale. arXiv:2010.11929 (2020)
  * [9] Flanders, A.E., Lakhani, P.: Radiology reporting and communications: a look forward. Neuroimaging Clinics 22(3), 477–496 (2012)
  * [10] Gowda, S.N.: Human activity recognition using combinatorial deep belief networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 1–6 (2017)
  * [11] Gowda, S.N.: Synthetic sample selection for generalized zero-shot learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 58–67 (2023)
  * [12] Gowda, S.N., Hao, X., Li, G., Sevilla-Lara, L., Gowda, S.N.: Watt for what: Rethinking deep learning’s energy-performance relationship. arXiv preprint arXiv:2310.06522 (2023)
  * [13] Gowda, S.N., Rohrbach, M., Sevilla-Lara, L.: Smart frame selection for action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 1451–1459 (2021)
  * [14] Gowda, S.N., Yuan, C.: Colornet: Investigating the importance of color spaces for image classification. In: Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14. pp. 581–596. Springer (2019)
  * [15] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 16000–16009 (2022)
  * [16] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 9729–9738 (2020)
  * [17] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. (June 2016)
  * [18] Huang, S.C., Shen, L., Lungren, M.P., Yeung, S.: Gloria: A multimodal global-local representation learning framework for medical image recognition. In: Int. Conf. Comput. Vis. pp. 3942–3951 (2021)
  * [19] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: AAAI. vol. 33, pp. 590–597 (2019)
  * [20] Jain, S., Agrawal, A., Saporta, A., Truong, S.Q., Duong, D.N., Bui, T., Chambon, P., Zhang, Y., Lungren, M.P., Ng, A.Y., et al.: Radgraph: Extracting clinical entities and relations from radiology reports. arXiv:2106.14463 (2021)
  * [21] Jhuang, H., Gall, J., Zuffi, S., Schmid, C., Black, M.J.: Towards understanding action recognition. In: Proceedings of the IEEE international conference on computer vision. pp. 3192–3199 (2013)
  * [22] Ji, W., Chung, A.C.: Unsupervised domain adaptation for medical image segmentation using transformer with meta attention. IEEE Trans. Med. Imag. (2023)
  * [23] Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data (2019)
  * [24] Kakogeorgiou, I., Gidaris, S., Psomas, B., Avrithis, Y., Bursuc, A., Karantzalos, K., Komodakis, N.: What to hide from your students: Attention-guided masked image modeling. In: Eur. Conf. Comput. Vis. pp. 300–318. Springer (2022)
  * [25] Kwon, G., Cai, Z., Ravichandran, A., Bas, E., Bhotika, R., Soatto, S.: Masked vision and language modeling for multi-modal representation learning. In: Int. Conf. Learn. Represent. (2022)
  * [26] for imaging informatics in medicine, S.: Siim-acr pneumothorax segmentation (2019), https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation
  * [27] Meijering, E., Jacob, M., Sarria, J.C., Steiner, P., Hirling, H., Unser, M.: Neurite tracing in fluorescence microscopy images using ridge filtering and graph searching: principles and validation. In: ISBI. pp. 1219–1222. IEEE (2004)
  * [28] Müller, P., Kaissis, G., Zou, C., Rueckert, D.: Joint learning of localized representations from medical images and reports. In: Eur. Conf. Comput. Vis. pp. 685–701. Springer (2022)
  * [29] Pavlova, M., Terhljan, N., Chung, A.G., Zhao, A., Surana, S., Aboutalebi, H., Gunraj, H., Sabri, A., Alaref, A., Wong, A.: Covid-net cxr-2: An enhanced deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Frontiers in Medicine 9, 861680 (2022)
  * [30] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Int. Conf. Mach. Learn. pp. 8748–8763. PMLR (2021)
  * [31] Romera-Paredes, B., Torr, P.: An embarrassingly simple approach to zero-shot learning. In: International conference on machine learning. pp. 2152–2161. PMLR (2015)
  * [32] Sato, Y., Nakajima, S., Shiraga, N., Atsumi, H., Yoshida, S., Koller, T., Gerig, G., Kikinis, R.: 3-d multi-scale line filter for segmentation and visualization of curvilinear structures in medical images. Medical image analysis (1998)
  * [33] Shih, G., Wu, C.C., Halabi, S.S., Kohli, M.D., Prevedello, L.M., Cook, T.S., Sharma, A., Amorosa, J.K., Arteaga, V., Galperin-Aizenberg, M., et al.: Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence 1(1) (2019)
  * [34] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Adv. Neural Inform. Process. Syst. 30 (2017)
  * [35] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 2097–2106 (2017)
  * [36] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Tienet:text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 9049–9058 (2018)
  * [37] Wen, Y., Chen, L., Deng, Y., Zhou, C.: Rethinking pre-training on medical imaging. Journal of Visual Communication and Image Representation 78, 103145 (2021)
  * [38] Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W.: Medklip: Medical knowledge enhanced language-image pre-training. Int. Conf. Comput. Vis. (2023)
  * [39] Zhang, L., Xiang, T., Gong, S.: Learning a deep embedding model for zero-shot learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2021–2030 (2017)
  * [40] Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference. pp. 2–25. PMLR (2022)
  * [41] Zhou, L., Liu, H., Bae, J., He, J., Samaras, D., Prasanna, P.: Self pre-training with masked autoencoders for medical image classification and segmentation. In: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI). pp. 1–6. IEEE (2023)