Further author information: (Send correspondence to Ali Vosoughi)
Ali Vosoughi: E-mail<EMAIL_ADDRESS>
# Cross Modal Global Local Representation Learning from Radiology Reports and
X-Ray Chest Images

Nathan Hadjiyski∗ Department of Computer Science, University of Rochester, NY,
USA  Ali Vosoughi∗ Department of Electrical and Computer Engineering,
University of Rochester, NY, USA  Axel Wismüller  Department of Electrical and
Computer Engineering, University of Rochester, NY, USA  Department of Imaging
Sciences, University of Rochester, NY, USA  Department of Biomedical
Engineering, University of Rochester, NY, USA  Faculty of Medicine and
Institute of Clinical Radiology, Ludwig Maximilian University, Munich, Germany

###### Abstract

Deep learning models can be applied successfully in real-work problems;
however, training most of these models requires massive data. Recent methods
use language and vision, but unfortunately, they rely on datasets that are not
usually publicly available. Here we pave the way for further research in the
multimodal language-vision domain for radiology. In this paper, we train a
representation learning method that uses local and global representations of
the language and vision through an attention mechanism and based on the
publicly available Indiana University Radiology Report (IU-RR) dataset.
Furthermore, we use the learned representations to diagnose five lung
pathologies: atelectasis, cardiomegaly, edema, pleural effusion, and
consolidation. Finally, we use both supervised and zero-shot classifications
to extensively analyze the performance of the representation learning on the
IU-RR dataset. Average Area Under the Curve (AUC) is used to evaluate the
accuracy of the classifiers for classifying the five lung pathologies. The
average AUC for classifying the five lung pathologies on the IU-RR test set
ranged from 0.85 to 0.87 using the different training datasets, namely
CheXpert and CheXphoto. These results compare favorably to other studies using
UI-RR. Extensive experiments confirm consistent results for classifying lung
pathologies using the multimodal global local representations of language and
vision information.

###### keywords:

Deep learning, cross-modal learning, computer vision, natural language
processing, zero-shot classification, chest X-ray, radiology reports

##  1 Introduction

Deep learning is prevalent thanks to its success in numerous real-life domains
and surpassing classic techniques in performance, cost, and ease of
applicability. However, despite the computational efficiency of deep learning
models, it requires a massive amount of data, which is scarce and expensive,
especially in radiology. Multimodal learning recently got considerable
attention. Language is a widely available form of data in radiology besides
images. Therefore, one can design deep learning models that use language and
vision in different ways to improve the quality of medical image analysis, for
instance, by using radiology reports to enhance the performance of downstream
tasks through x-ray images.

This paper addresses multimodal representation learning from a combination of
radiology reports and their corresponding medical images. We use x-ray images
and radiology reports of a publicly available dataset and tailor it to learn a
multimodal representation that encompasses local and global language and
visual information. Although many recent methods consider language and vision
in multimodal learning, one major problem in radiology’s vision and language
models is the lack of sufficient datasets, which is well addressed in this
paper. In many instances, the codes are publicly available without
corresponding datasets, preventing further developments of the methods in the
community [1].

In this paper, we learn multimodal representations of a publicly available
dataset of IU-RR and modify and clean the dataset for learning global and sub-
regional features of both images and radiology reports. We use a global-local
representation framework that attends to both the image-sub-regions and word-
piece language and global representations of vision and language modalities.
Therefore, as a direct result of this research, radiology reports can be used
to attend to related parts in the medical image and learn representations
encompassing local and global details of x-ray images and corresponding
descriptions. The architecture of the method is shown in Fig. 1.

The method here uses a small dataset of Indiana University Radiology Reports
(IU-RR) [2]. Despite its smaller size than other datasets, such as Chexpert,
we can train the model and perform downstream tasks such as classification in
supervised and zero-shot approaches. Our method reaches an area under the
curve performance (AUC) of 0.8618, which is slightly higher than the best
existing datasets. Finally, we show that we can perform zero-shot
classification using multimodal representations.
This work is embedded in our group’s endeavor to expedite artificial
intelligence in biomedical imaging by means of advanced pattern recognition
and machine learning methods for computational radiology and radiomics, e.g.,
[3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42,
43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61,
62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80,
81, 82, 83, 84, 85, 86].

Figure 1: The architecture of multimodal deep representation learning is
shown. The model encodes image and text features at global and local levels
and then fuses this information to encompass both local and global details for
later downstream tasks. The attention mechanism corresponds word-level
features to image sub-regions by leveraging a similarity matrix between the
local features of an image and its text. Global and local representations are
learned through maximizing the posterior probability of the image given text
and vice versa and using global and local contrastive learning.

##  2 Algorithm

###  2.1 Representation Learning Model

The GLoRIA model [1] was built on the premise that many anomalies observed in
medical imaging tests only occupy a small portion of the image and only
roughly match a handful of crucial sentences in medical reports. As a result,
the model creates an attention-based framework for multimodal representation
learning by contrasting image subregions to the words in the relevant report.
This allows for the generation of context-aware local representations of
pictures using attention weights that emphasize important visual sub-regions
for a particular word. Seen in Fig. 1.

The model takes advantage of the final adaptive average pooling layer of the
ResNet-50 model to help with the creation of Image Encoders. The GLoRIA text
encoder was pretrained on the BioClinicalBERT model using medical texts from
the MIMIC III dataset in order to produce clinically informed text embeddings.
Additionally, word-piece tokenization was employed to lessen the embeddings of
acronyms and some lexical problems in report drafting. The text encoder
extracts feature specific to each word component. The global text feature is
the whole of all word-piece features. Through the use of global, semantic data
that condenses the picture and report, and local, captures the semantics in
the image and, features. Word-level embeddings are used for local text
characteristics.
Supervised classification model: The model trains a linear classifier on top
of the already-trained image encoder GLoRIA model to evaluate the data
efficiency of the global picture representations.
Zero-shot classification: Zero-shot Classification analyzes the ability of our
learned representations to be categorized without the addition of additional
labels. The results of the Zero-shot Classification allow evaluation of how
good the representations between reports and images within GLoRIA model are
based on the achieved accuracy.

##  3 Data

We used three different datasets in this study due to the lack of radiology
reports for CheXpert and CheXphoto, and the smaller size of the IU-RR dataset
with radiology reports. In the following, we briefly describe each of these
datasets.

###  3.1 CheXpert

The CheXpert dataset [2] was used to train and evaluate the representation
learning framework for the supervised classification tasks. The 224,316 chest
radiographs from the 65,240 individuals in the CheXpert dataset are all linked
with the associated radiology reports. A total of 14 medical observations is
included on each radiograph’s label. This research used 191,229 frontal chest
radiograph image-text pairings (original train: 191,027 and original valid:
202). Example of pictures seen in Fig. 2.

Dataset | Atelectasis | Cardiomegaly | Consolidation | Edema | Pleural Effusion
---|---|---|---|---|---
CheXpert [87] |  |  |  |  |
CheXphoto [88] |  |  |  |  |
IU-RR [2] |  |  |  |  |
Figure 2:  Three datasets and their corresponding five types of lung diseases
are shown in this image. The radiology reports of CheXpert [87] and CheXphoto
[88] are not publicly available, while IU-RR [2] are labeled images and their
corresponding radiology reports. Despite the smaller size of the IU-RR
dataset, we could learn multimodal representations from the IU-RR dataset.

###  3.2 CheXphoto

CheXphoto dataset [87] is a noisy dataset comprised of a training set of
natural images and synthetic transformations applied to 10,507 x-rays from
3,000 different patients that were randomly selected from the CheXpert
training set. The validation and test sets are composed of 234 x-rays from 200
patients and 668 x-rays from 500 patients, respectively. The GLoRIA model only
focuses on the frontal images, so for the CheXphoto, we focus on 28,718 images
(original train: 28,112 and original valid: 606). Examples of x-ray images are
shown in Fig. 2.

Both CheXpert and CheXphoto had labels provided for each of the diseases
(positive = 1, negative = 0, unclear = -1). Although chest abnormalities are
frequently suspected, a conclusive diagnosis cannot be obtained without
further testing. However, at the time of this research, the CheXpert radiology
reports were still under review for the protected health information review
through the Health Insurance Portability and Accountability Act. The radiology
reports were not available for both CheXpert and CheXphoto datasets.

###  3.3 Chest X-Ray Collection from Indiana University

The open-access chest X-ray collection from Indiana University (IU-RR) [88]
consists of 3,996 radiology reports from the Indiana Network for Patient Care
and 8,121 associated images from the hospitals’ picture archiving systems. The
GLoRIA model only focuses on the frontal images so for the Indiana dataset,
and we focus on 3,279 images with reports (original train: 2,552 and original
valid: 727). The Chexpert-labeler [2] was used to assign the appropriate
labels to the IU-RR images in order to be consistent with the labels provided
for CheXpert and CheXphoto.

Multiple alternative labels for a target disease might create conflicting
results between categories because our Zero-shot classification and case
retrieval are focused on locating the most comparable target. Therefore, only
five diseases were selected (atelectasis, cardiomegaly, edema, pleural
effusion, and consolidation), and cases with only that disease were extracted
from the database [1]. For CheXpert there are 87 positive images for each of
the 5 diseases resulting in CheXphoto_5x87 dataset (total of 435 images). For
CheXphoto there are 62 positive images for each of the five diseases resulting
in the CheXphoto_5x62 dataset (total of 310 images). Each picture in this
collection has a positive annotation for just one particular circumstance. Due
to the fact that consolidation was only present in one instance, IU-RR 5x1
could only concentrate on one image per disease (5 cases in total).

The images from the above datasets are split into training, validation, and
test sets. For the validation, Chexpert-labeler chooses 5,000 photos randomly
from the original expert-labeled training sets for CheXpert and CheXphoto,
respectively. For IU-RR Chexpert-labeler chose 500 photos at random from the
original training set because of the smaller IU-RR dataset. The training sets
for all three datasets are determined by what is left after the corresponding
validation sets are selected. Then, from the expert-labeled training file,
CheXpert 5x87, CheXphoto 5x62, and IU-RR 5x1 are generated with exclusively
positive images. Lastly, the untouched expert-labeled validation file is used
as an independent test. Therefore, for CheXpert the frontal train split
(CheXpert_tr) consists of 185,778 images, the frontal validation split
(CheXpert_val) is 4,301 images, and the frontal test (CheXpert_ts) is 202
images. For CheXphoto the frontal train split (CheXphoto_ts) is 23,512 images,
the frontal validation split (CheXphoto_val) is 4,319 images, and the frontal
test (CheXphoto_ts) is 606 images. For IU-RR, the frontal train split (IU-
RR_tr) is 2,552 images, and the frontal test (IU-RR_ts) is 727 images. The
number of images in each split of these datasets is listed in Table 1.

Table 1: Three datasets and their corresponding train, validation, and test splits, number of images, and details of each split are listed here. Dataset split | Description | Number of images
---|---|---
CheXperttr | Frontal train split | 185,778
CheXpertval | Frontal validation split | 4,301
CheXpertts | Frontal test | 202
CheXphototr | Frontal train split | 23,512
CheXphotoval | Frontal validation split | 4,319
CheXphotots | Frontal test | 606
IU-RRtr | Frontal train split | 2,552
IU-RRts | Frontal test | 727

##  4 Experiments

Table 2: Evaluation of our representation learning performance based on supervised learning classification is listed here. The CheXphoto dataset [88] is used as a noisy dataset to train a classifier based on our representation learning. Both the representation learning dataset and test dataset are from IU-RR [2].  Train | Valid | Test | Train AUC | Valid AUC | Test AUC
---|---|---|---|---|---
CheXperttr | CheXpertval | IU-RRts | 0.680 | 0.681 | 0.856
CheXperttr | CheXphotoval | IU-RRts | 0.676 | 0.644 | 0.865
CheXperttr | IU-RRtr | IU-RRts | 0.674 | 0.881 | 0.868
CheXphototr | CheXphotoval | IU-RRts | 0.643 | 0.619 | 0.862
CheXphototr | CheXpertval | IU-RRts | 0.614 | 0.643 | 0.860
CheXphototr | IU-RRtr | IU-RRts | 0.636 | 0.868 | 0.860

We use the codes from the https://github.com/marshuang80/gloria, and use their
zero-shot classification and supervised classification to evaluate the
performance of our representation learning based on the IU-RR dataset that we
cleaned and prepared. Area Under the Curve (AUC) is used as the performance
metric for these downstream tasks.

Supervised classification The goal here is to evaluate the performance of the
representation learning on IU-RR dataset based on the performance of the
supervised learning problem. To this end, we train the model based on the
small-size, publicly available dataset of IU-RR [2]. Consequently, we use
noisy data of CheXphoto to train the classifier based on our encoder. The
results are listed in Table 2. As listed in Table 2, the performance of the
classifier achieves the AUC of 0.8618 $\pm$ 0.0042 (Mean + STD), which is
slightly lower than the best of the previously reported values in [89], which
is 0.8608. The reason that the test is higher than validation and the train is
that the test is on a different data set.

Zero-shot classification: The model here has been applied to three different
datasets in various configurations, noting that radiology reports of CheXpert
and CheXphoto are not publicly available. Therefore we rely on the IU-RR
dataset for the training of the representation learning. However, due to the
smaller size of the IU-RR dataset compared to the two other datasets, the
zero-shot learning is the model trained on the CheXpert dataset [1]. We
compare the performance of the zero-shot classification based on
representations that are learned on IU-RR, CheXpert, and CheXphoto, and based
on five types of lung diseases, namely atelectasis, cardiomegaly, edema,
pleural, and effusion. The detailed values of the classification results are
listed in Table 3.

For comparison, the average AUC for classifying the five lung pathologies on
the IU-RR test set ranged from 0.85 to 0.87 using the different training
datasets. These results compare favorably to other studies using IU-RR for the
classification of the five lung pathologies (AUC range:0.77-0.86) reported in
the literature [89].

Table 3: Zero-shot classification performance based on the representation model trained on IU-RR dataset  | Atelectasis | Cardiomegaly | Consolidation | Edema | Pleural Effusion | Average
---|---|---|---|---|---|---
CheXpert | 0.685 | 0.628 | 0.694 | 0.754 | 0.717 | 0.696
CheXphoto | 0.719 | 0.587 | 0.700 | 0.784 | 0.694 | 0.697

##  5 Conclusion

We developed a model that uses independent datasets utilizing language and
vision modalities to learn a representation encompassing local and global
features. The clinical significance of the model is the ability to utilize
both image features from the X-Rays and text characteristics from reports to
classify lung pathologies on chest X-rays. Five lung pathologies (atelectasis,
cardiomegaly, edema, pleural, and consolidation) were considered. The average
AUC for classifying the five lung pathologies on the IU-RR test set ranged
from 0.85 to 0.87 using the different training datasets. These results compare
favorably to other studies using UI-RR to classify the five lung pathologies
with a mean AUC of 0.86, which is favorably comparable to existing literature.
We have validated the cross-modal model on three independent datasets: IU-RR,
CheXpert, and CheXphoto. The robust performance of the model on different
datasets is promising and shows that it behaves satisfactorily.

###### Acknowledgements.

This research was partially funded by the American College of Radiology (ACR)
Innovation Award “AI-PROBE: A Novel Prospective Randomized Clinical Trial
Approach for Investigating the Clinical Usefulness of Artificial Intelligence
in Radiology” (PI: Axel Wismüller) and an Ernest J. Del Monte Institute for
Neuroscience Award from the Harry T. Mangurian Jr. Foundation (PI: Axel
Wismüller). This work was conducted as a Practice Quality Improvement (PQI)
project related to the American Board of Radiology (ABR) Maintenance of
Certificate (MOC) for A.W.

## References

  * [1] Huang, S.-C., Shen, L., Lungren, M. P., and Yeung, S., “Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition,” in [Proceedings of the IEEE/CVF International Conference on Computer Vision ], 3942–3951 (2021).
  * [2] Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., and McDonald, C. J., “Preparing a collection of radiology examinations for distribution and retrieval,” Journal of the American Medical Informatics Association 23(2), 304–310 (2016).
  * [3] Bunte, K., Hammer, B., Wismüller, A., and Biehl, M., “Adaptive local dissimilarity measures for discriminative dimension reduction of labeled data,” Neurocomputing 73(7-9), 1074–1092 (2010).
  * [4] Wismüller, A., Vietze, F., and Dersch, D. R., “Segmentation with neural networks,” in [Handbook of medical imaging ], 107–126, Academic Press, Inc. (2000).
  * [5] Wismüller, A., Vietze, F., Behrends, J., Meyer-Baese, A., Reiser, M., and Ritter, H., “Fully automated biomedical image segmentation by self-organized model adaptation,” Neural Networks 17(8-9), 1327–1344 (2004).
  * [6] Hoole, P., Wismüller, A., Leinsinger, G., Kroos, C., Geumann, A., and Inoue, M., “Analysis of tongue configuration in multi-speaker, multi-volume MRI data,” (2000).
  * [7] Wismüller, A., “Exploratory morphogenesis (XOM): a novel computational framework for self-organization,” Ph. D. thesis, Technical University of Munich, Department of Electrical and Computer Engineering (2006).
  * [8] Wismüller, A., Dersch, D. R., Lipinski, B., Hahn, K., and Auer, D., “A neural network approach to functional MRI pattern analysis—clustering of time-series by hierarchical vector quantization,” in [International Conference on Artificial Neural Networks ], 857–862, Springer (1998).
  * [9] Wismüller, A., Vietze, F., Dersch, D. R., Behrends, J., Hahn, K., and Ritter, H., “The deformable feature map-a novel neurocomputing algorithm for adaptive plasticity in pattern analysis,” Neurocomputing 48(1-4), 107–139 (2002).
  * [10] Behrends, J., Hoole, P., Leinsinger, G. L., Tillmann, H. G., Hahn, K., Reiser, M., and Wismüller, A., “A segmentation and analysis method for MRI data of the human vocal tract,” in [Bildverarbeitung für die Medizin 2003 ], 186–190, Springer (2003).
  * [11] Wismüller, A., “Neural network computation in biomedical research: chances for conceptual cross-fertilization,” Theory in Biosciences (1997).
  * [12] Bunte, K., Hammer, B., Villmann, T., Biehl, M., and Wismüller, A., “Exploratory observation machine (XOM) with Kullback-Leibler divergence for dimensionality reduction and visualization.,” in [ESANN ], 10, 87–92 (2010).
  * [13] Wismüller, A., Vietze, F., Dersch, D. R., Hahn, K., and Ritter, H., “The deformable feature map—adaptive plasticity for function approximation,” in [International Conference on Artificial Neural Networks ], 123–128, Springer (1998).
  * [14] Wismüller, A., “The exploration machine–a novel method for data visualization,” in [International Workshop on Self-Organizing Maps ], 344–352, Springer (2009).
  * [15] Huber, M. B., Nagarajan, M., Leinsinger, G., Ray, L. A., and Wismüller, A., “Classification of interstitial lung disease patterns with topological texture features,” in [Medical Imaging 2010: Computer-Aided Diagnosis ], 7624, 762410, International Society for Optics and Photonics (2010).
  * [16] Wismüller, A., “The exploration machine: a novel method for analyzing high-dimensional data in computer-aided diagnosis,” in [Medical Imaging 2009: Computer-Aided Diagnosis ], 7260, 72600G, International Society for Optics and Photonics (2009).
  * [17] Bunte, K., Hammer, B., Villmann, T., Biehl, M., and Wismüller, A., “Neighbor embedding XOM for dimension reduction and visualization,” Neurocomputing 74(9), 1340–1350 (2011).
  * [18] Meyer-Bäse, A., Lange, O., Wismüller, A., and Ritter, H., “Model-free functional MRI analysis using topographic independent component analysis,” International journal of neural systems 14(04), 217–228 (2004).
  * [19] Wismüller, A., “Exploration-organized morphogenesis (XOM): A general framework for learning by self-organization,” Forschungsberichte-Institut für Phonetik und Sprachliche Kommunikation der Universität München (37), 205–239 (2001).
  * [20] Saalbach, A., Twellmann, T., Nattkemper, T. W., Wismüller, A., Ontrup, J., and Ritter, H. J., “A hyperbolic topographic mapping for proximity data.,” in [Artificial Intelligence and Applications ], 2005, 106–111 (2005).
  * [21] Wismüller, A., “A computational framework for nonlinear dimensionality reduction and clustering,” in [International Workshop on Self-Organizing Maps ], 334–343, Springer (2009).
  * [22] Leinsinger, G., Teipel, S., Wismüller, A., Born, C., Meindl, T., Flatz, W., Schönberg, S., Pruessner, J., Hampel, H., and Reiser, M., “Volumetric MRI for evaluation of regional pattern and progressin of neocortical degeneration in alzheimer’s disease,” Der Radiologe 43(7), 537–542 (2003).
  * [23] Meyer-Base, A., Auer, D., and Wismüller, A., “Topographic independent component analysis for fMRI signal detection,” in [Proceedings of the International Joint Conference on Neural Networks, 2003. ], 1, 601–605, IEEE (2003).
  * [24] Wismüller, A., “The exploration machine-a novel method for structure-preserving dimensionality reduction.,” in [ESANN ], Citeseer (2009).
  * [25] Wismüller, A., Lange, O., Auer, D., and Leinsinger, G., “Model-free functional MRI analysis for detecting low-frequency functional connectivity in the human brain,” in [Medical Imaging 2010: Computer-Aided Diagnosis ], 7624, 76241M, International Society for Optics and Photonics (2010).
  * [26] Meyer-Baese, A., Schlossbauer, T., Lange, O., and Wismüller, A., “Small lesions evaluation based on unsupervised cluster analysis of signal-intensity time courses in dynamic breast MRI,” International journal of biomedical imaging 2009 (2009).
  * [27] Meyer-Bäse, A., Saalbach, A., Lange, O., and Wismüller, A., “Unsupervised clustering of fMRI and MRI time series,” Biomedical Signal Processing and Control 2(4), 295–310 (2007).
  * [28] Wismüller, A., Verleysen, M., Aupetit, M., and Lee, J. A., “Recent advances in nonlinear dimensionality reduction, manifold and topological learning.,” in [ESANN ], (2010).
  * [29] Meyer-Baese, A., Lange, O., Wismüller, A., and Hurdal, M. K., “Analysis of dynamic susceptibility contrast MRI time series based on unsupervised clustering methods,” IEEE Transactions on Information Technology in Biomedicine 11(5), 563–573 (2007).
  * [30] Wismüller, A., Vietze, F., Dersch, D. R., Hahn, K., and Ritter, H. J., “A neural network approach to adaptive pattern analysis-the deformable feature map.,” in [ESANN ], 189–194 (2000).
  * [31] Wismüller, A., Behrends, J., Hoole, P., Leinsinger, G. L., Reiser, M. F., and Westesson, P.-L., “Human vocal tract analysis by in vivo 3d MRI during phonation: a complete system for imaging, quantitative modeling, and speech synthesis,” in [International Conference on Medical Image Computing and Computer-Assisted Intervention ], 306–312, Springer (2008).
  * [32] Wismüller, A., Dersch, D. R., Lipinski, B., Hahn, K., and Auer, D., “Hierarchical clustering of functional MRI time-series by deterministic annealing,” in [International Symposium on Medical Data Analysis ], 49–54, Springer (2000).
  * [33] Wismüller, A., “Method and device for representing multichannel image data,” (Nov. 17 2015).  US Patent 9,189,846.
  * [34] Huber, M. B., Bunte, K., Nagarajan, M. B., Biehl, M., Ray, L. A., and Wismüller, A., “Texture feature ranking with relevance learning to classify interstitial lung disease patterns,” Artificial intelligence in medicine 56(2), 91–97 (2012).
  * [35] Wismüller, A., Meyer-Baese, A., Lange, O., Reiser, M. F., and Leinsinger, G., “Cluster analysis of dynamic cerebral contrast-enhanced perfusion MRI time-series,” IEEE transactions on medical imaging 25(1), 62–73 (2005).
  * [36] Twellmann, T., Saalbach, A., Muller, C., Nattkemper, T. W., and Wismüller, A., “Detection of suspicious lesions in dynamic contrast enhanced MRI data,” in [The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society ], 1, 454–457, IEEE (2004).
  * [37] Otto, T. D., Meyer-Baese, A., Hurdal, M., Sumners, D., Auer, D., and Wismüller, A., “Model-free functional MRI analysis using cluster-based methods,” in [Intelligent Computing: Theory and Applications ], 5103, 17–24, International Society for Optics and Photonics (2003).
  * [38] Varini, C., Nattkemper, T. W., Degenhard, A., and Wismüller, A., “Breast MRI data analysis by lle,” in [2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541) ], 3, 2449–2454, IEEE (2004).
  * [39] Meyer-Baese, A., Lange, O., Wismuller, A., and Leinsinger, G., “Computer-aided diagnosis in breast MRI based on ICA and unsupervised clustering techniques,” in [Independent Component Analyses, Wavelets, Unsupervised Smart Sensors, and Neural Networks III ], 5818, 38–49, International Society for Optics and Photonics (2005).
  * [40] Huber, M. B., Lancianese, S. L., Nagarajan, M. B., Ikpot, I. Z., Lerner, A. L., and Wismüller, A., “Prediction of biomechanical properties of trabecular bone in mr images with geometric features and support vector regression,” IEEE Transactions on Biomedical Engineering 58(6), 1820–1826 (2011).
  * [41] Meyer-Base, A., Pilyugin, S. S., and Wismüller, A., “Stability analysis of a self-organizing neural network with feedforward and feedback dynamics,” in [2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541) ], 2, 1505–1509, IEEE (2004).
  * [42] Wismüller, A., Dersch, D., Lipinski, B., Hahn, K., and Auer, D., “Hierarchical clustering of fMRI time-series by deterministic annealing,” Neuroimage 7(4), S593 (1998).
  * [43] Meyer-Baese, A., Lange, O., Schlossbauer, T., and Wismüller, A., “Computer-aided diagnosis and visualization based on clustering and independent component analysis for breast MRI,” in [2008 15th IEEE International Conference on Image Processing ], 3000–3003, IEEE (2008).
  * [44] Wismüller, A., De, T., Lochmüller, E., Eckstein, F., and Nagarajan, M. B., “Introducing anisotropic minkowski functionals and quantitative anisotropy measures for local structure analysis in biomedical imaging,” in [Medical Imaging 2013: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 8672, 86720I, International Society for Optics and Photonics (2013).
  * [45] Bhole, C., Pal, C., Rim, D., and Wismüller, A., “3d segmentation of abdominal ct imagery with graphical models, conditional random fields and learning,” Machine vision and applications 25(2), 301–325 (2014).
  * [46] Nagarajan, M. B., Coan, P., Huber, M. B., Diemoz, P. C., Glaser, C., and Wismüller, A., “Computer-aided diagnosis in phase contrast imaging x-ray computed tomography for quantitative characterization of ex vivo human patellar cartilage,” IEEE Transactions on Biomedical Engineering 60(10), 2896–2903 (2013).
  * [47] Wismüller, A., Behrends, J., Dersch, D., Leinsinger, G., Vietze, F., and Hahn, K., “Automatic segmentation of cerebral contours in multispectral MRI data sets of the human brain by self-organizing neural networks,” in [Radiology ], 221, 461–461 (2001).
  * [48] Wismueller, A., Vietze, F., Dersch, D., Leinsinger, G., Ritter, H., and Hahn, K., “Adaptive self-organized template matching of the gray-level feature space for automatic segmentation of multispectral MRI data of the human brain,” in [Radiology ], 213, 364–364 (1999).
  * [49] Meyer-Baese, A., Wismüller, A., Lange, O., and Leinsinger, G., “Computer-aided diagnosis in breast MRI based on unsupervised clustering techniques,” in [Intelligent Computing: Theory and Applications II ], 5421, 29–37, International Society for Optics and Photonics (2004).
  * [50] Nagarajan, M. B., Coan, P., Huber, M. B., Diemoz, P. C., Glaser, C., and Wismüller, A., “Computer-aided diagnosis for phase-contrast x-ray computed tomography: quantitative characterization of human patellar cartilage with high-dimensional geometric features,” Journal of digital imaging 27(1), 98–107 (2014).
  * [51] Pester, B., Leistritz, L., Witte, H., and Wismueller, A., “Exploring effective connectivity by a Granger causality approach with embedded dimension reduction,” Biomedical Engineering/Biomedizinische Technik 58(SI-1-Track-G), 000010151520134172 (2013).
  * [52] Nagarajan, M. B., Huber, M. B., Schlossbauer, T., Leinsinger, G., Krol, A., and Wismüller, A., “Classification of small lesions on dynamic breast MRI: Integrating dimension reduction and out-of-sample extension into cadx methodology,” Artificial intelligence in medicine 60(1), 65–77 (2014).
  * [53] Yang, C.-C., Nagarajan, M. B., Huber, M. B., Carballido-Gamio, J., Bauer, J. S., Baum, T. H., Eckstein, F., Lochmüller, E.-M., Majumdar, S., Link, T. M., et al., “Improving bone strength prediction in human proximal femur specimens through geometrical characterization of trabecular bone microarchitecture and support vector regression,” Journal of electronic imaging 23(1), 013013 (2014).
  * [54] Wismüller, A., Nagarajan, M. B., Witte, H., Pester, B., and Leistritz, L., “Pair-wise clustering of large scale Granger causality index matrices for revealing communities,” in [Medical Imaging 2014: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 9038, 90381R, International Society for Optics and Photonics (2014).
  * [55] Wang, X., Nagarajan, M. B., Conover, D., Ning, R., O’Connell, A., and Wismüller, A., “Investigating the use of texture features for analysis of breast lesions on contrast-enhanced cone beam CT,” in [Medical Imaging 2014: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 9038, 903822, International Society for Optics and Photonics (2014).
  * [56] Wismüller, A., Wang, X., DSouza, A. M., and Nagarajan, M. B., “A framework for exploring non-linear functional connectivity and causality in the human brain: mutual connectivity analysis (mca) of resting-state functional MRI with convergent cross-mapping and non-metric clustering,” arXiv preprint arXiv:1407.3809 (2014).
  * [57] Meyer-Bäse, A., Pilyugin, S., Wismüller, A., and Foo, S., “Local exponential stability of competitive neural networks with different time scales,” Engineering Applications of Artificial Intelligence 17(3), 227–232 (2004).
  * [58] Schmidt, C., Pester, B., Nagarajan, M., Witte, H., Leistritz, L., and Wismüller, A., “Impact of multivariate Granger causality analyses with embedded dimension reduction on network modules,” in [2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society ], 2797–2800, IEEE (2014).
  * [59] Nagarajan, M. B., Coan, P., Huber, M. B., Diemoz, P. C., and Wismüller, A., “Integrating dimension reduction and out-of-sample extension in automated classification of ex vivo human patellar cartilage on phase contrast x-ray computed tomography,” Plos one 10(2), e0117157 (2015).
  * [60] Wismüller, A., Abidin, A. Z., D’Souza, A. M., Wang, X., Hobbs, S. K., Leistritz, L., and Nagarajan, M. B., “Nonlinear functional connectivity network recovery in the human brain with mutual connectivity analysis (MCA): convergent cross-mapping and non-metric clustering,” in [Medical Imaging 2015: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 9417, 94170M, International Society for Optics and Photonics (2015).
  * [61] Nagarajan, M. B., Checefsky, W. A., Abidin, A. Z., Tsai, H., Wang, X., Hobbs, S. K., Bauer, J. S., Baum, T., and Wismüller, A., “Characterizing trabecular bone structure for assessing vertebral fracture risk on volumetric quantitative computed tomography,” in [Medical Imaging 2015: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 9417, 94171E, International Society for Optics and Photonics (2015).
  * [62] Abidin, A. Z., Nagarajan, M. B., Checefsky, W. A., Coan, P., Diemoz, P. C., Hobbs, S. K., Huber, M. B., and Wismüller, A., “Volumetric characterization of human patellar cartilage matrix on phase contrast x-ray computed tomography,” in [Medical Imaging 2015: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 9417, 94171F, International Society for Optics and Photonics (2015).
  * [63] Wismüller, A., Abidin, A. Z., DSouza, A. M., and Nagarajan, M. B., “Mutual connectivity analysis (MCA) for nonlinear functional connectivity network recovery in the human brain using convergent cross-mapping and non-metric clustering,” in [Advances in Self-Organizing Maps and Learning Vector Quantization ], 217–226, Springer (2016).
  * [64] Abidin, A. Z., D’Souza, A. M., Nagarajan, M. B., and Wismüller, A., “Investigating changes in brain network properties in hiv-associated neurocognitive disease (HAND) using mutual connectivity analysis (MCA),” in [Medical Imaging 2016: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 9788, 97881W, International Society for Optics and Photonics (2016).
  * [65] Schmidt, C., Pester, B., Schmid-Hertel, N., Witte, H., Wismüller, A., and Leistritz, L., “A multivariate Granger causality concept towards full brain functional connectivity,” PloS one 11(4) (2016).
  * [66] Abidin, A. Z., Jameson, J., Molthen, R., and Wismüller, A., “Classification of micro-CT images using 3D characterization of bone canal patterns in human osteogenesis imperfecta,” in [Medical Imaging 2017: Computer-Aided Diagnosis ], 10134, 1013413, International Society for Optics and Photonics (2017).
  * [67] Abidin, A. Z., Chockanathan, U., DSouza, A. M., Inglese, M., and Wismüller, A., “Using large-scale Granger causality to study changes in brain network properties in the clinically isolated syndrome (CIS) stage of multiple sclerosis,” in [Medical Imaging 2017: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 10137, 101371B, International Society for Optics and Photonics (2017).
  * [68] DSouza, A. M., Abidin, A. Z., Leistritz, L., and Wismüller, A., “Exploring connectivity with large-scale Granger causality on resting-state functional MRI,” Journal of neuroscience methods 287, 68–79 (2017).
  * [69] Abidin, A. Z., DSouza, A. M., Nagarajan, M. B., Wang, L., Qiu, X., Schifitto, G., and Wismüller, A., “Alteration of brain network topology in HIV-associated neurocognitive disorder: A novel functional connectivity perspective,” NeuroImage: Clinical 17, 768–777 (2018).
  * [70] Abidin, A. Z., Deng, B., DSouza, A. M., Nagarajan, M. B., Coan, P., and Wismüller, A., “Deep transfer learning for characterizing chondrocyte patterns in phase contrast x-ray computed tomography images of the human patellar cartilage,” Computers in biology and medicine 95, 24–33 (2018).
  * [71] Chockanathan, U., Abidin, A. Z., DSouza, A. M., Schifitto, G., and Wismüller, A., “Resilient modular small-world directed brain networks in healthy subjects with large-scale Granger causality analysis of resting-state functional MRI,” in [Medical Imaging 2018: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 10578, 105780B, International Society for Optics and Photonics (2018).
  * [72] DSouza, A. M., Abidin, A. Z., Chockanathan, U., Schifitto, G., and Wismüller, A., “Mutual connectivity analysis of resting-state functional MRI data with local models,” NeuroImage 178, 210–223 (2018).
  * [73] Chockanathan, U., DSouza, A. M., Abidin, A. Z., Schifitto, G., and Wismüller, A., “Automated diagnosis of HIV-associated neurocognitive disorders using large-scale Granger causality analysis of resting-state functional MRI,” Computers in Biology and Medicine 106, 24–30 (2019).
  * [74] Abidin, A. Z., Dar, I., D’Souza, A. M., Lin, E. P., and Wismüller, A., “Investigating a quantitative radiomics approach for brain tumor classification,” in [Medical Imaging 2019: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 10953, 109530B, International Society for Optics and Photonics (2019).
  * [75] DSouza, A. M., Abidin, A. Z., and Wismüller, A., “Classification of autism spectrum disorder from resting-state fMRI with mutual connectivity analysis,” in [Medical Imaging 2019: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 10953, 109531D, International Society for Optics and Photonics (2019).
  * [76] Abidin, A. Z., DSouza, A. M., Schifitto, G., and Wismüller, A., “Detecting cognitive impairment in HIV-infected individuals using mutual connectivity analysis of resting state functional MRI,” Journal of neurovirology 26(2), 188–200 (2020).
  * [77] Wismüller, A. and Stockmaster, L., “A prospective randomized clinical trial for measuring radiology study reporting time on artificial intelligence-based detection of intracranial hemorrhage in emergent care head ct,” in [Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 11317, 113170M, International Society for Optics and Photonics (2020).
  * [78] Wismüller, A., DSouza, A. M., Abidin, A. Z., and Vosoughi, M. A., “Large-scale nonlinear Granger causality: A data-driven, multivariate approach to recovering directed networks from short time-series data,” arXiv preprint arXiv:2009.04681 (2020).
  * [79] Vosoughi, A., DSouza, A., Abidin, A., and Wismüller, A., “Relation discovery in nonlinearly related large-scale settings,” in [ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) ], 5103–5107, IEEE (2022).
  * [80] Vosoughi, M. A. and Wismüller, A., “Large-scale kernelized Granger causality to infer topology of directed graphs with applications to brain networks,” arXiv preprint arXiv:2011.08261 (2020).
  * [81] Vosoughi, M. A. and Wismüller, A., “Large-scale extended granger causality for classification of marijuana users from functional mri,” in [Medical Imaging 2021: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 11600, 116000D, International Society for Optics and Photonics (2021).
  * [82] Wismüller, A. and Vosoughi, M. A., “Classification of schizophrenia from functional MRI using large-scale extended Granger causality,” International Society for Optics and Photonics (2021).
  * [83] Wismüller, A. and Vosoughi, M. A., “Large-scale augmented Granger causality (lsAGC) for connectivity analysis in complex systems: From computer simulations to functional MRI (fMRI),” in [Medical Imaging 2021: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 11600, 1160009, International Society for Optics and Photonics (2021).
  * [84] Wismüller, A., Dsouza, A. M., Vosoughi, M. A., and Abidin, A., “Large-scale nonlinear Granger causality for inferring directed dependence from short multivariate time-series data,” Scientific reports 11(1), 1–11 (2021).
  * [85] Vosoughi, M. A. and Wismüller, A., “Leveraging pre-images to discover nonlinear relationships in multivariate environments,” in [2021 European Signal Processing Conference (EUSIPCO) ], IEEE (August 2021).
  * [86] Vosoughi, M. A. and Wismüller, A., “Large-scale kernelized granger causality (lskgc) for inferring topology of directed graphs in brain networks,” in [Medical Imaging 2022: Biomedical Applications in Molecular, Structural, and Functional Imaging ], 12036, 1203603, SPIE (2022).
  * [87] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al., “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in [Proceedings of the AAAI conference on artificial intelligence ], 33(01), 590–597 (2019).
  * [88] Phillips, N. A., Rajpurkar, P., Sabini, M., Krishnan, R., Zhou, S., Pareek, A., Phu, N. M., Wang, C., Jain, M., Du, N. D., et al., “Chexphoto: 10,000+ photos and transformations of chest x-rays for benchmarking deep learning robustness,” in [Machine Learning for Health ], 318–327, PMLR (2020).
  * [89] Xue, Y. and Huang, X., “Improved disease classification in chest x-rays with transferred features from report generation,” in [International Conference on Information Processing in Medical Imaging ], 125–138, Springer (2019).