# Towards Generalist Foundation Model for Radiology

Chaoyi Wu  Shanghai Jiao Tong University  Shanghai AI Laboratory
https://chaoyi-wu.github.io/RadFM Xiaoman Zhang  Shanghai Jiao Tong University
Shanghai AI Laboratory
https://chaoyi-wu.github.io/RadFM
Ya Zhang  Shanghai Jiao Tong University  Shanghai AI Laboratory
https://chaoyi-wu.github.io/RadFM Yanfeng Wang  Shanghai Jiao Tong University
Shanghai AI Laboratory
https://chaoyi-wu.github.io/RadFM Weidi Xie  Shanghai Jiao Tong University
Shanghai AI Laboratory
https://chaoyi-wu.github.io/RadFM

###### Abstract

In this study, we aim to initiate the development of Radiology Foundation
Model, termed as RadFM. We consider the construction of foundational models
from the perspectives of data, model design, and evaluation thoroughly. Our
contribution can be concluded as follows: (i), we construct a large-scale
Medical Multi-modal Dataset, MedMD, consisting of 16M 2D and 3D medical scans.
To the best of our knowledge, this is the first multi-modal dataset containing
3D medical scans. (ii), We propose an architecture that enables visually
conditioned generative pre-training, allowing for the integration of text
input interleaved with 2D or 3D medical scans to generate response for diverse
radiologic tasks. The model was initially pre-trained on MedMD and
subsequently domain-specific fine-tuned on RadMD, a radiologic cleaned version
of MedMD, containing 3M radiologic visual-language pairs. (iii), we propose a
new evaluation benchmark that comprises five tasks, aiming to comprehensively
assess the capability of foundation models in handling practical clinical
problems. Our experimental results confirm that RadFM significantly
outperforms existing multi-modal foundation models. The codes, data, and model
checkpoint will all be made publicly available to promote further research and
development in the field.

##  1 Introduction

Generalist foundation models [5], the latest generation of AI models pre-
trained on large-scale data, have demonstrated remarkable success in various
domains, for example, natural language processing, computer vision [37, 20].
Their ability to address diverse and challenging tasks, often without
requiring extensive labeled data, has also attracted tremendous attention
among researchers in the field of Artificial Intelligence for Medicine
(AI4Medicine) [24, 19, 44, 38, 26]. Despite the promising clinical use cases,
the progress in developing medical foundation models has been fundamentally
hindered by the challenges from three aspects:

  * •

Lack of multimodal datasets for training: medicine by its nature, requires
understanding multimodal data, spanning text (electronic health record,
medical reports), 1D signals (ECG), 2D images (ultrasound, X-ray), 3D images
(CT or MRI scans), genomics, and more. To support the training of the medical
generalist foundation model, a large-scale, comprehensive, multimodal dataset
is desperately required;

  * •

Lack of general architecture formulation: in the literature of AI4Medicine,
various clinical tasks have largely been tackled by following a divide-and-
conquer paradigm, i.e., different architectures are designed for the problem
of interest, like diagnosis [43, 36] or segmentation [13, 6]. In contrast,
developing a medical foundation model requires one general architecture that
is capable of tackling a wide spectrum of clinical tasks, by fusing
information from a mixture of different modalities;

  * •

Lack of effective benchmark to monitor progress: benchmarking the models’
clinical knowledge predominantly relies on task-specific datasets with a
limited number of testing cases. A large-scale, complex benchmark is yet to be
established, to comprehensively measure the progress of the development on
medical foundation model across a diverse range of clinical tasks.

Considering the above-mentioned challenges, we take a preliminary, yet
realistic step toward developing a generalist medical foundation model, by
concentrating on radiology, which has shown to play a vital role in clinical
scenarios, for example, disease diagnosis, treatment planning, and monitoring
patient progression. Specifically, we present our progress towards such a
Radiology Foundation Model (RadFM), that aims to tackle a wide spectrum of
clinical radiology tasks, by learning from corresponding medical scans, for
example, X-ray, CT, MRI, and well-structured reports, with the same set of
model weights.

Figure 1: The general comparison between RadFM and different SOTA methods,
_i.e._ , OpenFlamingo [1] and MedVInT [44]. On the left we plot the radar
figure of the three methods, the average of different metrics are plotted and
the coordinate axes are logarithmized. On the right, we draw the comparison on
the five different tasks with different metrics. Both two can indicate the
superiority of RadFM, surpassing former methods significantly.

In order to train our proposed RadFM, we start by constructing a novel, large-
scale, Medical Multi-modal Dataset, named MedMD, with around 16M medical scans
in total, consisting of 15.5M 2D scans and 180k 3D radiology scans (equivalent
to 7M 2D slices) accompanied with textual descriptions, for example, radiology
reports, visual-language instruction, or crucial disease diagnosis labels.
MedMD encompasses a wide range of radiological modalities and anatomical
regions of the human body, featuring over 5000 diseases, thus can potentially
serve as the cornerstone for developing foundation models in radiology.

Architecturally, RadFM refers to a visually conditioned autoregressive text
generation model, that empowers to seamlessly integrate natural language with
2D or 3D medical scans, and addresses a wide range of medical tasks by
outputting natural language as output. The proposed model is initially pre-
trained on the MedMD dataset and subsequently fine-tuned through visual
instruction on a radiologic filtered dataset, comprising 3M meticulously
curated multi-modal samples with only radiologic cases, termed as RadMD,
ensuring a high-quality and reliable dataset for the domain-specific fine-
tuning process.

To monitor the developmental progress on the foundation model for radiology,
we establish a novel, comprehensive evaluation benchmark, covering a variety
of clinical tasks, for example, disease diagnosis, report generation, and
visual question-answering on radiologic modalities and anatomical regions. We
conduct a comparison with the existing strong models that are open-source, for
example, Open-flamingo [3], MedVInT [44], and observe significant benefits
across all considered tasks.

Overall, in this work, we demonstrate the potential of developing a generalist
foundation model for radiology, by making the following contributions:

  * •

A large-scale multimodal biomedical dataset, that encompasses diverse
modalities of medical imaging and clinical reports, featuring over 5000
diseases, thus can potentially serve as the cornerstone for developing
foundation models in radiology.

  * •

A demonstration of radiology foundation model (RadFM), that surpasses the
constraints of existing models by overcoming limitations on data size,
modalities, and applications. RadFM sets a new paradigm as the first multi-
modal foundation model for seamlessly integrating natural languages with both
2D and 3D radiologic images, across various imaging modalities, that enables
to tackle a wide spectrum of downstream clinical tasks with one single model.

  * •

A comprehensive benchmark for radiology (RadBench), that enables to monitor
the progress of developing foundation models. Specifically, we conduct
comparison on the proposed benchmark with a set of open-source, strong models,
RadFM is shown to be superior in all of the tasks as shown in Fig 1.

##  2 Related Work.

With the success of generative language foundation models such as GPT-4 [31]
and PaLM-2 [2], there has been a surge of interest in multi-modal foundation
models. While significant strides have been made in the realm of natural
scenery, as evidenced by BLIP-2 [20] and Flamingo [1], the development of
generalist medical artificial intelligence is still in its nascent stages
[25]. The relevant research can be bifurcated into two primary areas, namely,
dataset construction and model training.

Dataset Construction. Contrary to the natural scenery domain, which boasts
numerous large-scale multi-modal datasets such as MMC4 [47], Visual Genome
[17], and LION-5B [34], the medical domain is somewhat lacking. The most
widely utilized medical multi-modal dataset is MIMIC-CXR [16], which only
contains chest X-ray images with caption reports and its quantity (224K) is
relatively small. In PMC-OA [22], the authors have compiled a dataset
containing 1.6M image-caption pairs. Although it encompasses various image
modalities, many 3D medical scans are presented as 2D slices since the images
are extracted from papers. There are also some medical VQA datasets, such as
VQA-RAD [18], SLAKE [23], and PMC-VQA [44], but they are also limited to 2D
images. In Med-Flamingo [26], they have collected a dataset, MTB, consisting
of approximately 0.8M images interleaved with texts while it is not open-
source.

Model Training. To date, several works have focused on building medical
foundation models. LLaVA-Med [19] and MedVInT [44] utilize image captions in
PubMed Central to generate medical visual instruction tuning data, but they
are limited to 2D images and single image inputs. In Med-PaLM M [38], the
authors amalgamate existing medical image or multi-modal datasets, but the
majority of images are X-rays, which are not sufficiently accurate for
clinical practice. Similar to previous works, it also limits itself to 2D
images. MedBLIP [7] focuses on a specific 3D imaging modality, MRI. A very
recent work MedFlamingo [26] has enabled multi-interleaved 2D image input.

In conclusion, there is currently a dearth of a large-scale medical multi-
modal dataset that satisfies all the following characteristics: containing
both 3D and 2D image inputs, multi-image input per case, and images
interleaved with text. On the other hand, there is also no model that can
support this series of heterogeneous input forms simultaneously. This paper
aims to address these gaps, aligning more closely with clinical practice.

Figure 2: Overview of Medical Multimodal Dataset (MedMD). Our collected data
covers the majority of radiologic modalities and anatomical regions of the
human body, such as brain, head and neck, thorax, spine, abdomen, upper limb,
lower limb, and pelvis, etc. The dataset mixes two types of datasets, i.e.,
interleaved datasets and visual instruction datasets. $\mathcal{T}$ refers to
the text of interleaved data, $\mathcal{I}$ refers to the instruction input
text, and $\mathcal{R}$ refers to the response text.

##  3 Dataset

Here, we start by introducing a large-scale Medical Multimodal Dataset (MedMD)
in Sec. 3.1, to the best of our knowledge, this is the first large-scale
medical vision-language dataset, covering a wide range of anatomies with over
5000 diseases. In Sec. 3.2, we further contribute a filtered radiology subset,
named Radiology Multimodal Dataset (RadMD), dedicated to domain-specific fine-
tuning towards building foundation model for radiology.

###  3.1 Medical Multimodal Dataset (MedMD)

To start, we construct a candidate data pool by pulling a variety of existing
visual-language medical datasets together, for example, MIMIC-CXR [16] and
PMC-OA [22]. Despite the scale of these high-quality datasets, they are
fundamentally limited in several aspects: (i) Data format. These datasets are
only composed of 2D medical images, which do not fully capture the
complexities in medicine, for example, 3D medical imaging modalities, like CT,
MRI; (ii) Modality diversity. A noteworthy limitation arises from the fact
only chest X-ray images are provided with medical reports, which has clearly
restricted the generalizability of the models to a broader range of imaging
modalities and anatomical regions; (iii) Report quality. Another critical
limitation lies in the use of data extracted from figures and captions from
research papers. The gap between research-oriented data and real-world
clinical scenarios may hinder the model’s ability to provide accurate and
reliable diagnoses and limit its applicability toward clinical decision-
making. To support the training of our proposed Radiology Foundation Model
(RadFM), we thus augment the dataset with four new ones, including PMC-Inline,
PMC-CaseReport, Rad3D, and MPx, totalling 15.5M 2D scans, 180k 3D scans
(equivalent to 7M 2D slices), with corresponding captions or diagnosis labels,
as shown in Fig. 2.

To construct MedMD, we split the candidate data pool into two splits, one is
of interleaved image-language data that is collected from academic papers,
while the other one contains data constructed for visual-language instruction
tuning, as detailed below.

Table 1: Description of the collected dataset Medical Multimodal Dataset
(MedMD) for model pre-training.

Dataset Name |  Description | Data size
---|---|---
Interleaved Dataset
2D Image-text data |  | Pair number
PMC-Inline |  A medical dataset containing PMC-papers which links with 11M images through inline reference, _e.g._ , “as shown in fig.X”. | 11M
Visual Instruction Tuning Dataset
Image data |  | Image number
VinDr-Mammo [28] |  A mammography dataset consists of 5,000 four-view exams with level assessment and finding annotations. | 20K
VinDr-SpineXR [29] |  A spine X-ray dataset consists of 10,468 images from 5,000 studies with annotations. | 10K
VinDr-PCXR [30] |  A pediatric chest X-ray dataset consists of 9,125 studies with annotations. | 9K
CXR-Mix [40] |  A collection of different chest X-ray diagnosis datasets containing 449K images | 449K.
2D Image-text data |  | Pair number
PMC-OA [22] |  A medical dataset contains 1.65M figures and captions collected from PubMed Central. | 1.65M
PMC-VQA [44] |  A medical visual question-answering dataset contains 413k VQA pairs. | 413k
PMC-CaseReport |  A sub-dataset filtered from PMC-Inline containing 103K cases report papers. We generate 1.1M VQA pairs by querying ChatGPT. we keep some background information of the patient to form context input. | 1.1M
MPx-Single |  A medical vision-language dataset contains the modality, plane and captions for each image. | 114K
MPx-Multi |  A medical vision-language dataset contains findings, discussion, and diagnoses for each case which may contain a series of radiology images. | 39K
VQA-RAD [18] |  A medical visual question-answering dataset consists of 3,515 question–answer pairs on 315 radiology images. | 3.5K
SLAKE [23] |  A bilingual visual question-answering dataset consisting of 642 images and 5,980 question-answer pairs. | 6K
MIMIC-CXR [16] |  A chest image-report dataset contains 377k images corresponding to 227k studies. | 227K
2D $\&$ 3D Image-text data |  | Pair number
Rad3D-Caption |  A medical dataset consists of images and corresponding captions. | 73k
Rad3D-VQA |  A medical visual question-answering dataset consists of 205k VQA pairs generated from the captions in Rad3D-Caption. | 205k
Rad3D-Modality |  A medical vision-language dataset contains the modality question for each image. | 264K
Rad3D-Rationale |  A medical vision-language dataset contains disease rationale diagnosis for each case which may contain a series of radiology images. | 73K

00footnotetext:

Table 2: Description of the filtered dataset Radiology Multimodal Dataset
(RadMD) for domain-specific fine-tuning.

Dataset Name |  Filter Strategy | Filtered Data Size
---|---|---
Image data |  | Image number
Diagnosis datasets |  Balancing classes and prompt some judegment diagnosis question answer pairs. | 487K
Image-text data |  | Pair number
PMC-OA [22] |  All cases will be dismissed. | 0
PMC-Inline |  All cases will be dismissed. | 0
PMC-VQA [44] |  Keep all data without any exclusions. | 413K
PMC-CaseReport |  Dismiss the non-radiologic cases; avoid some cheating cases in context; remove the question about size and age. | 438K
MPx-Single |  Keep all data without any exclusions. | 114K
MPx-Multi |  Keep all data without any exclusions. | 39K
VQA-RAD [18] |  Keep all data without any exclusions. | 3.5K
SLAKE [23] |  Keep all data without any exclusions. | 14K
MIMIC-CXR [16] |  Keep all data without any exclusions. | 227K
2D $\&$ 3D Image-text data |  | Pair number
Rad3D-Caption |  Dismiss the non-radiologic cases; remove phrases that contain size. | 48K
Rad3D-VQA |  Dismiss the non-radiologic cases; Remove the question about size and age. | 205K
Rad3D-Modality |  Dismiss the non-radiologic cases. | 176k
Rad3D-Rationale |  Dismiss the non-radiologic cases. | 48K

00footnotetext:

####  3.1.1 Interleaved Dataset

PMC-Inline. PMC-Inline contains 11M 2D radiology images that are collected
from PubMed Central papers. In contrast to existing work, for example, PMC-OA
[22], that only contains figures and corresponding captions, here, we focus on
the inline reference from the main body of papers. For example, one paper may
contain many sentences like “As shown in Fig.2, we can see …”, we track the
keyword “Fig.2” and link its corresponding figure back into sentences, ending
up with interleaved images and texts, with rich context. This dataset shares
the same format as MMC4 [47], which has shown to be effective in training
foundation models in computer vision community, for example, Flamingo [1].

####  3.1.2 Visual-language Instruction Tuning Dataset

PMC-CaseReport. Inspired by PMC-Patients [46], PMC-CaseReport is a filtered
subset of PMC-Inline with around 103K cases report papers, where the doctors
typically document the valuable clinical cases, based on their contact with
the patients, such as family medical history, preliminary diagnosis,
radiographic exam results, surgical records, etc., together with critical
radiologic scans, that generally follows the real timeline.

Similar to PMC-VQA [44] that generates VQA pairs by querying ChatGPT with
image captions, we also generate 1.1M question-answer pairs by querying
ChatGPT with the sentences containing inline references in case reports.
However, in contrast to PMC-VQA, we keep background information of the
patients to simulate the clinical diagnosis scenario, thus can be seen as a
medical contextual VQA dataset. For example, a question-answer pair may like
“Question: A 58-year-old woman presented to the emergency department
…Postoperatively, her pain significantly relieved. What did the MRI indicate?
Answer: The MRI indicated tumor recurrence at L2 and S1-S2.”

Rad3D. Rad3D is a novel dataset with 3D radiology scans, sourced from
Radiopaedia website 111https://radiopaedia.org/, that has already resolved the
privacy issue. Specifically, each patient case comprises one or more modality
images, accompanied with high-quality captions that have been meticulously
verified by expert editors. In addition, for each disease, we can get
corresponding radiological features across different modalities.

In this paper, we convert the image-caption pairs into a variety of formats,
namely, Rad3D-Caption, Rad3D-Modality, and Rad3D-Rationale, Rad3D-VQA,
depending on their corresponding text content. Specifically, Rad3D-Caption
represents the images paired with their corresponding captions. Rad3D-Modality
refers to images with modality labels. Rad3D-Rationale incorporates
radiological features with disease labels for each case. Rad3D-VQA involves
visual question-answering pairs generated from captions by querying ChatGPT,
as illustrated in Fig. 3.

Figure 3: messages we use to prompt ChatGPT to generate medical visual
question-answering data for Rad3D. Manually curated few-shot examples are
included in the prompt, where each example has input sample[‘context’] and
output sample[‘response’].

MPx. MPx is collected from MedPix website 222https://medpix.nlm.nih.gov/ and
organized by cases. Each case contains multiple radiologic scans, along with
general clinical findings, discussions, and diagnostic results. In addition,
MPx also provides annotations on the scan-level, including information such as
image modality, shooting plane, and captions for each scan. Thus we separate
it into MPx-Single and MPx-Multi, containing annotations on case-level and
scan-level respectively.

Figure 4: The data statistics of RadMD. The left image shows the distribution
of 2D and 3D images. The center image shows the distribution of different
modalities. The collected dataset comprises more than 16M data samples from a
diverse range of radiology modalities. And the right image shows the
distribution of five downstream tasks, including modality recognition, disease
diagnosis, medical visual question answering, radiology report generation, and
rationale diagnosis.

###  3.2 Radiology Multimodal Dataset (RadMD)

For domain-specific finetuning, we filter out the non-radiology images from
MedMD, and construct a clean subset, named Radiology Multimodal Dataset
(RadMD), dedicating to supervised visual instruction-tuning. It contains a
total of 3M images, spanning various data formats, modalitieis, and tasks, as
shown in Fig. 4.

In general, we have conducted the following filtering process: (i) remove the
PMC-OA and PMC-Inline datasets due to their noisy nature, also the
inconsistency in writing styles between academic papers and real clinical
reports; (ii) remove non-radiologic images; (iii) filter out the information
about patient age or structure size, as the image spacing and patient
background information are not provided, thus only incomplete information
exists; (iv) balance the number of normal and abnormal patients in the
diagnosis datasets, as generative models are sensitive to data imbalances.
More comprehensive details regarding the filtering process and the resulting
dataset sizes can be found in Tab. 2.

##  4 Towards Building Generalist Foundation Model for Radiology

In this section, we start by describing the learning paradigm for unifying
different medical tasks into a generative framework, followed by detailing the
proposed RadFM model, and its training details. Our training adopts two types
of datasets, namely, interleaved datasets and visual instruction datasets. It
is worth noting that their training objectives differ slightly, which will be
detailed in the following.

###  4.1 Unified Learning Paradigm

In both of our proposed multimodal datasets, i.e., MedMD and RadMD, each
training sample is essentially consisting of two elements, i.e.,
$\mathcal{X}=\\{\mathcal{T},\mathcal{V}\\}$, where $\mathcal{T}$ refers to the
language part in the case, with special placeholder tokens for images, _e.g._
, “The patient is 47-year-old. <image-1> <image-2> We can see opacity on the
X-ray.”. $\mathcal{V}$ is the visual parts containing a set of 2D or 3D image
scans, i.e., $\mathcal{V}=\\{v_{1},v_{2},\dots,v_{N}\\}$,
$v_{i}\in\mathbb{R}^{H\times W\times C}$ or $v_{i}\in\mathbb{R}^{H\times
W\times D\times C}$, $H,W,D,C$ are height, width, depth, channel respectively,
corresponding to the “<image-$i$>” token in $\mathcal{T}$. In general,
$\mathcal{T}$ and $\mathcal{V}$ can be considered as prompts input to model
with interleaved language and image.

The goal is therefore to model the likelihood of generated text tokens in
$\mathcal{T}$, conditioned on interleaved medical scans as follows:

$p(\mathcal{T}|\mathcal{V})=\prod p(\mathcal{T}_{l}|\mathcal{V}_{\textless
l},\mathcal{T}_{\textless l}),$ (1)

where $\mathcal{T}_{l}$ represents the l-th token in $\mathcal{T}$ and
$\mathcal{V}_{\textless l}$, $\mathcal{T}_{\textless l}$ represent the scans
and language text appearing before the l-th token. We use a generative model
($\Phi_{\text{RadFM}}$) to parameterize the probability $p$ and our final loss
object can be expressed as the negative log-likelihood of the correct next
token in the text sequence:

$\mathcal{L}_{\text{reg}}=-\sum
w_{l}\log\Phi_{\text{RadFM}}(\mathcal{T}_{l}|\mathcal{V}_{\textless
l},\mathcal{T}_{\textless l}),$ (2)

where $w_{l}$ refers to a per-token weighting, aiming to either emphasize key
tokens or skip special tokens. Its value differs for different datasets and we
detail this in the following.

Interleaved Dataset. For samples in visual-language interleaved dataset, i.e.,
PMC-Inline, there are no strong question-and-answer relationships between
contexts, we extract medical-related words in each sentence by using Unified
Medical Language System (UMLS) [4], and give them a high loss weights.
Additionally, we avoid calculate loss on the image placeholder token. Overall,
$w_{l}$ can be formulated as,

$w_{l}=\left\\{\begin{aligned} 3,\quad&{\mathcal{T}_{l}\in\text{USML}}\\\
1,\quad&{\mathcal{T}_{l}\notin\text{USML}}\\\
0,\quad&{\mathcal{T}_{l}=\text{<image-i>}}\end{aligned}\right.,$ (3)

Note that, PMC-Inline is the only datasets fit in this case.

Visual Instruction Datasets. For samples from visual instruction datasets like
PMC-VQA [44] or PMC-CaseReport, they are often in the format of dialogue, for
example, “What can you see from the image? <image-1> I can see lesions.” or
“Please dscribe the scans <image-1>. The scan is …”, we further separate the
language part $\mathcal{T}$ into instruction and response, denoted as
$\mathcal{I}$ and $\mathcal{R}$ respectively. For example, as in the former
two cases, $\mathcal{I}$ refers to “‘What can you see from the image?
<image-1>” and “Please dscribe the scans <image-1>”. In a practical scenario,
$\mathcal{I}$ is expected to be given by users, and the model is only required
to output correct responses. Overall, $w_{l}$ can be formulated as,

$w_{l}=\left\\{\begin{aligned}
3,\quad&{\mathcal{T}_{l}\in\mathcal{R}}\quad\&\quad{\mathcal{T}_{l}\in\text{USML}}\\\
1,\quad&{\mathcal{T}_{l}\in\mathcal{R}}\quad\&\quad{\mathcal{T}_{l}\notin\text{USML}}\\\
0,\quad&{\mathcal{T}_{l}\in\mathcal{I}}\end{aligned}\right..$ (4)

Most samples from MedMD fit weighting formulation, and we describe the
detailed prompting for different problem settings:

  * •

Modality recognition. Here, we adopt two types of prompts, (i) we use
judgment-type templates and the 2D or 3D medical scan as input, for example,
“<image-1> Is this image shot by {modality}?”, and the modality category is
randomly sampled from the modality set, forming the text input $\mathcal{I}$
and if the modality matches the ground truth labels we set the $\mathcal{R}$
as “yes” otherwise “no”. (ii) we use open prompts, like “What’s the modality
of the input scan <image-1>?” to form the $\mathcal{I}$, and translate the
corresponding modality label into $\mathcal{R}$. Samples for training such
functionality are from Rad3D-Modality and MPx-Single, with modality
annotations available.

  * •

Disease diagnosis. All the datasets listed as “image data” in Table 1 are
built for diagnosis, they only have binary labels for diseases. Similarly to
modality recognition, we use two prompts to transform them into our desired
format, (i) we use judgment prompts, like “<image-1> Does the patient have
{disease}?” and the disease category is randomly sampled from a disease set,
forming the text input $\mathcal{I}$ and if the disease matches the ground
truth labels we set the $\mathcal{R}$ as “yes” otherwise “no”, note that,
during sampling, we balance the positive and negative ratio, (ii) we use open
diagnosis prompts, like “Please make diagnosis based on the images <image-1>
<image-2>.” to construct the instruction ($\mathcal{I}$), and translate the
positive disease labels into response ($\mathcal{R}$), by simply using their
category names. A simple example is, $\mathcal{I}$=“Please make diagnosis
based on the image <image-1>.” with $\mathcal{R}$=“Edema, pneumothorax.”

  * •

Visual question answering. Beyond the abovementioned task formulation, there
are many more complex questions that can be asked, such as questions about the
spatial relationships among objects (What is the location of the lesion?) and
common sense reasoning questions (Given the image context and patient history,
what is likely to be cause of the observed symptoms?). A robust medical VQA
system must be capable of solving a wide range of classic medical diagnosis
tasks as well as the ability to reason about images. Existing medical VQA
datasets like VQA-RAD [18], SLAKE [23], PMC-VQA [44] and Rad3D-VQA naturally
fit into this paradigm. They contain a mixture of question types, thus the
language questions can naturally be treated as text instruction
($\mathcal{I}$) and the corresponding answer as response ($\mathcal{R}$). It
is worth noting that, our constructed PMC-CaseReport dataset also falls into
this category, with more contextual information available for instruction, for
example, history diagnosis, is also available, thus providing critical
information for answering the question.

  * •

Report generation. MIMIC-CXR [16], Rad3D-Caption, PMC-OA [22], MPx-Multi and
some of MPx-Single are all captioning datasets, the task for the model is to
write a long caption or report given one or a set of images. The language
instruction for this task are like “What can you find from the scans <image-1>
<image-2>?”.

  * •

Rationale Diagnosis. We construct Rad3D-Rationale based on the Rad3D dataset.
This task encompasses disease prediction and the generation of typical
radiological features associated with the diagnosed disease. Specifically, we
design some prompts like “What disease can be diagnosed from these
radiological images and what specific features are typically observed on the
images? <image-1> <image-2>” as instruction ($\mathcal{I}$), and response
($\mathcal{R}$) refers to the disease label along with radiological features
collected from the Radiopaedia website.

Figure 5: (a) shows the the component of our datasets and the colored datasets
are new proposed in this paper. (b) shows our training procedure, for better
radiologic performance, we first pre-train our model on the whole medical
domain with 16M scans (MedMD) than fine-tuned on a cleaned dataset with 3M
radiologic scanss (RadMD). (c) shows the main architecture of our method. Our
architecture enables multi-image input interleaving with texts regardless of
whether they are 3D or 2D.

###  4.2 Architecture Detail

In this section, we aim to describe the proposed model in detail. As shown
Fig. 5, our proposed RadFM model consists of a visual encoder
$\Phi_{\text{vis}}$, that can process both 2D and 3D medical scans; a
perceiver module $\Phi_{\text{per}}$ for aggregating a sequence of scans, for
example, taken with different modalities (CT, MRI) or various time point; and
a Large Language Model (LLM) $\Phi_{\text{llm}}$ for fusing the visual-
language information, and outputting free-form text responses.

Visual encoding. Given one sample instance from our dataset, denoted as
$\mathcal{X}=\\{\mathcal{T},\mathcal{V}\\}$, where
$\mathcal{V}=\\{v_{1},v_{2},\dots,v_{N}\\}$, we first encode each input image
separately with an image encoder $\Phi_{\text{vis}}$. Specifically, we adopt
3D ViT here to be compatible with both 2D and 3D image input. For 2D images,
we expand a new dimension for depth with replication. Therefore, each image
scan can be denoted as $v_{i}\in\mathbb{R}^{H\times W\times D_{i}\times C}$,
where $C$ denotes the image channels and $H,W,D_{i}$ are the height, width,
and depth of the image respectively.

Note that, comparing to the typical visual encoding scenario that assumes
different images have unified height width depth size, we do not normalize the
depth dimension into an exact size, only round into a factor of 4, depending
on their original resolution. We convert the image into 3D patches, and embed
them into a token sequence and feed into the encoder ($\Phi_{\text{vis}}$). To
retain the 3D position of these tokens, we adopt relatively learnable 3D
position embeddings, the detailed procedure can be formulated as

$\bm{v}_{i}=\Phi_{\text{vis}}(v_{i})\in\mathbb{R}^{P_{i}\times d},$ (5)

where $\bm{v}_{i}$ is the embedding for image $v_{i}$, $P_{i}$ is the total
number of tokens, and $d$ is the feature dimension. Due to the inconsistency
in depth dimension, $P_{i}$ also varies across 2D and 3D images. Generally, 3D
cases may have much larger $P_{i}$ than 2D cases, as it is difficult to put
them together into a unified training flow.

Aggregation with Perceiver. After visual encoding, we adopt a perceiver [15]
module $\Phi_{\text{per}}$ to aggregate visual representation. Specifically,
$\Phi_{\text{per}}$ is a transformer decoder with a fix number of learnable
queries, and the visual input is treated as a key and value for the decoder,
such that final output embeddings will be normalized into the same dimension
with the pre-defined learnable query sequence. The aggregation procedure can
be formulated as:

$\bm{u}_{i}=\Phi_{\text{per}}(v_{i})\in\mathbb{R}^{P\times d},$ (6)

where $\bm{u}_{i}$ refers to the aggregated visual embedding, $P$ is the pre-
defined number of queries. Leveraging perceiver architecture, we can map
arbitrary patch tokens into the same amount of queries so that images of
different sizes can be viewed equally in the following fusion flow.

Multi-modal fusion. To fuse the visual-language information, we interleave the
visual embedding with text embeddings from tokenization, where the special
image placeholder token is simply replaced with the corresponding visual
embedding. The resulting interleaved sequence is then passed into a decoder-
only large language model ($\Phi_{\text{llm}}$) to obtain the output, its
self-attention transformer layers can thus naturally be re-used as multi-modal
fusion modules:

$p=\Phi_{\text{llm}}(\text{concat}(\bm{t}_{1},\bm{u}_{1},\bm{t}_{2},\bm{u}_{2},\bm{t}_{3},\dots)),$
(7)

where $t_{i},u_{i}$ refer to the text and visual embeddings, $p$ is the
probability distribution for the next token.

###  4.3 Training Procedure

Our training procedure includes two stages, namely, pre-training, and domain-
specific fine-tuning, as shown in Fig. 5. Note that, all training settings are
the same in the two stages, with the only distinction lying in the training
data, from generalist to radiologic-specific.

Pre-training. At this stage, we use all available data in MedMD as listed in
Table 1, the main components of the data are PMC-Inline and PMC-OA [22], which
are all collected from 2.4M PMC papers. These two datasets contain diverse
medical vocabularies and images with cutting-edge medical knowledge, however,
they are relatively noisy, so we only use them during pre-training in the hope
that the network can accumulate enough knowledge about medical-specific
terminologies and images. Additionally, we also include other VQA, captioning,
and diagnosis datasets, as they are much cleaner.

Domain-specific fine-tuning. At this stage, we adopt RadMD for domain-specific
finetuning, which contains over 3M radiologic images, with high-quality
language instruction or response.

####  4.3.1 Training Details

Image preprocessing. To dismiss the differences of medical images in different
modalities, certain preprocessing steps are applied. Specifically, (i) to
align the intensity distributions, we employ min-max normalization. Note that,
since the MRI and CT images in our dataset are collected from online
resources, they have already been clipped to $[0,255]$ thus we do not apply
clip operation to them again here as a usual preprocessing pipeline. (ii)
given that medical images can exist in either 3D or 2D formats (such as MRI
being 3D and X-ray being 2D), we convert all 2D images to 3D simply by
expanding an extra dimension. Consequently, all images, irrespective of their
original format, can be processed uniformly as 3D images; (iii) to ensure
consistent sizes across all images, we resize them using the
torchvision.transforms.Resize function. For height and weight dimensions, we
resize them to $512\times 512$ for 2D images and $256\times 256$ for 3D images
because 3D data has more slices, thus taking too much computational
memorization. For depth dimension, since our visual encoder, a 3D Vision
Transformer (ViT), requires the input image sizes to be divisible by the patch
size of $32\times 32\times 4$, we resize the depth dimension to the nearest
multiple of 4 and will not surpass 64.

Implementation. For the visual encoder, we adopt a 12-layer 3D ViT with 768
feature dimensions and the perceiver is chosen as a 6-layer transformer
structure with 32 query tokens and 5120 feature dimensions so that all images
will embed as a $32\times 5120$ feature embedding after visual encoding. When
inserting them into the text embedding, we will add two extra special tokens
<image>, </image> at the beginning and ending respectively to distinguish them
from common text tokens. For the large language model, we initialize it with
the MedLLaMA-13B model introduced by PMC-LLaMA [41], which has further fine-
tuned the LLaMA-13B [37] model on the medical corpus. Our final model has 14B
parameters.

In training, we use a diverse batch size strategy, _i.e._ , 1 batch size per
device for 3D images and 4 batch size per device for 2D images with 4-step
gradient accumulation, and the max token length is set to be $2048$. We
totally train the model for 6 epochs, 4 epochs for pre-training and 2 epochs
for instruction-tuning. In the first 1 epoch, we freeze the language model to
align image embedding space with that of texts and in the following epochs,
all parameters will be trained. To improve the training speed, we adopt FSDP
acceleration strategy [45], together with Automatic Mixed Precision (AMP) and
gradient checkpointing [8]. All models are implemented in PyTorch and trained
on 32 NVIDIA A100 GPUs with 80 GB memory.

##  5 Evaluation

In this section, we first introduce the evaluation metrics employed,
subsequently, we present five evaluation tasks and corresponding datasets.

###  5.1 Metrics

In our evaluation, we adopt various standard metrics developed in the
community, for example, F1 stands for “F1 score”, ACC stands for “Accuracy”,
BLEU stands for “BiLingual Evaluation Understudy” [32], ROUGE stands for
“Recall-Oriented Understudy for Gisting Evaluation” [21]. For BLEU and ROUGE
we all use 1-gram by default.

In addition, inspired by the score RadCliQ [42] designed specific for
evaluating generated chest X-ray reports, we also propose two new metrics,
UMLS-Precision and UMLS-Recall, which aim to measure the overlapping ratio of
medical-related words between ground truth and predicted response for common
medical paragraphs. Specifically, given a pair of ground-truth and prediction
sentences, we extract the medical-related words from them by using Unified
Medical Language System (UMLS) [4], and count the overlap words as true-
positive. UMLS-Precision is defined following classical precision concept,
_i.e._ , the number of true-positive divides the whole generated medical-
related word number. On the other hand, UMLS-Recall also follows the recall
concept, _i.e._ , the number of true-positive words divides the total number
of medical-related words in ground-truth.

###  5.2 Evaluation Tasks

Modality recognition. This task involves analyzing the radiology images to
determine the modality of the input radiology images. Here we modify this task
to a prompt-based visual question-answering task, i.e., given a medical image,
we randomly select a prompt sentence like “What is the modality of the given
image?”, and match the output with a ground-truth list {‘CT’, ‘MRI’,
‘Ultrasound’, ‘PET’, ‘X-ray’, ‘angiography’} using difflib.SequenceMatcher,
and choose the most similar one as the prediction of the model to calculate
the ACC and F1 score.

Disease diagnosis. This task involves analyzing the radiology images to
determine the likelihood of specific diseases. Here we modify this task to a
prompt-based visual question-answering task, given a medical image, we
randomly select a disease and randomly select a prompt sentence like “Is
{disease} shown in this image” as input to ask the network to answer whether
the case has this disease. Due to this being formulated as a generation task,
“AUC” cannot be calculated, so we match the output with ground-truth to
calculate the ACC and F1 score. Similarly, we match the output with a closed
ground-truth list {‘yes’, ‘no’} using difflib.SequenceMatcher, and choosing
the most similar one as the prediction of the model. Considering ACC scores
may suffer from data unbalancing, we keep the same ratio to sample positive
and negative cases. In our dataset, we do not put prior on the disease, and
over 5000 diseases are considered, with a balanced ratio of ‘yes’ or ‘no’
responses.

Medical visual question answering. This task is a combination of popular
visual question-answering challenges. Given a medical image and a clinically
relevant question in natural language as a prompt, the medical VQA system is
expected to predict a plausible and convincing answer. We adopt the widely-
used evaluation metrics BLEU [32]. Additionally, we use UMLS-Precision and
UMLS-Recall to evaluate the accuracy of medical-related word prediction.

Radiology report generation. This task focuses on the automatic generation of
diagnostic reports, i.e., summarizing the radiologic findings and conclusions
based on radiology images, such as X-rays, CT scans, and MRI scans. Given a
medical image, we randomly select a prompt sentence like “Please caption this
scan with findings and impression.” as input. We adopt the widely-used
evaluation metrics BLEU [32], and our proposed UMLS-Precision and UMLS-Recall
to evaluate the accuracy of medical-related word prediction.

Rationale diagnosis. This task involves analyzing radiology images to predict
both the underlying disease and the typical radiologic features of different
modalities such as X-rays, CT scans, and MRI scans associated with that
disease. Given a medical image, we randomly select a prompt sentence like
“Determine the disease that corresponds to the given radiographic images and
describe the characteristic radiologic features.” Since We have evaluated
disease diagnosis accuracy in the common “Disease Diagnosis” setting, for
rationale diagnosis, we mainly focus on how well the foundation model can give
reasons. Thus, we adopt BLEU, ROUGE, UMLS-Precision and UMLS-Recall here, to
measure the plausibility of the rationale prediction.

###  5.3 Evaluation Datasets

We use the combination of 13 datasets across a wide variety of distributions
and tasks. As shown in Table 3, we provide detailed information on the task
description and modalities of each evaluated dataset. In Fig.4, we also show
which task each dataset contributes to.

Table 3: Description of the downstream evaluation datasets.

Dataset Name |  Task Description |  Modalities |  Split
---|---|---|---
Rad3D-Modality |  Modality Recognition |  Image / Text |  Random split
VinDr-Mammo [28] |  Disease diagnosis |  Image |  Official split
VinDr-SpineXR [29] |  Disease diagnosis |  Image |  Official split
VinDr-PCXR [30] |  Disease diagnosis |  Image |  Official split
CXR-Mix [40] |  Disease diagnosis |  Image |  Official split
PMC-VQA [44] |  Medical visual question answering |  Image / Text |  Official split
PMC-CaseReport |  Medical visual question answering |  Image / Text |  Random split
VQA-RAD [18] |  Medical visual question answering |  Image / Text |  Official split
SLAKE [23] |  Medical visual question answering |  Image / Text |  Official split
Rad3D-VQA |  Medical visual question answering |  Image / Text |  Random split
MIMIC-CXR [16] |  Report generation |  Image / Text |  Random split
Rad3D-Caption |  Report generation |  Image / Text |  Random split
Rad3D-Rationale |  Rationale diagnosis. |  Image / Text |  Random split

22footnotetext:

###  5.4 Baseline Models

To our knowledge, there are currently no existing medical foundation models
that cam effectively foundation models that can effectively handle both 2D and
3D radiology images. For comparison, we pick two baseline models, OpenFlamingo
[3] and MedVInT [44], which have demonstrated efficacy in processing single
slices and making predictions.

OpenFlamingo [3]. This is an open-source implementation of the prior state-of-
the-art generalist visual language model Flamingo [1], that was trained on
large-scale data from general visual-language domain. We utilized the released
checkpoint for zero-shot evaluation in our study.

MedVInT [44]. This is a visual instruction-tuned visual language model based
on LLaMA [37], which was trained on PMC-VQA [44]. We directly use the released
checkpoint of the MedVInT-TD model with PMC-LLaMA and PMC-CLIP backbone for
zero-shot evaluation.

##  6 Results

In this section, we present the evaluation results for RadFM across five
diverse tasks (Fig. 6), including modality recognition, disease diagnosis,
report generation, and visual question-answering on radiologic modalities and
anatomical regions. Results are summarized in Tab. 4.

Figure 6: Examples of inputs and outputs of five different evaluation tasks obtained from RadFM. Table 4: Comparison of proposed RadFM with SOTA model on RadBench. The benchmark includes 5 tasks, modality recognition, disease diagnosis, medical visual question answering, report generation, and rationale diagnosis. ACC, F1, BLEU, ROUGE, $\text{UMLS}\\_\text{Precision}$, and $\text{UMLS}\\_\text{Recall}$ are reported, and the metrics refer to the average score on all test samples. Numbers within parentheses indicate $95\%$ CI.  Tasks | Test Cases | Metric |  OpenFlamingo [3] |  MedVInT [44] | RadFM
---|---|---|---|---|---
Modality Recognition | n=17896 | ACC |  | 14.47%
---
(12.67%,16.61%)
| 86.19%
---
(84.33%,87.62%)
| 86.33%
---
(85.55%,87.13%)
Disease Diagnosis | n=129985 | ACC |  | 53.76%
---
(53.35%,54.19%)
| 54.96%
---
(54.04%,55.88%)
77.88% (77.47%,78.25%)
F1 |  | 23.08%
---
(22.44%,23.84%)
| 45.94%
---
(44.83%,47.04%)
| 77.20%
---
(76.78%,77.62%)
Medical VQA | n=154540 | BLEU |  | 13.23%
---
( 13.11%, 13.35%)
| 14.07%
---
(13.79%, 14.28%)
| 31.43%
---
(31.20%,31.65%)
ROUGE |  | 21.16%
---
(20.93%,21.38%)
| 18.85%
---
(18.57%, 19.08%)
| 36.84%
---
(36.58%,37.14%)
UMLS_Precision |  | 14.55%
---
(14.34%,14.76%)
| 13.09%
---
(12.78%,13.39%)
| 34.13%
---
(33.81%,34.47%)
UMLS_Recall |  | 14.56%
---
(14.33%,14.79%)
| 10.52%
---
(10.26%,10.78%)
| 30.11%
---
(29.68%,30.25%)
Report Generation | n=6552 | BLEU |  | 7.37%
---
(6.54%,8.21%)
| 1.99%
---
(1.63%,2.47%)
| 14.68%
---
(14.12%,15.20%)
ROUGE |  | 9.49%
---
(8.59%,10.35%)
| 4.31%
---
(3.95%,4.79%)
| 20.71%
---
(20.33%,21.19%)
UMLS_Precision |  | 8.86%
---
(7.91%,9.81%)
| 6.90%
---
(5.81%,7.98%)
| 23.42%
---
(22.79%,24.10%)
UMLS_Recall |  | 5.88%
---
(5.07%,6.69%)
| 1.85%
---
(1.43%,2.26%)
| 13.21%
---
(12.76%,13.76%)
Rationale Diagnosis | n=1392 | BLEU |  | 3.06%
---
(2.21%,4.05%)
| 1.30%
---
(0.48%,2.28%)
| 11.56%
---
(10.25%,13.02%)
ROUGE |  | 5.10%
---
(4.09%,6.20%)
| 2.10%
---
(1.29%,2.95%)
| 20.56%
---
(19.26%,21.97%)
UMLS_Precision |  | 8.03%
---
(6.17%,9.88%)
| 9.63%
---
(7.68%,11.57%)
| 31.19%
---
(28.42%,33.60%)
UMLS_Recall |  | 2.07%
---
(1.14%,3.22%)
| 1.45%
---
(0.59%,2.31%)
| 13.03%
---
(11.66%,14.36%)

###  6.1 Modality Recognition

Recognizing image modality is one basic skill expected of any advanced
foundation models, however, as depicted in Tab. 4, the foundation model
(OpenFlamingo) developed in computer vision community encounters significant
challenges on this task, with a disappointing accuracy score (ACC) of merely
$14.47\%$, highlighting the lack of basic medical knowledge in latest multi-
modal foundation model. In contrast, our proposed RadFM model, surpasses the
most advanced state-of-the-art (SOTA) model in the field. Note that, such task
has been formulated as an open-ended question, where the model’s output needs
to explicitly state the recognized modality as ‘CT’, ‘MRI’, etc.

###  6.2 Disease Diagnosis

In clinical practice, the primary and indispensable function of any advanced
medical foundation model is to provide valuable assistance in disease
diagnosis. As depicted in Tab. 4, the performance of previous disease
diagnosis methods has been rather poor, with an achieved accuracy score (ACC)
of only $54.96\%$. Considering that we prompt the problem with a judgment
format, _i.e._ , “Does the patient have {disease}?”, this score is nearly
random. In contrast, our proposed model, RadFM, proves its superiority over
existing methods with an ACC score of $77.88\%$, making it more aptly suited
for real-world clinical applications. Essentially, this resembles a multi-
label classification task, with over 5000 distinct categories, further
compounding the challenges of models.

Figure 7: Qualitative examples of medical visual question answering (VQA). We
present several examples with answers generated by RadFM along with the target
ground truth. The green color highlights accurate keywords, while the red
color indicates prediction errors.

###  6.3 Medical VQA

In contrast to the aforementioned specific tasks, Medical Visual Question
Answering (VQA) represents a more comprehensive and versatile challenge in the
field of medical image analysis. In a clinical setting, patients and
radiologists may pose a wide variety of questions related to medical images,
ranging from simple inquiries about image modality to more complex reasoning
queries. Here, we combine 5 different medical VQA datasets for evaluation. As
shown in Fig. 7, RadFM demonstrates its ability to comprehend the questions
and provide answers in a consistent format, accurately addressing the
questions. However, in some challenging cases, such as the first example where
the question pertains to the type of abnormality, the model faces difficulty
predicting “ectopic ACTH-producing tumor” and mistakenly identifies it as
“primary lung neoplasm”, that requires fine-grained discrimination within
tumor types.

MedVInT, the model trained on visual question answering, is still struggling
when the VQA task (on PMC-Casereport) requires a deeper understanding of the
context related to the patient and image. This challenge arises due to the
need for not only image-specific knowledge but also a comprehensive grasp of
the patient’s clinical background. In contrast, our RadFM model surpasses all
other methods on medical VQA. Specifically, we observe a substantial
improvement in UMLS_Precision from 14.55% to 34.13% and UMLS_Recall from 4.56%
to 30.11%, demonstrating its proficiency in comprehensively understanding the
given textual information.

###  6.4 Report Generation

Report generation is a crucial and prominent use case for generative medical
foundational models. Unlike Medical VQA, this application generally requires
the model to emphasize clinically significant observations based on the image.
As shown in Tab. 4, MedVInT, which is trained on VQA datasets with limited
long responses, fails to perform well on this task. Although OpenFlamingo
performs better than MedVInT, it is still disappointing. On the other hand,
RadFM shows significant improvement across various metrics, particularly in
relation to medical-specific terminology. For instance, RadFM improves
UMLS_Precision from 8.86% to 23.42%, and UMLS_Recall from 5.88% to 13.21%.

In Fig. 8, we provide qualitative examples of radiology reports generated by
RadFM. It can be observed that the model is capable of identifying fundamental
diseases and, in some cases, performs exceptionally well. However, the report
generated by MedFM may lack specific location information like the ‘left’ or
‘right’ of an anatomy region.

Figure 8: Qualitative examples of report generation. We present several
examples with reports generated by RadFM and reference reports. The green
color highlights accurate keywords, while the red color indicates prediction
errors. Additionally, the blue color denotes instances where the model missed
this information that has been mentioned in the reference reports.

###  6.5 Rationale Diagnosis

In addition to basic diagnosis, the ability to scrutinize diagnostic
prediction outcomes is crucial, particularly in light of the stringent demands
for precision and interpretability within medical contexts. Much like report
generation, this task also requires proficiency in generating extended
paragraphs and comprehensive understanding on medical knowledge.

As indicated in Table 4, the findings highlight RadFM as the sole foundational
model that effectively responds on this task. RadFM outperforms traditional
BLEU and ROUGE scores by 8.56% and 15.46% respectively. Moreover, it exhibits
significant improvements in UMLS_Precision and UMLS_Recall scores, showcasing
advancements of 23.16% and 10.96% respectively. These results suggest that
RadFM is capable of generating rationale sentences that are more accurate and
meaningful.

Fig. 9 shows two rationale diagnosis cases. The first case is a patient with
Pulmonary embolism and the latter is with Subarachnoid haemorrhage. On both
cases, RadFM can make accurate diagnosis in free form and give further related
radiologic reasoning. However, the limitation can also be observed that the
reasoning results are still general and more like background medical
knowledge, yet not specific to the input case.

Figure 9: Qualitative examples of rationale diagnosis. We present several
examples where the model predicts diseases correctly.

##  7 Discussion

###  7.1 RadFM is the first foundation model that unified 2D and 3D radiologic
images.

In the field of radiologic images, one significant challenge on developing
foundation model lies on the disparity of image dimensions, i.e., medical
scans are either 2D or 3D, posing challenges on integrating real 3D MRI or CT
images alongside with 2D images like X-rays or ultrasounds. As a consequence,
the development of foundational models has been significantly impeded, with
most current models only accommodating 2D images. To overcome these
limitations, we propose a new training structure that unifies 2D and 3D
images, allowing a wide variety of real clinical images to be fed into one
network. By unifying the training process, our model benefits from a more
comprehensive understanding of the diverse clinical images, leading to
improved performance and versatility. Additionally, to facilitate research and
foster collaboration in the field, we collect a new medical multimodal
dataset, containing 16M 2D or 3D medical scans with text descriptions or
labels.

###  7.2 RadFM unifies the medical tasks with a generative model.

While developing AI for medicine, traditional approaches consider a divide-
and-conquer idea, that tackles a myriad of specific tasks individually, such
as diagnosis, report generation, and medical Visual Question Answering (VQA),
resulting in separated approaches with limited generalization ability and
efficiency. Here, we formulate diverse medical tasks as multi-modal question-
answering and develop a generative visual-language model RadFM that can answer
arbitrary questions or follow instructions. By unifying the tasks, RadFM
achieves promising performance across a wide spectrum of clinical tasks. On
the medical VQA task, RadFM surpasses the performance of MedVINT, a pre-
trained model trained solely on a single Medical VQA dataset.

###  7.3 RadFM supports multiple images as input.

Till now, most existing multi-modal foundation models in the medical field
have been limited to supporting only single image input per interaction.
However, such design poses critical challenges in medical scenarios where
diagnosis and treatment decisions often necessitate longitudinal clinical
follow-ups, that comprise a series of radiologic images. To overcome this
limitation and pave the way for more comprehensive medical image analysis, our
proposed RadFM supports multi-image input. To support the training, our
constructed dataset is largely composed of multi-image input data, and our
innovative training flow seamlessly accommodates this unique medical scenery,
fostering advancements in medical image analysis.

###  7.4 A general evaluation benchmark for radiology foundation models.

Evaluating the performance of medical foundation models is a complex
undertaking due to the specialized nature of medical tasks. In the pursuit of
advancing radiology foundation models, we propose a novel benchmark that
encompasses a diverse range of medical scenarios. Our benchmark comprises 5
tasks, namely modality recognition, disease diagnosis, visual question
answering, and report generation. By incorporating both 2D and 3D images, our
benchmark offers a more comprehensive and realistic evaluation platform
compared to existing benchmarks. Additionally, as existing evaluation metrics
are primarily designed for general natural language tasks, which may not
adequately capture the intricacies and nuances specific to medical image
analysis, thus may not reflect the model’s true capabilities in real-world
clinical scenarios. To address this limitation, we propose two new evaluation
metrics, namely UMLS Precision and UMLS Recall. Unlike conventional metrics,
UMLS Precision and Recall are tailored to measure the model’s performance in
medical tasks. By leveraging the Unified Medical Language System (UMLS), a
comprehensive medical knowledge resource, these metrics provide a more
tailored evaluation, ensuring that the model’s outputs align with medical
domain expertise.

###  7.5 The Superiority of RadFM.

As shown in Tab. 4 and Fig.1, RadFM outperforms previous methods by a
significant margin across all five tasks, showcasing its exceptional
capabilities. Notably, RadFM excels in particularly challenging tasks such as
medical VQA, report generation, and rationale diagnosis, which demand profound
understanding of both textual information and images. In medical VQA, the
questions can be drastically varying, from simple queries like “What modality
is the given image?” to more complex and context-rich questions, such as
“Based on the provided images, patient data (age, gender, medical history),
can you identify the disease that is commonly associated with such
radiological manifestations?” The complexity of questions makes medical VQA a
comprehensive and versatile task. By integrating visual and textual
information, RadFM can handle these varying question types, delivering
accurate and meaningful answers. Similarly, in report generation, RadFM
showcases significant improvement. The model’s ability to discern relevant
information from the provided images and weave it cohesively with textual
prompts leads to highly informative and contextually-rich reports, setting it
apart from traditional methods. Overall, the performance of RadFM across these
diverse tasks confirms its versatility and transformative potential in medical
image analysis.

###  7.6 Limitations.

Despite our efforts in developing a foundation model for radiology, RadFM
still exhibits several limitations:

First, the capacity to generate meaningful and accurate long sentences remains
underdeveloped. As demonstrated in Tab. 4, for rationale diagnosis and report
generation, the quantitative results surpass previous works but are still far
from satisfactory.

Second, the proportion of actual 3D images in the data remains limited. As
illustrated in Fig.4, although we attempt to compensate for the lack of 3D
images, yet 2D images continue to dominate.

Third, the evaluation metrics fall short of expectations. Compared to general
contexts where the emphasis is placed on the overall coherence and fluency of
sentences, medical texts prioritize precision in key statements and contain
many synonyms, like ‘MRI’ and ‘Magnetic Resonance Imaging’, overlooking minor
syntax errors. Although we employ UMLS-Precision and UMLS-Recall to mitigate
this issue, they do not fully reflect true performance. A robust evaluation
metric is essential to guide the construction of reliable and robust medical
foundation models.

Lastly, as the 3D images in our dataset are downloaded from the internet, some
metadata is missing, for example, the imaging spacing. Such lack of precise
distance measurement makes it impossible to make certain statements, such as
“The tumor is 3cm large”. This specification is crucial for report writing in
CT and MRI images.

###  7.7 Conclusion.

In conclusion, in this paper, we have constructed a complete set of medical
foundation model-building processes, including data collection, problem
formulation, model design, training, and evaluation. We construct the largest
medical multi-modal database in this paper. In model capabilities, compared to
existing work, our model is able to process multiple 3D or 2D image inputs
interleaved with texts, which fits the practical usage more. We surpass the
latest open-source multi-modal foundation model significantly. We will release
all corresponding data, codes, and models. We believe this can greatly promote
the development of medical foundation models.

##  8 Data availability

The datasets used for pre-training are listed in Table 5.

Table 5: Data availability. For the Rad3D-series and MPx-series datasets, researchers may obtain by reasonable request from the authors.  Dataset Name |  Link | Access
---|---|---
Rad3D-series |  - | Restricted Access
MPx-series |  - | Restricted Access
PMC-Inline |  https://huggingface.co/datasets/chaoyi-wu/PMC-Inline | Open Access
PMC-CaseReport |  https://huggingface.co/datasets/chaoyi-wu/PMC-CaseReport_original https://huggingface.co/datasets/chaoyi-wu/PMC-CaseReport | Open Access
VinDr-Mammo [28] |  https://www.physionet.org/content/vindr-mammo/1.0.0/ | Credentialed Access
VinDr-SpineXR [29] |  https://www.physionet.org/content/vindr-spinexr/1.0.0/ | Credentialed Access
VinDr-PCXR [30] |  https://physionet.org/content/vindr-pcxr/1.0.0/ | Credentialed Access
PMC-OA [22] |  https://huggingface.co/datasets/axiong/pmc_oa_beta | Open Access
PMC-VQA [44] |  https://huggingface.co/datasets/xmcmic/PMC-VQA | Open Access
VQA-RAD [18] |  https://osf.io/89kps/ | Open Access
SLAKE [23] |  https://www.med-vqa.com/slake/ | Open Access
MIMIC-CXR [16] |  https://physionet.org/content/mimic-cxr/2.0.0 | Credentialed Access
VinDr-CXR [27] |  https://physionet.org/content/vindr-cxr/1.0.0/ | Credentialed Access
NIH ChestXray14 [39] |  https://nihcc.app.box.com/v/ChestXray-NIHCC/folder/36938765345 | Open Access
CheXpert [12] |  https://aimi.stanford.edu/chexpert-chest-x-rays | Open Access
Covid-CXR2 [33] |  https://www.kaggle.com/datasets/andyczhao/covidx-cxr2 | Open Access
NLM-TB [14] |  https://openi.nlm.nih.gov/imgs/collections/NLM-MontgomeryCXRSet.zip https://openi.nlm.nih.gov/imgs/collections/ChinaSet_AllFiles.zip | Open Access
Object-CXR [11] |  https://web.archive.org/web/20201127235812/https://jfhealthcare.github.io/object-CXR/ | Open Access
OpenI [9] |  https://www.kaggle.com/datasets/raddar/chest-xrays-indiana-university | Open Access
RSNA [35] |  https://www.rsna.org/education/ai-resources-and-training/ai-image-challenge/rsna-pneumonia-detection-challenge-2018 | Open Access
SIIM-ACR [10] |  https://www.kaggle.com/datasets/jesperdramsch/siim-acr-pneumothorax-segmentation-data | Open Access

##  9 CODE availability

The code is available on GitHub at https://github.com/chaoyi-wu/RadFM

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning.  ArXiv, abs/2204.14198, 2022.
  * [2] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Tachard Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Z. Chen, Eric Chu, J. Clark, Laurent El Shafey, Yanping Huang, Kathleen S. Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Michael Brooks, Michele Catasta, Yongzhou Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, C Crépy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, M. C. D’iaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fan Feng, Vlad Fienber, Markus Freitag, Xavier García, Sebastian Gehrmann, Lucas González, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, An Ren Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wen Hao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Mu-Li Li, Wei Li, Yaguang Li, Jun Yu Li, Hyeontaek Lim, Han Lin, Zhong-Zhong Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alexandra Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Marie Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniela Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Ke Xu, Yunhan Xu, Lin Wu Xue, Pengcheng Yin, Jiahui Yu, Qiaoling Zhang, Steven Zheng, Ce Zheng, Wei Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu.  Palm 2 technical report.  ArXiv, abs/2305.10403, 2023.
  * [3] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt.  Openflamingo, Mar. 2023.
  * [4] Olivier Bodenreider.  The unified medical language system (umls): integrating biomedical terminology.  Nucleic acids research, 32(suppl_1):D267–D270, 2004.
  * [5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258, 2021.
  * [6] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.
  * [7] Qiuhui Chen, Xinyue Hu, Zirui Wang, and Yi Hong.  Medblip: Bootstrapping language-image pre-training from 3d medical images and texts.  arXiv preprint arXiv:2305.10799, 2023.
  * [8] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.  Training deep nets with sublinear memory cost.  arXiv preprint arXiv:1604.06174, 2016.
  * [9] Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald.  Preparing a collection of radiology examinations for distribution and retrieval.  Journal of the American Medical Informatics Association, 23(2):304–310, 2016.
  * [10] Ross W Filice, Anouk Stein, Carol C Wu, Veronica A Arteaga, Stephen Borstelmann, Ramya Gaddikeri, et al.  Crowdsourcing pneumothorax annotations using machine learning annotations on the nih chest x-ray dataset.  Journal of digital imaging, 33:490–496, 2020.
  * [11] JF Healthcare.  Object-cxr - automatic detection of foreign objects on chest x-rays.  2020\.
  * [12] Jeremy Irvin et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019.
  * [13] Fabian Isensee, Jens Petersen, Andre Klein, David Zimmerer, Paul F Jaeger, Simon Kohl, Jakob Wasserthal, Gregor Koehler, Tobias Norajitra, Sebastian Wirkert, et al.  nnu-net: Self-adapting framework for u-net-based medical image segmentation.  arXiv preprint arXiv:1809.10486, 2018.
  * [14] Stefan Jaeger, Sema Candemir, Sameer Antani, Yì-Xiáng J Wáng, Pu-Xuan Lu, and George Thoma.  Two public chest x-ray datasets for computer-aided screening of pulmonary diseases.  Quantitative imaging in medicine and surgery, 4(6):475, 2014.
  * [15] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  * [16] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  Scientific data, 6(1):317, 2019.
  * [17] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123:32–73, 2017.
  * [18] Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman.  A dataset of clinically generated visual questions and answers about radiology images.  Scientific data, 5(1):1–10, 2018.
  * [19] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  arXiv preprint arXiv:2306.00890, 2023.
  * [20] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  ArXiv, abs/2301.12597, 2023.
  * [21] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In Text summarization branches out, pages 74–81, 2004.
  * [22] Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Pmc-clip: Contrastive language-image pre-training using biomedical documents.  arXiv preprint arXiv:2303.07240, 2023.
  * [23] Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu.  Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering.  In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE, 2021.
  * [24] Michael Moor, Oishi Banerjee, Zahra F H Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar.  Foundation models for generalist medical artificial intelligence.  Nature, 616:259–265, 2023.
  * [25] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar.  Foundation models for generalist medical artificial intelligence.  Nature, 616(7956):259–265, 2023.
  * [26] Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec.  Med-flamingo: A multimodal medical few-shot learner.  July 2023.  arXiv:2307.15189.
  * [27] Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, et al.  Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations.  Scientific Data, 9(1):429, 2022.
  * [28] Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu.  Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography.  Scientific Data, 10(1):277, 2023.
  * [29] Hieu T Nguyen, Hieu H Pham, Nghia T Nguyen, Ha Q Nguyen, Thang Q Huynh, Minh Dao, and Van Vu.  Vindr-spinexr: A deep learning framework for spinal lesions detection and classification from radiographs.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, pages 291–301. Springer, 2021.
  * [30] Ngoc H Nguyen, Hieu H Pham, Thanh T Tran, Tuan NM Nguyen, and Ha Q Nguyen.  Vindr-pcxr: An open, large-scale chest radiograph dataset for interpretation of common thoracic diseases in children.  medRxiv, pages 2022–03, 2022.
  * [31] OpenAI.  Gpt-4 technical report.  ArXiv, abs/2303.08774, 2023.
  * [32] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [33] Maya Pavlova et al.  Covid-net cxr-2: An enhanced deep convolutional neural network design for detection of covid-19 cases from chest x-ray images.  Frontiers in Medicine, 9, 2022.
  * [34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [35] George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  Radiology: Artificial Intelligence, 1(1):e180041, 2019.
  * [36] Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar.  Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning.  Nature Biomedical Engineering, 6(12):1399–1406, 2022.
  * [37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [38] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al.  Towards generalist biomedical ai.  arXiv preprint arXiv:2307.14334, 2023.
  * [39] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In CVPR, pages 2097–2106, 2017.
  * [40] Chaoyi Wu, Xiaoman Zhang, Yanfeng Wang, Ya Zhang, and Weidi Xie.  K-diag: Knowledge-enhanced disease diagnosis in radiographic imaging.  MICCAI Workshop, 2023.
  * [41] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Pmc-llama: Further finetuning llama on medical papers.  arXiv preprint arXiv:2304.14454, 2023.
  * [42] Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al.  Evaluating progress in automatic chest x-ray radiology report generation.  medRxiv, pages 2022–08, 2022.
  * [43] Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Knowledge-enhanced pre-training for auto-diagnosis of chest radiology images.  Nature Communication, 2023.
  * [44] Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Pmc-vqa: Visual instruction tuning for medical visual question answering.  ArXiv, abs/2305.10415, 2023.
  * [45] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al.  Pytorch fsdp: experiences on scaling fully sharded data parallel.  arXiv preprint arXiv:2304.11277, 2023.
  * [46] Zhengyun Zhao, Qiao Jin, Fangyuan Chen, Tuorui Peng, and Sheng Yu.  Pmc-patients: A large-scale dataset of patient summaries and relations for benchmarking retrieval-based clinical decision support systems.  arXiv preprint arXiv:2202.13876, 2022.
  * [47] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal C4: An open, billion-scale corpus of images interleaved with text.  arXiv preprint arXiv:2304.06939, 2023.


$\left|{H}\right|_{\infty}$ is exponentially bounded. Thanks to the fact that
the Hermite form is triangular, we can immediately extract from $H\cdot x=0$
the existence of a cancelling relation (CR-2) for $f_{1}$ where $i^{*},j^{*}$
are polynomially bounded, the degree of $p_{i^{*},j^{*}}$ is polynomially
bounded, and the height of $\left|{p_{i^{*},j^{*}}}\right|_{\infty}$ is
exponentially bounded.

Moreover, consider the one-dimensional sections
$f(0,k),\dots,f(i^{*},k)\in\mathbb{Q}^{\mathbb{N}}$. By Lemma 2.2, they are
linrec of order $\leq m\cdot(i^{*}+3)$, degree $\leq d$, and height $\leq
h\cdot(i^{*})^{d}$, and thus there are associated matrices
$A_{0},\dots,A_{i^{*}}$ of the appropriate dimensions
$\leq(m\cdot(i^{*}+3))\times(m\cdot(i^{*}+3))$ with coefficients in
$\mathbb{Q}[k][\partial_{2};\sigma_{2}]$. The bounds from Corollary E.8 can be
applied to this case as well and we obtain for each $0\leq i\leq i^{*}$ a
cancelling relation (CR-1) $R_{i}$ with leading polynomial coefficient
$q_{i,\ell_{i}^{*}}(k)$ where $\ell_{i}^{*}$ is polynomially bounded, its
degree in $k$ is polynomially bounded, and the height
$\left|{q_{i,\ell_{i}^{*}}}\right|_{\infty}$ is exponentially bounded.

###  E.1 Extended example

We conclude this section with an extended example showing how to compute the
Hermite form of a skew polynomial matrix, thus illustrating the techniques of
Giesbrecht and Kim [26] leading to Theorem E.4. We apply the algorithm on our
running example. For $n\in\mathbb{N}$, denote with
$\mathbb{F}[\partial;\sigma]_{n}$ the semiring of skew polynomials of degree
at most $n$ with coefficients in the field $\mathbb{F}$. Let
$\phi_{n}:\mathbb{F}[\partial;\sigma]_{n}\to\mathbb{F}^{n+1}$ be the bijection
that associates to a skew polynomial of degree $\leq n$ the vector of its
coefficients, starting from the one of highest degree. For instance,

$\phi_{5}(5\cdot\partial^{3}+4\cdot\partial^{2}+7)=(0,0,5,4,0,7).$

The _$m$ -Sylvester matrix_ of a skew polynomial
$P\in\mathbb{F}[\partial;\sigma]_{n-m}$ of degree $\leq n-m$ is the matrix
$S_{n}^{m}(P)\in\mathbb{F}^{(m+1)\times(n+1)}$ defined by

$\displaystyle S_{n}^{m}(P)=\begin{pmatrix}\phi_{n}(\partial^{m}P)\\\
\phi_{n}(\partial^{m-1}P)\\\ \vdots\\\ \phi_{n}(\partial^{0}P)\end{pmatrix}.$
(35)

For example, for $P=5\cdot\partial^{3}+4\cdot\partial^{2}+7$ we have

$\displaystyle S_{5}^{2}(P)=\begin{pmatrix}\phi_{5}(\partial^{2}P)\\\
\phi_{5}(\partial^{1}P)\\\
\phi_{5}(\partial^{0}P)\end{pmatrix}=\begin{pmatrix}5&4&0&7&0&0\\\
0&5&4&0&7&0\\\ 0&0&5&4&0&7\end{pmatrix}.$

The next lemma shows that sufficiently large Sylvester matrices can be used to
express product of polynomials in terms of products of matrices. This crucial
idea allows one to transform problems on skew polynomials in
$\mathbb{F}[\partial,\sigma]$ to linear algebra problems in the underlying
field (or just semiring) $\mathbb{F}$.

######  Lemma E.10 (c.f. [6, Sec. 1, eq. (1)]).

Let $P,Q\in\mathbb{F}[\partial;\sigma]$ and $n,m\in\mathbb{N}$ s.t. $\deg
P\leq m$ and $\deg Q\leq n-\deg P$. Then,

$\displaystyle\phi_{n}(Q\cdot P)=\phi_{m}(Q)\cdot S_{n}^{m}(P).$

We extend both $\phi_{n}$ and $S_{n}^{m}$ to skew polynomial matrices in
$\mathbb{F}[\partial;\sigma]_{n}^{k\times k}$ by point-wise application and
then merging all the obtained matrices into a single one.

######  Example E.11.

For instance, $\phi_{2}(A)$ with
$A\in\mathbb{Q}[k][\partial_{1};\sigma_{1}][\partial_{2};\sigma_{2}]^{2\times
2}$ from (28) equals

$\displaystyle\phi_{2}(A)$
$\displaystyle=\phi_{2}\begin{pmatrix}(\partial_{1}-1)\partial_{2}&-\partial_{2}\\\
-k\partial_{2}-1&\partial_{1}\partial_{2}\end{pmatrix}=\begin{pmatrix}\phi_{2}((\partial_{1}-1)\partial_{2})&\phi_{2}(-\partial_{2})\\\
\phi_{2}(-k\partial_{2}-1)&\phi_{2}(\partial_{1}\partial_{2})\end{pmatrix}$
$\displaystyle=\begin{pmatrix}0&\partial_{1}-1&0&0&{-1}&0\\\
0&{-k}&{-1}&0&\partial_{1}&0\end{pmatrix}\in\mathbb{Q}[k][\partial_{1};\sigma_{1}]^{2\times
6}$

and thus $S_{2}^{1}(A)\in\mathbb{Q}[k][\partial_{1};\sigma_{1}]^{4\times 6}$
is

$\displaystyle S_{2}^{1}(A)$
$\displaystyle=S_{2}^{1}\begin{pmatrix}(\partial_{1}-1)\partial_{2}&-\partial_{2}\\\
-k\partial_{2}-1&\partial_{1}\partial_{2}\end{pmatrix}=$
$\displaystyle=\begin{pmatrix}S_{2}^{1}((\partial_{1}-1)\partial_{2})&S_{2}^{1}(-\partial_{2})\\\
S_{2}^{1}(-k\partial_{2}-1)&S_{2}^{1}(\partial_{1}\partial_{2})\end{pmatrix}=$
$\displaystyle=\begin{pmatrix}\begin{pmatrix}\phi_{2}(\partial_{2}(\partial_{1}-1)\partial_{2})\\\
\phi_{2}((\partial_{1}-1)\partial_{2})\end{pmatrix}&\begin{pmatrix}\phi_{2}(\partial_{2}(-\partial_{2}))\\\
\phi_{2}(-\partial_{2})\end{pmatrix}\\\
\begin{pmatrix}\phi_{2}(\partial_{2}(-k\partial_{2}-1))\\\
\phi_{2}(-k\partial_{2}-1)\end{pmatrix}&\begin{pmatrix}\phi_{2}(\partial_{2}\partial_{1}\partial_{2})\\\
\phi_{2}(\partial_{1}\partial_{2})\end{pmatrix}\end{pmatrix}=$
$\displaystyle=\begin{pmatrix}\begin{pmatrix}(\partial_{1}-1\ 0\ 0)\\\ (0\
\partial_{1}-1\ 0)\end{pmatrix}&\begin{pmatrix}({-1}\ 0\ 0)\\\ (0\ {-1}\
0)\end{pmatrix}\\\ \begin{pmatrix}({-(k+1)}\ {-1}\ 0)\\\ (0\ {-k}\
{-1})\end{pmatrix}&\begin{pmatrix}(\partial_{1}\ 0\ 0)\\\ (0\ \partial_{1}\
0)\end{pmatrix}\end{pmatrix}=$
$\displaystyle=\begin{pmatrix}\partial_{1}-1&0&0&{-1}&0&0\\\
0&\partial_{1}-1&0&0&{-1}&0\\\ {-(k+1)}&{-1}&0&\partial_{1}&0&0\\\
0&{-k}&{-1}&0&\partial_{1}&0\end{pmatrix}.$

By definition of the Hermite form, we have that $H=U\cdot A$. By (30) every
degree of skew polynomials appearing therein is bounded by $n\cdot\deg A$.
Hence setting $\rho=n\cdot\deg A$, we have the following matrix equation with
coefficients in $\mathbb{F}$:

$\displaystyle\phi_{\rho+d}(H)=\phi_{\rho}(U)\cdot S^{\rho}_{\rho+d}(A).$

The _diagonal degree vector_ of the Hermite form for $A$ is the unique vector
$d$ s.t. $d_{i}=\deg H_{i,i}$. The algorithm will guess such a vector, and it
can detect whether the guess was correct or not. If it is the right one, then
$H$ and $U$ can be computed.

######  Example E.12.

The correct diagonal degree vector for our running example is $(0,2)$. The
Hermite normal form $H=U\cdot A$ of the $2\times 2$ matrix $A$ from our
running example has the form

$\displaystyle H=\begin{pmatrix}H_{11}&H_{12}\\\ 0&H_{22}\end{pmatrix},\quad
U=\begin{pmatrix}U_{11}&U_{12}\\\
U_{21}&U_{22}\end{pmatrix}\in\mathbb{Q}[k][\partial_{1};\sigma_{1}][\partial_{2};\sigma_{2}]^{2\times
2}$

where
$H_{11},H_{22}\in\mathbb{Q}[k][\partial_{1};\sigma_{1}][\partial_{2};\sigma_{2}]$
are _monic_ skew polynomials of degree respectively $0$ and $2$ and
$H_{12},U_{11},U_{12},U_{21},U_{22}\in\mathbb{Q}[k][\partial_{1};\sigma_{1}][\partial_{2};\sigma_{2}]$
are skew polynomials of degree $1$. It follows that

$\displaystyle\phi_{2}(H)$
$\displaystyle=\begin{pmatrix}\phi_{2}(H_{11})&\phi_{2}(H_{12})\\\
0&\phi_{2}(H_{22})\end{pmatrix}=$
$\displaystyle=\begin{pmatrix}\phi_{2}(1)&\phi_{2}(a_{121}\partial_{2}+a_{120})\\\
0&\phi_{2}(\partial_{2}^{2}+a_{221}\partial_{2}+a_{220})\end{pmatrix}=$
$\displaystyle=\begin{pmatrix}0&0&1&0&a_{121}&a_{120}\\\
0&0&0&1&a_{221}&a_{220}\end{pmatrix}\in\mathbb{Q}[k][\partial_{1};\sigma_{1}]^{2\times
6}.$

Similarly,

$\displaystyle\phi_{1}(U)$
$\displaystyle=\begin{pmatrix}\phi_{1}(U_{11})&\phi_{1}(U_{12})\\\
\phi_{1}(U_{21})&\phi_{1}(U_{22})\end{pmatrix}=\begin{pmatrix}\phi_{1}(u_{111}\partial_{2}+u_{110})&\phi_{1}(u_{121}\partial_{2}+u_{120})\\\
\phi_{1}(u_{211}\partial_{2}+u_{210})&\phi_{1}(u_{221}\partial_{2}+u_{220})\end{pmatrix}=$
$\displaystyle=\begin{pmatrix}u_{111}&u_{110}&u_{121}&u_{120}\\\
u_{211}&u_{210}&u_{221}&u_{220}\end{pmatrix}\in\mathbb{Q}[k][\partial_{1};\sigma_{1}]^{2\times
4}.$

By putting the pieces together, we obtain the following matrix equation with
entries in $\mathbb{Q}[k][\partial_{1};\sigma_{1}]$

$\displaystyle\underbrace{\begin{pmatrix}0&0&1&0&a_{121}&a_{120}\\\
0&0&0&1&a_{221}&a_{220}\end{pmatrix}}_{\phi_{2}(H)}=$
$\displaystyle\qquad\underbrace{\begin{pmatrix}u_{111}&u_{110}&u_{121}&u_{120}\\\
u_{211}&u_{210}&u_{221}&u_{220}\end{pmatrix}}_{\phi_{1}(U)}\cdot\underbrace{\begin{pmatrix}\partial_{1}-1&0&0&{-1}&0&0\\\
0&\partial_{1}-1&0&0&{-1}&0\\\ {-(k+1)}&{-1}&0&\partial_{1}&0&0\\\
0&{-k}&{-1}&0&\partial_{1}&0\end{pmatrix}}_{S^{1}_{2}(A)}.$

It is shown in [26, Theorem 5.2] that if we guessed the diagonal degree vector
right, then we can remove columns from $\phi_{\rho+d}(H)$ corresponding to
under-determined entries, and corresponding columns in $S^{\rho}_{\rho+d}(A)$,
in order to obtain two matrices $\tilde{A}$ and $\tilde{H}$ such that:

  * •

$\tilde{H}$ is only made of $0$’s and $1$’s.

  * •

$\tilde{A}$ is a square matrix.

  * •

The matrix equation $T\tilde{A}=\tilde{H}$ of unknown $T$ (of the same
dimensions as $\phi_{\rho}(U)$) has a unique solution. In particular,
$\tilde{A}$ has full rank and hence is invertible.

######  Example E.13.

The reduced system $\tilde{H}=\phi_{1}(U)\cdot\tilde{A}$ in our running
example is obtained by removing columns $5,6$ from $\phi_{2}(H)$ and
correspondingly from $S_{2}^{1}(A)$:

$\displaystyle\underbrace{\begin{pmatrix}0&0&1&0\\\
0&0&0&1\end{pmatrix}}_{\tilde{H}}=\underbrace{\begin{pmatrix}u_{111}&u_{110}&u_{121}&u_{120}\\\
u_{211}&u_{210}&u_{221}&u_{220}\end{pmatrix}}_{\phi_{1}(U)}\cdot\underbrace{\begin{pmatrix}\partial_{1}-1&0&0&-1\\\
0&\partial_{1}-1&0&0\\\ -(k+1)&-1&0&\partial_{1}\\\
0&-k&-1&0\end{pmatrix}}_{\tilde{A}}.$

Now the obtained $\tilde{A}$ is invertible. Hence we can determine $U$ thanks
to the equation $\phi_{1}(U)=\tilde{H}\tilde{A}^{-1}$.

######  Example E.14.

In the example, we obtain

$\displaystyle T=\begin{pmatrix}-\frac{k}{\partial_{1}-1}&-1\\\
\frac{k+1}{\partial_{1}^{2}-\partial_{1}-(k+1)}\partial_{2}+\frac{1}{\partial_{1}^{2}-\partial_{1}-(k+1)}&\frac{1}{\partial_{1}^{2}-\partial_{1}-(k+1)}\partial_{2}\end{pmatrix},$

yielding the Hermite form:

$\displaystyle H=T\cdot A$
$\displaystyle=\begin{pmatrix}-\frac{k}{\partial_{1}-1}&-1\\\
\frac{k+1}{\partial_{1}^{2}-\partial_{1}-(k+1)}\partial_{2}+\frac{1}{\partial_{1}^{2}-\partial_{1}-(k+1)}&\frac{1}{\partial_{1}^{2}-\partial_{1}-(k+1)}\partial_{2}\end{pmatrix}\cdot\begin{pmatrix}(\partial_{1}-1)\partial_{2}&-\partial_{2}\\\
-k\partial_{2}-1&\partial_{1}\partial_{2}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}1&(\frac{k}{\partial_{1}-1}-\partial_{1})\partial_{2}\\\
0&\partial_{2}^{2}-\frac{1}{\partial_{1}^{2}-\partial_{1}-(k+1)}\partial_{2}\end{pmatrix}.$
(36)