# Towards Generalist Biomedical AI

Tao Tu  Google Research,  Shekoofeh Azizi  Google DeepMind
Danny Driess  Google DeepMind  Mike Schaekermann  Google Research,  Mohamed
Amin  Google Research,  Pi-Chuan Chang  Google Research,  Andrew Carroll
Google Research,
Chuck Lau  Google Research,  Ryutaro Tanno  Google DeepMind  Ira Ktena  Google
DeepMind  Basil Mustafa  Google DeepMind  Aakanksha Chowdhery  Google DeepMind
Yun Liu  Google Research,
Simon Kornblith  Google DeepMind  David Fleet  Google DeepMind  Philip
Mansfield  Google Research,  Sushant Prakash  Google Research,  Renee Wong
Google Research,  Sunny Virmani  Google Research,  Christopher Semturs  Google
Research,  S Sara Mahdavi  Google DeepMind  Bradley Green  Google Research,
Ewa Dominowska  Google Research,  Blaise Aguera y Arcas  Google Research,
Joelle Barral  Google DeepMind  Dale Webster  Google Research,  Greg S.
Corrado  Google Research,  Yossi Matias  Google Research,  Karan Singhal
Google Research,  Pete Florence  Google DeepMind
Alan Karthikesalingam  Google Research,  Vivek Natarajan  Google Research,

###### Abstract

Medicine is inherently multimodal, with rich data modalities spanning text,
imaging, genomics, and more. Generalist biomedical artificial intelligence
(AI) systems that flexibly encode, integrate, and interpret this data at scale
can potentially enable impactful applications ranging from scientific
discovery to care delivery. To enable the development of these models, we
first curate MultiMedBench, a new multimodal biomedical benchmark.
MultiMedBench encompasses 14 diverse tasks such as medical question answering,
mammography and dermatology image interpretation, radiology report generation
and summarization, and genomic variant calling. We then introduce Med-PaLM
Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI
system. Med-PaLM M is a large multimodal generative model that flexibly
encodes and interprets biomedical data including clinical language, imaging,
and genomics with the same set of model weights. Med-PaLM M reaches
performance competitive with or exceeding the state of the art on all
MultiMedBench tasks, often surpassing specialist models by a wide margin. We
also report examples of zero-shot generalization to novel medical concepts and
tasks, positive transfer learning across tasks, and emergent zero-shot medical
reasoning. To further probe the capabilities and limitations of Med-PaLM M, we
conduct a radiologist evaluation of model-generated (and human) chest X-ray
reports and observe encouraging performance across model scales. In a side-by-
side ranking on 246 retrospective chest X-rays, clinicians express a pairwise
preference for Med-PaLM M reports over those produced by radiologists in up to
40.50% of cases, suggesting potential clinical utility. While considerable
work is needed to validate these models in real-world use cases, our results
represent a milestone towards the development of generalist biomedical AI
systems.

##  1 Introduction

Medicine is a multimodal discipline. Clinicians routinely interpret data from
a wide range of modalities including clinical notes, laboratory tests, vital
signs and observations, medical images, genomics, and more when providing
care.

Despite significant progress in biomedical AI, most models today are unimodal
single task systems [1, 2, 3]. Consider an existing AI system for interpreting
mammograms [4]. Although the system obtains state-of-the-art (SOTA)
performance on breast cancer screening, it cannot incorporate relevant
information such as patient health records (e.g., breast cancer gene screening
status), other modalities such as MRI, or published medical literature that
might help contextualize, refine, and improve performance. Further, the
system’s output is constrained to a pre-specified set of possible
classifications. It cannot verbally explain its prediction or engage in a
collaborative dialogue to learn from a physician’s feedback. This bounds
performance and utility of these narrow, single-task, unimodal, specialist AI
systems in real-world applications.

Figure 1: Med-PaLM M overview. A generalist biomedical AI system should be
able to handle a diverse range of biomedical data modalities and tasks. To
enable progress towards this overarching goal, we curate MultiMedBench, a
benchmark spanning 14 diverse biomedical tasks including question answering,
visual question answering, image classification, radiology report generation
and summarization, and genomic variant calling. Med-PaLM Multimodal (Med-PaLM
M), our proof of concept for such a generalist biomedical AI system (denoted
by the shaded blue area) is competitive with or exceeds prior SOTA results
from specialists models (denoted by dotted red lines) on all tasks in
MultiMedBench. Notably, Med-PaLM M achieves this using a single set of model
weights, without any task-specific customization.

The emergence of foundation models [5] offers an opportunity to rethink the
development of medical AI systems. These models are often trained on large-
scale data with self-supervised or unsupervised objectives and can be rapidly
and effectively adapted to many downstream tasks and settings using in-context
learning or few-shot finetuning [6, 7]. Further, they often have impressive
generative capabilities that can enable effective human-AI interaction and
collaboration. These advances enable the possibility of building a unified
biomedical AI system that can interpret multimodal data with complex
structures to tackle many challenging tasks. As the pace of biomedical data
generation and innovation increases, so will the potential impact of such
models, with a breadth of possible downstream applications spanning
fundamental biomedical discovery to care delivery.

In this work, we detail our progress towards such a generalist biomedical AI
system - a unified model that can interpret multiple biomedical data
modalities and handle many downstream tasks with the same set of model
weights. One of the key challenges of this goal has been the absence of
comprehensive multimodal medical benchmarks. To address this unmet need, we
curate MultiMedBench, an open source multimodal medical benchmark spanning
language, medical imaging, and genomics modalities with 14 diverse biomedical
tasks including question answering, visual question answering, medical image
classification, radiology report generation and summarization, and genomic
variant calling.

We leverage MultiMedBench to design and develop Med-PaLM Multimodal (Med-PaLM
M), a large-scale generalist biomedical AI system building on the recent
advances in language [8, 9] and multimodal foundation models [10, 11]. In
particular, Med-PaLM M is a flexible multimodal sequence-to-sequence
architecture that can easily incorporate and interleave various types of
multimodal biomedical information. Further, the expressiveness of the
modality-agnostic language decoder enables the handling of various biomedical
tasks in a simple generative framework with a unified training strategy.

To the best of our knowledge, Med-PaLM M is the first demonstration of a
generalist biomedical AI system that can interpret multimodal biomedical data
and handle a diverse range of tasks with a single model. Med-PaLM M reaches
performance competitive with or exceeding the state-of-the-art (SOTA) on all
tasks in MultiMedBench, often surpassing specialized domain and task-specific
models by a large margin. In particular, Med-PaLM M exceeds prior state-of-
the-art on chest X-ray (CXR) report generation (MIMIC-CXR dataset) by over 8%
on the common success metric (micro-F1) for clinical efficacy. On one of the
medical visual question answering tasks (Slake-VQA [12]) in MultiMedBench,
Med-PaLM M outperforms the prior SOTA results by over 10% on the BLEU-1 and F1
metrics.

We perform ablation studies to understand the importance of scale in our
generalist multimodal biomedical models and observe significant benefits for
tasks that require higher-level language capabilities, such as medical
(visual) question answering. Preliminary experiments also suggest evidence of
zero-shot generalization to novel medical concepts and tasks across model
scales, and emergent capabilities [13] such as zero-shot multimodal medical
reasoning. We further perform radiologist evaluation of AI-generated chest
X-ray reports and observe encouraging results across model scales.

Overall, these results demonstrate the potential of generalist biomedical AI
systems for medicine. However, significant work remains in terms of large-
scale biomedical data access for training such models, validating performance
in real world applications, and understanding the safety implications. We
outline these key limitations and directions of future research in our study.
To summarize, our key contributions are as follows:

  * •

Curation of MultiMedBench We introduce MultiMedBench, a new multimodal
biomedical benchmark spanning multiple modalities including medical imaging,
clinical text and genomics with 14 diverse tasks for training and evaluating
generalist biomedical AI systems.

  * •

Med-PaLM M, the first demonstration of a generalist biomedical AI system We
introduce Med-PaLM M, a single multitask, multimodal biomedical AI system that
can perform medical image classification, medical question answering, visual
question answering, radiology report generation and summarization, genomic
variant calling, and more with the same set of model weights. Med-PaLM M
reaches performance competitive with or exceeding state-of-the-art (SOTA)
specialist models on multiple tasks in MultiMedBench without any task-specific
customization.

  * •

Evidence of novel emergent capabilities in Med-PaLM M Beyond quantitative
evaluations of task performance, we observe evidence of zero-shot medical
reasoning, generalization to novel medical concepts and tasks, and positive
transfer across tasks. These experiments suggest promising potential of such
systems in downstream data-scarce biomedical applications.

  * •

Human evaluation of Med-PaLM M outputs Beyond automated metrics, we perform
radiologist evaluation of chest X-ray reports generated by Med-PaLM M across
different model scales. In a blinded side-by-side ranking on 246 retrospective
chest X-rays, clinicians expressed a pairwise preference for Med-PaLM M
reports over those produced by radiologists in up to 40.50% of cases.
Furthermore, the best Med-PaLM M model has on average 0.25 clinically
significant errors per report. These results are on par with human baselines
from prior work [14], suggesting potential clinical utility.

##  2 Related Work

###  2.1 Foundation models, multimodality, and generalists

The emergence of the foundation model paradigm [5] has had widespread impact
across a variety of applications in language [8], vision [15], and other
modalities [16]. While the idea of transfer learning [17, 18] using the
weights of pretrained models has existed for decades [19, 20, 21, 22], a shift
has come about due to the scale of data and compute used for pretraining such
models [23]. The notion of a foundation model further indicates that the model
can be adapted to a wide range of downstream tasks [5].

Within the foundation model paradigm, multimodality [24] has also had a
variety of important impacts – in the datasets [25], in the inter-modality
supervision [26], and in the generality and unification of task specification
[27, 28]. For example, language has specifically been an important enabler of
foundation models in other modalities [11, 29]. Visual foundation models such
as CLIP [30] are made possible by training on language-labeled visual datasets
[31, 25], which are easier to collect from large-scale internet data than
classification datasets with pre-determined class labels (i.e., ImageNet
[32]). The benefits of joint language-and-vision supervision has also been
noteworthy in generative modeling of images [33], where text-to-image
generative modeling has been notably more successful at producing high-
fidelity image generation [34] than purely unconditioned generative image
modeling [35]. Further, the flexibility of language also enables a wide range
of task specifications all via one unified output space [36] – it is possible
to phrase tasks traditionally addressed by different output spaces, such as
object detection and object classification, all jointly via the output space
of language [37]. Med-PaLM M additionally benefits from the generality of
multimodality, both via a model [10] pretrained on large vision-language
datasets [11], and also by further biomedical domain finetuning through a
unified generative language output space.

A related notion to that of a foundation model is that of a generalist model –
the same model with the same set of weights, without finetuning, can excel at
a wide variety of tasks. A single multitask [17] model which can address many
tasks has been of long standing interest [38, 39], including for example in
the reinforcement learning community [40]. Language-only models such as GPT-3
[6] and PaLM [8] simultaneously excel at many tasks using only prompting and
in-context learning. Recent work has also explored generalist models capable
not only of performing many tasks, but also of processing many modalities
[41]. For example, the capabilities of Gato [42] span language, vision, and
agent policy learning. PaLM-E [10] further shows that it is possible to obtain
a single generalist model which excels at language-only tasks, vision-language
tasks, and embodied vision-language tasks. Med-PaLM M is specifically a
generalist model designed for the biomedical domain, built by finetuning and
aligning the PaLM-E generalist model.

###  2.2 Multimodal foundation models in biomedicine

Given the potential, there has been significant interest in multimodal
foundation models for different biomedical applications. [43] discuss the
notion of generalist medical AI, albeit without implementation or empirical
results. [44] introduce Geneformer, a transformer [45] based model pretrained
on a corpus of about 30 million single-cell transcriptomes to enable context-
specific predictions in low data network biology applications. BiomedGPT [46]
is a multi-task biomedical foundation model pretrained on a diverse source of
medical images, medical literature, and clinical notes using a combination of
language model (LM) and masked image infilling objectives. However, all these
efforts are pretrained models and as such they require further task-specific
data and finetuning to enable downstream applications. In contrast, Med-PaLM M
is directly trained to jointly solve many biomedical tasks at the same time
without requiring any further finetuning or model parameter updates. LLaVA-Med
[47] is perhaps most similar to our effort. The authors use PubMed and GPT-4
[48] to curate a multimodal instruction following dataset and finetune a LLaVA
model with it. However, the experiments are limited to three medical visual
question answering datasets and qualitative examples of conversations
conditioned on a medical image. In contrast, our work is more comprehensive,
spanning multiple modalities including medical imaging, clinical text, and
genomics with 14 diverse tasks and expert evaluation of model outputs.

###  2.3 Multimodal medical AI benchmarks

To the best of our knowledge, there have been limited attempts to curate
benchmarks for training and evaluating generalist biomedical AI models.
Perhaps the work closest in spirit is BenchMD [49]. The benchmark spans 19
publicly available datasets and 7 medical modalities, including 1D sensor
data, 2D images, and 3D volumetric scans. However, their tasks are primarily
focused on classification whereas our benchmark also includes generative tasks
such as medical (visual) question answering, radiology report generation and
summarization. Furthermore, there is currently no implementation of a
generalist biomedical AI system that can competently handle all these tasks
simultaneously.

##  3 MultiMedBench: A Benchmark for Generalist Biomedical AI

We next describe MultiMedBench, a benchmark we curated to enable the
development and evaluation of generalist biomedical AI. MultiMedBench is a
multi-task, multimodal benchmark comprising 12 de-identified open source
datasets and 14 individual tasks. It measures the capability of a general-
purpose biomedical AI to perform a variety of clinically-relevant tasks. The
benchmark covers a wide range of data sources including medical questions,
radiology reports, pathology, dermatology, chest X-ray, mammography, and
genomics. Tasks in MultiMedBench vary across the following axes:

  * •

Task type: question answering, report generation and summarization, visual
question answering, medical image classification, and genomic variant calling.

  * •

Modality: text, radiology (CT, MRI, and X-ray), pathology, dermatology,
mammography, and genomics.

  * •

Output format: open-ended generation for all tasks including classification.

Table 1: MultiMedBench overview. Summary of MultiMedBench, the benchmark we introduce for the development and evaluation of Med-PaLM M. MultiMedBench consists of 14 individual tasks across 5 task types and 12 datasets spanning 7 biomedical data modalities. In total, the benchmark contains over 1 million samples. Task Type | Modality | Dataset | Description
---|---|---|---
Question Answering | Text | MedQA |  | US medical licensing exam-style, multiple-choice
---
MedMCQA |  | Indian medical entrance exams, multiple-choice
---
PubMedQA |  | Biomedical literature questions, multiple-choice
---
Report Summarization | Radiology | MIMIC-III |  | Summarizing findings in radiology reports
---
Visual Question Answering | Radiology | VQA-RAD |  | Close/open-ended VQA on radiology images
---
Slake-VQA |  | English-Chinese bilingual VQA on radiology images
---
Pathology | Path-VQA |  | Close/open-ended VQA on pathology images
---
Report Generation | Chest X-ray | MIMIC-CXR |  | Chest X-ray report generation
---
Medical Image Classification | Chest X-ray | MIMIC-CXR |  | Binary classification of chest X-ray abnormalities
---
Dermatology | PAD-UFES-20 |  | 6-class skin lesion image classification
---
Mammography | VinDr-Mammo |  | 5-class breast-level BI-RADS classification
---
CBIS-DDSM |  | 3-class lesion-level classification (mass)
---
CBIS-DDSM |  | 3-class lesion-level classification (calcification)
---
| Genomics
---
| PrecisionFDA
---
Truth Challenge V2
| Genomic variant calling as 3-class image classification
---

Language-only tasks consist of medical question answering, including three of
the MultiMedQA tasks used in [9], and radiology report summarization. They
were selected to assess a model’s ability to comprehend, recall, and
manipulate medical knowledge. Multimodal tasks include medical visual question
answering (VQA), medical image classification, chest X-ray report generation,
and genomic variant calling, which are well-suited to evaluate both the visual
understanding and multimodal reasoning capabilities of these models. Table 1
includes an overview of the datasets and tasks in MultiMedBench - in total,
the benchmark contains over 1 million samples. For detailed descriptions of
individual datasets and tasks, see Section A.1.

##  4 Med-PaLM M: A Proof of Concept for Generalist Biomedical AI

In this section, we detail the methods underpinning the development of the
Med-PaLM M model. We first review preliminaries of the pretrained models in
Section 4.1 from which Med-PaLM M inherits, then discuss the datasets and
training details involved in the finetuning and specialization of the model to
the biomedical domain Section 4.2.

###  4.1 Model preliminaries

Note that Med-PaLM M inherits not only the architectures of these pretrained
models, but also the general domain knowledge encoded in their model
parameters.

##### Pathways Language Model (PaLM)

introduced by [8] is a densely-connected decoder-only Transformer [45] based
large language model (LLM) trained using Pathways [50], a large-scale ML
accelerator orchestration system that enables highly efficient training across
TPU pods. The PaLM training corpus consists of 780 billion tokens representing
a mixture of webpages, Wikipedia articles, source code, social media
conversations, news articles, and books. PaLM models were trained at sizes of
8, 62, and 540 billion parameters, and all three PaLM model variants are
trained for one epoch of the training data. At the time of its announcement,
PaLM 540B achieved breakthrough performance, outperforming finetuned state-of-
the-art models on a suite of multi-step reasoning tasks and exceeding average
human performance on BIG-bench [51].

##### Vision Transformer (ViT)

introduced by [52] extends the Transformer [45] architecture to visual data
such as images and videos. In this work, we consider two ViT pre-trained
models as vision encoders, the 4 billion (4B) parameters model from [11] and
the 22 billion (22B) parameters model from [15]. Both of these models were
pretrained via supervised learning on a large classification dataset [53, 54]
of approximately 4 billion images.

##### PaLM-E

introduced by [10] is a multimodal language model that can process sequences
of multimodal inputs including text, vision, and sensor signals. The primary
PaLM-E model uses pretrained PaLM and ViT, and was initially developed for
embodied robotics applications but demonstrated strong performance on multiple
vision language benchmarks such as OK-VQA [55] and VQA v2 [56]. Furthermore,
PaLM-E offers the flexibility to interleave images, text and sensor signals in
a single prompt, enabling the model to make predictions with a fully
multimodal context. PaLM-E also exhibits a wide array of capabilities
including zero-shot multimodal chain-of-thought (CoT) reasoning, and few-shot
in-context learning. We therefore leverage the PaLM-E model as the base
architecture for Med-PaLM M.

We consider three different combinations of LLM and vision encoders in our
study - PaLM 8B with ViT 4B (PaLM-E 12B), PaLM 62B with ViT 22B (PaLM-E 84B)
and PaLM 540B with ViT 22B (PaLM-E 562B). All models were pretrained on
diverse vision-language datasets in addition to tasks across multiple robot
embodiments as described in [10].

###  4.2 Putting it all together: Med-PaLM M

Med-PaLM M is developed by finetuning and aligning the PaLM-E model to the
biomedical domain using MultiMedBench. The following summarizes important
methodological details underlying the development of the model.

##### Dataset and preprocessing

We resized all the images in MultiMedBench to $224\times 224\times 3$, while
preserving the original aspect ratio with padding if needed. The gray-scale
images were converted to 3-channel images by stacking up the same image along
the channel dimension. Task-specific prepossessing methods such as class
balancing and image data augmentation are described in detail for each task in
Section A.1.

##### Instruction task prompting and one-shot exemplar

Our goal is to train a generalist biomedical AI model to perform multiple
tasks with multimodal inputs using a unified model architecture and a single
set of model parameters. To this end, we trained the model with a mixture of
distinct tasks simultaneously via instruction tuning [57]. Specifically, we
provided the model with task-specific instructions to prompt the model to
perform different types of tasks in a unified generative framework. The task
prompt consists of an instruction, relevant context information, and a
question. For example, as shown in Figure 2, in the chest X-ray report
generation task, we included the reason for the study and the image
orientation information as additional context information for the model to
condition its prediction on. Similarly, for the dermatology classification
task, we provided the patient clinical history associated with the skin lesion
image. We formulated all classification tasks as multiple choice questions
where all possible class labels are provided as individual answer options and
the model was prompted to generate the most likely answer as the target
output. For other generative tasks such as visual question answering and
report generation and summarization, the model was finetuned on the target
response.

In order to enable the model to better follow instructions, for the majority
of tasks (see Table A.1), we added a text-only “one-shot exemplar” to the task
prompt to condition the language model’s prediction. The one-shot exemplar
helps prompt the model with a partial input-output pair. Importantly, for
multimodal tasks, we replaced the actual image in the exemplar with a dummy
text placeholder (with the text string “<img>”): this (i) preserves training
compute efficiency for single-image training, and also (ii) bypasses potential
interference from cross-attention between a given text token and image tokens
from multiple images [28]. Our results show that this scheme is effective in
prompting the model to generate the desired format of responses as detailed in
Section 6.

##### Model training

We finetuned the pretrained 12B, 84B, and 562B parameter variants of PaLM-E on
MultiMedBench tasks with mixture ratios denoted in Table A.1. These mixture
ratios were empirically determined such that they are approximately
proportional to the number of training samples in each dataset and ensuring at
least one sample from each task is present in one batch. We performed an end-
to-end finetuning of the PaLM-E model with the entire set of model parameters
updated during training. For multimodal tasks, image tokens were interleaved
with text tokens to form multimodal context input to the PaLM-E model. The
multimodal context input contains at most 1 image for all finetuning tasks.
However, we note that Med-PaLM M is able to process inputs with multiple
images during inference.

We used the Adafactor optimizer [58] with momentum of $\beta_{1}=0.9$, dropout
rate of 0.1, and a constant learning rate schedule. We used different sets of
hyperparameters in our finetuning experiments for different model sizes, which
are further detailed in Table A.2.

The resulting model, Med-PaLM M (12B, 84B, and 562B), is adapted to the
biomedical domain with the capability to encode and interpret multimodal
inputs and perform tasks including medical (visual) question answering,
radiology report generation and summarization, medical image classification,
and genomic variant calling.

Figure 2: Illustration of instruction task prompting with one-shot exemplar.
(top) shows the task prompt for the chest X-ray report generation task. It
consists of task-specific instructions, a text-only “one-shot exemplar”
(omitting the corresponding image but preserving the target answer), and the
actual question. The X-ray image is embedded and interleaved with textual
context including view orientation and reason for the study in addition to the
question. (bottom) shows the task prompt for the dermatology classification
task. We formulate the skin lesion classification task as a multiple choice
question answering task with all the class labels provided as individual
answer options. Similar to the chest X-ray report generation task, skin lesion
image tokens are interleaved with the patient clinical history as additional
context to the question. The blue <img> denotes the position in the prompt
where the image tokens are embedded.

##  5 Evaluation

In this section, we describe the purpose, scope, and methods of experimental
evaluations. Results are presented in Section 6. Evaluation experiments of
Med-PaLM M were designed for the following purposes:

  * •

Evaluate generalist capabilities We evaluated Med-PaLM M on all tasks in
MultiMedBench across model scales. We provide initial insights on the effect
of scaling ViT and LLM components across different tasks. We compared
performance to previous SOTA (including specialist single-task or single-
modality methods) and a state-of-art generalist model (PaLM-E) without
biomedical finetuning.

  * •

Explore novel emergent capabilities One hypothesized benefit of training a
single flexible multimodal generalist AI system across diverse tasks is the
emergence of novel capabilities arising from language enabled combinatorial
generalization, such as to novel medical concepts and tasks. We explored this
via qualitative and qualitative experiments.

  * •

Measure radiology report generation quality Automatic natural language
generation (NLG) metrics do not provide sufficient evaluation of the clinical
applicability of AI-generated radiology reports. We therefore performed expert
radiologist evaluation of AI-generated reports on the MIMIC-CXR dataset,
including comparison to the radiologist-provided reference reports.

###  5.1 Evaluation on MultiMedBench

Med-PaLM M was simultaneously finetuned on a mixture of language-only and
multimodal biomedical tasks in MultiMedBench. We assessed the model’s in-
distribution performance on these tasks by comparing to the corresponding SOTA
results obtained from separate specialist models. Specifically, we used the
same few-shot setup as in training for each task during evaluation. Task-
specific metrics were computed on the test split of each task and compared to
prior SOTA specialist AI systems. Note that for a small number of tasks
described in Table 1, we were not able to find a sufficiently similar prior
attempt for comparison.

###  5.2 Evaluation of language enabled zero-shot generalization

To probe Med-PaLM M’s ability to generalize to previously unseen medical
concepts, we evaluate the model’s ability to predict the presence or absence
of tuberculosis (TB) from chest X-ray images. We used the Montgomery County
chest X-ray set (MC) for this purpose. The dataset contains 138 frontal chest
X-rays, of which 80 are normal cases and 58 cases have manifestations of TB
[59]. Each case also contains annotations on the abnormality seen in the lung.
We note that Med-PaLM M has been trained on MIMIC-CXR dataset; however, it is
not trained to explicitly predict the TB disease label.

We evaluated the accuracy across model scales by formulating this problem as a
two-choice question answering task where the model was prompted (with a text-
only one-shot exemplar) to generate a yes/no answer about the presence of TB
in the input image.

We further explored zero-shot chain-of-thought (CoT) multimodal medical
reasoning ability of the model by prompting with a text-only exemplar (without
the corresponding image) and prompting the model to generate the class
prediction and an accompanying report describing the image findings. We note
that while we did prompt the model with a single text-only input-output pair,
we omitted the image (used a dummy text placeholder instead) and the text
exemplar was hand-crafted rather than drawn from the training set. Hence, this
approach can be considered zero-shot rather than one-shot.

In order to assess Med-PaLM M’s ability to generalize to novel task scenarios,
we evaluated the model performance on two-view chest X-ray report generation -
this is a novel task given the model was trained to generate reports only from
a single-view chest X-ray.

Finally, we also probed for evidence of positive task transfer as a result of
jointly training a single generalist model to solve many different biomedical
tasks. To this end, we performed an ablation study where we trained a Med-PaLM
M 84B variant by excluding the MIMIC-CXR classification tasks from the task
mixture. We compared this model variant to the Med-PaLM M 84B variant trained
on the complete MultiMedBench mixture on the chest X-ray report generation
task with the expectation of improved performance in the latter.

###  5.3 Clinician evaluation of radiology report generation

To further assess the quality and clinical applicability of chest X-ray
reports generated by Med-PaLM M and understand the effect of model scaling, we
conducted a human evaluation using the MIMIC-CXR dataset. The evaluation was
performed by four qualified thoracic radiologists based in India.

##### Dataset

The evaluation set consisted of 246 cases selected from the MIMIC-CXR test
split. To match the expected input format of Med-PaLM M, we selected a single
image from each study. We excluded studies that had ground truth reports
mentioning multiple X-ray views or past examinations of the same patient.

##### Procedure

We conducted two complementary human evaluations: (1) side-by-side evaluation
where raters compared multiple alternative report findings and ranked them
based on their overall quality, and (2) independent evaluation where raters
assessed the quality of individual report findings. Prior to performing the
final evaluation, we iterated upon the instructions for the raters and
calibrated their grades using a pilot set of 25 cases that were distinct from
the evaluation set. Side-by-side evaluation was performed for all 246 cases,
where each case was rated by a single radiologist randomly selected from a
pool of four. For independent evaluation, each of the four radiologists
independently annotated findings generated by three Med-PaLM M model variants
(12B, 84B, and 562B) for every case in the evaluation set. Radiologists were
blind to the source of the report findings for all evaluation tasks, and the
reports were presented in a randomized order.

##### Side-by-side evaluation

The input to each side-by-side evaluation was a single chest X-ray, along with
the “indication” section from the MIMIC-CXR study. Four alternative options
for the “findings” section of the report were shown to raters as depicted in
Figure A.3. The four alternative “findings” sections corresponded to the
dataset reference report’s findings, and findings generated by three Med-PaLM
M model variants (12B, 84B, 562B). Raters were asked to rank the four
alternative findings based on their overall quality using their best clinical
judgement.

##### Independent evaluation

For independent evaluation, raters were also presented with a single chest
X-ray, along with the indication and reference report’s findings from the
MIMIC-CXR study (marked explicitly as such), but this time only a single
findings paragraph generated by Med-PaLM M as shown in Figure A.4. Raters were
asked to assess the quality of the Med-PaLM M generated findings in the
presence of the reference inputs provided and their own judgement of the chest
X-ray image. The rating schema proposed in [60] served as inspiration for our
evaluation task design.

First, raters assessed whether the quality and view of the provided image were
sufficient to perform the evaluation task fully. Next, they annotated all
passages in the model-generated findings that they disagreed with (errors),
and all missing parts (omissions). Raters categorized each error passage by
its type (no finding, incorrect finding location, incorrect severity,
reference to non-existent view or prior study), assessed its clinical
significance, and suggested alternative text to replace the selected passage.
Likewise, for each omission, raters specified a passage that should have been
included and determined if the omission had any clinical significance.

Table 2: Performance comparison on MultiMedBench. We compare Med-PaLM M with specialist SOTA models and a generalist model (PaLM-E 84B) without biomedical domain finetuning. Across all tasks, datasets and metrics combination in MultiMedBench, we observe Med-PaLM M performance near or exceeding SOTA. Note that these results are achieved by Med-PaLM M with the same set of model weights without any task-specific customization. Task Type | Modality | Dataset | Metric | SOTA |  | PaLM-E
---
(84B)
| Med-PaLM M
---
(Best)
Question Answering | Text | MedQA | Accuracy |  86.50% [61] | 28.83% | 69.68%
MedMCQA | Accuracy |  72.30% [61] | 33.35% | 62.59%
PubMedQA | Accuracy |  81.80% [61] | 64.00% | 80.00%
Report Summarization | Radiology | MIMIC-III | ROUGE-L |  38.70% [62] | 3.30% | 32.03%
BLEU |  16.20% [62] | 0.34% | 15.36%
F1-RadGraph |  40.80% [62] | 8.00% | 34.71%
Visual Question Answering | Radiology | VQA-RAD | BLEU-1 |  71.03% [63] | 59.19% | 71.27%
F1 | N/A | 38.67% | 62.06%
Slake-VQA | BLEU-1 |  78.60% [64] | 52.65% | 92.7%
F1 |  78.10% [64] | 24.53% | 89.28%
Pathology | Path-VQA | BLEU-1 |  70.30% [64] | 54.92% | 72.27%
F1 |  58.40% [64] | 29.68% | 62.69%
Report Generation | Chest X-ray | MIMIC-CXR | Micro-F1-14 |  44.20% [65] | 15.40% | 53.56%
Macro-F1-14 |  30.70% [65] | 10.11% | 39.83%
Micro-F1-5 |  56.70% [66] | 5.51% | 57.88%
Macro-F1-5 | N/A | 4.85% | 51.60%
F1-RadGraph |  24.40% [14] | 11.66% | 26.71%
BLEU-1 |  39.48% [65] | 19.86% | 32.31%
BLEU-4 |  13.30% [66] | 4.60% | 11.50%
ROUGE-L |  29.60% [67] | 16.53% | 27.49%
CIDEr-D |  49.50% [68] | 3.50% | 26.17%
Image Classification | Chest X-ray |  MIMIC-CXR (5 conditions) | Macro-AUC |  81.27% [69] | 51.48% | 79.09%
Macro-F1 | N/A | 7.83% | 41.57%
Dermatology | PAD-UFES-20 | Macro-AUC | N/A | 63.37% | 97.27%
Macro-F1 | N/A | 1.38% | 84.32%
Mammography | VinDr-Mammo | Macro-AUC |  64.50% [49] | 51.49% | 71.76%
Macro-F1 | N/A | 16.06% |  35.70%
CBIS-DDSM (mass) | Macro-AUC | N/A | 47.75% | 73.31%
Macro-F1 | N/A | 7.77% | 51.12%
CBIS-DDSM (calcification) | Macro-AUC | N/A | 40.67% | 82.22%
Macro-F1 |  70.71% [70] | 11.37% | 67.86%
Genomics (Variant Calling) |  PrecisionFDA (Truth Challenge V2) | Indel-F1 |  99.40% [71] | 53.01% | 97.04%
SNP-F1 |  99.70% [71] | 52.84% | 99.35%
Table 3: Performance of Med-PaLM M on MultiMedBench across model scales. We summarize the performance of Med-PaLM M across three model scale variants 12B, 84B, 562B. All models were finetuned and evaluated on the same set of tasks in MultiMedBench. We observe that scaling plays a key role in language-only tasks and multimodal tasks that require reasoning such as visual question answering. However, scaling has diminishing benefit for image classification and chest X-ray report generation task. Task Type | Modality | Dataset | Metric |  | Med-PaLM M
---
(12B)
| Med-PaLM M
---
(84B)
| Med-PaLM M
---
(562B)
Question Answering | Text | MedQA | Accuracy | 29.22% | 46.11% | 69.68%
MedMCQA | Accuracy | 32.20% | 47.60% | 62.59%
PubMedQA | Accuracy | 48.60% | 71.40% | 80.00%
Report Summarization | Radiology | MIMIC-III | ROUGE-L | 29.45% | 31.47% | 32.03%
BLEU | 12.14% |  15.36% | 15.21%
F1-RadGraph | 31.43% | 33.96% | 34.71%
Visual Question Answering | Radiology | VQA-RAD | BLEU-1 | 64.02% | 69.38% | 71.27%
F1 | 50.66% | 59.90% | 62.06%
Slake-VQA | BLEU-1 | 90.77% | 92.70% | 91.64%
F1 | 86.22% | 89.28% | 87.50%
Pathology | Path-VQA | BLEU-1 | 68.97% | 70.16% | 72.27%
F1 | 57.24% | 59.51% | 62.69%
Report Generation | Chest X-ray | MIMIC-CXR | Micro-F1-14 | 51.41% | 53.56% | 51.60%
Macro-F1-14 | 37.31% | 39.83% | 37.81%
Micro-F1-5 | 56.54% | 57.88% | 56.28%
Macro-F1-5 | 50.57% | 51.60% | 49.86%
F1-RadGraph | 25.20% | 26.71% | 26.06%
BLEU-1 | 30.90% | 32.31% | 31.73%
BLEU-4 | 10.43% | 11.31% | 11.50%
ROUGE-L | 26.16% | 27.29% | 27.49%
CIDEr-D | 23.43% | 26.17% | 25.27%
Image Classification | Chest X-ray |  MIMIC-CXR (5 conditions) | Macro-AUC | 76.67% | 78.35% | 79.09%
Macro-F1 | 38.33% | 36.83% | 41.57%
Dermatology | PAD-UFES-20 | Macro-AUC | 95.57% | 97.27% | 96.08%
Macro-F1 | 78.42% | 84.32% | 77.03%
Mammography | VinDr-Mammo | Macro-AUC | 66.29% | 71.76% | 71.42%
Macro-F1 | 29.81% |  35.70% | 33.90%
CBIS-DDSM (mass) | Macro-AUC | 70.11% | 73.09% | 73.31%
Macro-F1 | 47.23% | 49.98% | 51.12%
CBIS-DDSM (calcification) | Macro-AUC | 81.40% | 82.22% | 80.90%
Macro-F1 |  67.86% | 63.81% | 63.03%
Genomics | Variant Calling | Indel-F1 | 96.42% | 97.04% | 95.46%
SNP-F1 | 99.35% | 99.32% | 99.16%

##  6 Results

Here we present results across the three different evaluation setups
introduced in Section 5.

###  6.1 Med-PaLM M performs near or exceeding SOTA on all MultiMedBench tasks

##### Med-PaLM M performance versus baselines

We compared Med-PaLM M with two baselines:

  * •

prior SOTA specialist models for each of the MultiMedBench tasks

  * •

a baseline generalist model (PaLM-E 84B) without any biomedical domain
finetuning. We used this model size variant (and not PaLM-E 562B) due to
compute constraints.

Results are summarized in Table 2. Across MultiMedBench tasks, Med-PaLM M’s
best result (across three model sizes) exceeded prior SOTA results on 5 out of
12 tasks (for two tasks, we were unable to find a prior SOTA comparable to our
setup) while being competitive on the rest. Notably, these results were
achieved with a generalist model using the same set of model weights without
any task-specific architecture customization or optimization.

On medical question answering tasks, we compared against the SOTA Med-PaLM 2
results [61] and observed higher performance of Med-PaLM 2. However, when
compared to the baseline PaLM model on which Med-PaLM M was built, Med-PaLM M
outperformed the previous best PaLM results [9] by a large margin in the same
few-shot setting on all three question answering datasets.

Further, when compared to PaLM-E 84B as a generalist baseline without
biomedical domain finetuning, Med-PaLM M exhibited performance improvements on
all 14 tasks often by a significant margin, demonstrating the importance of
domain adaptation. Taken together, these results illustrate the strong
capabilities of Med-PaLM M as a generalist biomedical AI model. We further
describe the results in detail for each of the individual tasks in Section
A.3.

##### Med-PaLM M performance across model scales

We summarize Med-PaLM M performance across model scales (12B, 84B, and 562B)
in Table 3. The key observations are:

  * •

Language reasoning tasks benefit from scale For tasks that require language
understanding and reasoning such as medical question answering, medical visual
question answering and radiology report summarization, we see significant
improvements as we scale up the model from 12B to 562B.

  * •

Multimodal tasks bottlenecked by vision encoder performance For tasks such as
mammography or dermatology image classification, where nuanced visual
understanding is required but minimal language reasoning is needed (outputs
are classification label tokens only), the performance improved from Med-PaLM
M 12B to Med-PaLM 84B but plateaued for the 562B model, possibly because the
vision encoder is not further scaled in that step (both the Med-PaLM M 84B and
562B models use the same 22B ViT as the vision encoder), thereby acting as a
bottleneck to observing a scaling benefit. We note the possibility of
additional confounders here such as the input image resolution.

The scaling results on the chest X-ray report generation task are interesting
(Table 3). While on the surface, the task seems to require complex language
understanding and reasoning capabilities and would thus benefit from scaling
the language model, we find the Med-PaLM M 84B model to be roughly on-par or
slightly exceeding the 562B model on a majority of metrics, which may simply
be due to fewer training steps used for the larger model. Another possibility
for the diminishing return of increasing the size of language model is likely
that the output space for chest X-ray report generation in the MIMIC-CXR
dataset is fairly confined to a set of template sentences and limited number
of conditions. This insight has motivated the use of retrieval based
approaches as opposed to a fully generative approach for the chest X-ray
report generation task on this dataset [72, 73]. Additionally, the larger 562B
model has a tendency towards verbosity rather than the comparative brevity of
the 84B model, and without further preference alignment in training, this may
impact its metrics.

###  6.2 Med-PaLM M demonstrates zero-shot generalization to novel medical
tasks and concepts

Training a generalist biomedical AI system with language as a common grounding
across different tasks allows the system to tackle new tasks by combining the
knowledge it has learned for other tasks (i.e. combinatorial generalization).
We highlight preliminary evidence which suggests Med-PaLM M can generalize to
novel medical concepts and unseen tasks in a zero-shot fashion. We further
observe zero-shot multimodal reasoning as an emergent capability [13] of Med-
PaLM M. Finally, we demonstrate benefits from positive task transfer as a
result of the model’s multi-task, multimodal training.

####  6.2.1 Evidence of generalization to novel medical concepts

We probed the zero-shot generalization capability of Med-PaLM M for an unseen
medical concept by evaluating its ability to detect tuberculosis (TB)
abnormality from chest X-ray images in the Montgomery County (MC) dataset. As
shown in Table 4, Med-PaLM M performed competitively compared to SOTA results
obtained by a specialized ensemble model optimized for this dataset [74]. We
observed similar performance across three model variants, consistent with
findings on other medical image classification tasks in MultiMedBench. Given
the classification task was set up as an open-ended question answering task,
we did not report the AUC metric which requires the normalized predicted
probability of each possible class.

Table 4: Zero-shot classification performance of Med-PaLM M on the tuberculosis (TB) detection task. Med-PaLM M performs competitively to the SOTA model [74] finetuned on the Montgomery County TB dataset using model ensemble. Notably, Med-PaLM M achieves this result with a simple task prompt consisting of a single text-only exemplar (without task-specific image and hence zero-shot), in contrast to the specialist model that requires training on all the samples in the dataset. Model | # Training samples | Accuracy
---|---|---
SOTA [74] | 138 | 92.60%
Med-PaLM M (12B) | 0 | 86.96%
Med-PaLM M (84B) | 0 | 82.60%
Med-PaLM M (562B) | 0 | 87.68%

####  6.2.2 Evidence of emergent zero-shot multimodal medical reasoning

We also qualitatively explored the zero-shot chain-of-thought (CoT) capability
of Med-PaLM M on the MC TB dataset. In contrast to the classification setup,
we prompted the model with a text-only exemplar to generate a report
describing the findings in a given image in addition to a yes/no
classification prediction. In Figure 3, we present qualitative examples of
zero-shot CoT reasoning from the Med-PaLM M 84B and 562B variants. In
particular, both Med-PaLM M variants were able to identify the major TB
related lesion in the correct location. However, according to expert
radiologist review, there are still some omissions of findings and errors in
the model generated report, suggesting room for improvement. It is noteworthy
that Med-PaLM M 12B failed to generate a coherent visually conditioned
response, which indicates that scaling of the language model plays a key role
in the zero-shot CoT multimodal reasoning capability (i.e. this might be an
emergent capability [13]).

Figure 3: Evidence of emergent zero-shot multimodal medical reasoning with
Med-PaLM M. Large Med-PaLM M models exhibit zero-shot CoT reasoning capability
in identifying and describing tuberculosis related findings in chest X-ray
images. The model is prompted with task-specific instructions and a text-only
exemplar (without the corresponding image) to generate a report describing
findings in the given X-ray image. Model predictions from Med-PaLM M 84B and
562B are shown together with the annotations from an expert radiologist. Both
models correctly localized the major TB related cavitory lesion in the right
upper lobe. However, both models did not address the small cavitory lesion in
left upper lobe (Med-PaLM M 562B was considered better than Med-PaLM M 64B in
this example as it also alluded to the opacity in the right middle lobe and
did not make the incorrect statement of left lung being clear). Notably, Med-
PaLM M 12B failed to generate a coherent report, indicating the importance of
scaling for zero-shot COT reasoning.

####  6.2.3 Evidence of generalization to novel tasks

Although Med-PaLM M was only trained with single-view chest X-ray image
inputs, we observed the capability of the model to generalize to a novel task
setup with multi-view visual inputs. Specifically, on a subset of studies from
MIMIC-CXR where each report is accompanied with both a frontal and a lateral
view X-ray image. we observe that Med-PaLM M is able to attain zero-shot
performance comparable to the single-view report generation task as detailed
in Table 5. This ability is promising given medical imaging studies often
benefit from the interpretation of prior historical studies in addition to the
current instance for optimal performance.

Table 5: Zero-shot generalization to two-view chest X-ray report generation. Med-PaLM M performance remains competitive on a novel two-view report generation task setup despite having not been trained with two visual inputs before. Med-PaLM M achieves SOTA results on clinical efficacy metrics for the two view report generation task. Metric | SOTA | Med-PaLM M (12B) | Med-PaLM M (84B) | Med-PaLM M (562B)
---|---|---|---|---
Micro-F1-14 | 44.20% | 49.80% | 50.54% | 48.85%
Macro-F1-14 | 30.70% | 37.69% | 37.78% | 37.29%
Micro-F1-5 | 56.70% | 54.49% | 56.37% | 54.36%
Macro-F1-5 | N/A | 48.33% | 51.23% | 48.49%
F1-RadGraph | 24.40% | 26.73% | 28.30% | 27.28%
BLEU-1 | 39.48% | 33.31% | 34.58% | 33.83%
BLEU-4 | 13.30% | 11.51% | 12.44% | 12.47%
ROUGE-L | 29.60% | 27.84% | 28.71% | 28.49%
CIDEr-D | 49.50% | 27.58% | 29.80% | 29.80%

####  6.2.4 Evidence of positive task transfer

To demonstrate the positive task transfer arising from joint training across
modalities and tasks, we performed an ablation study where we trained a Med-
PaLM M 84B variant by excluding the MIMIC-CXR classification task from the
task mixture and compared this model variant against Med-PaLM M 84B trained on
the full MultiMedBench mixture. As seen in Table 6, we observed that the model
trained jointly on both report generation and classification has higher
performance across the board on all report generation metrics. We also observe
that the model trained only on chest X-ray report generation can generalize to
abnormality detection in a zero-shot fashion with compelling performance, as
evidenced by a higher macro-F1 score. This is another example of
generalization to a novel task setting where the model learns to differentiate
between types of abnormalities from training on the more complex report
generation task.

Table 6: Positive task transfer between CXR report generation and abnormality classification. We observe positive transfer as a result of multi-task training with Med-PaLM M model trained jointly on both chest X-ray report generation and classification tasks. It exhibits higher performance on report generation metrics compared to a Med-PaLM M model trained without chest X-ray report classification. We also observe that training on the chest X-ray report generation task alone enables Med-PaLM M to generalize to abnormality detection in a zero-shot fashion. Dataset | Metric | Med-PaLM M (84B) |  | Med-PaLM M (84B)
---
No CXR classification
MIMIC-CXR | Micro-F1-14 | 53.56% | 52.94%
Macro-F1-14 | 39.83% | 38.92%
Micro-F1-5 | 57.88% | 57.58%
Macro-F1-5 | 51.60% | 51.32%
F1-RadGraph | 26.71% | 26.08%
BLEU-1 | 32.31% | 31.72%
BLEU-4 | 11.31% | 10.87%
ROUGE-L | 27.29% | 26.67%
CIDEr-D | 26.17% | 25.17%
MIMIC-CXR (5 conditions) | Macro-AUC | 78.35% | 73.88%
Macro-F1 | 36.83% |  43.97%

###  6.3 Med-PaLM M performs encouragingly on radiology report generation
across model scales

To further understand the clinical applicability of Med-PaLM M, we conducted
radiologist evaluations of model-generated chest X-ray reports (and reference
human baselines). Under this evaluation framework, we observe encouraging
quality of Med-PaLM M generated reports across model scales as detailed below.

####  6.3.1 Side-by-side evaluation

In a side-by-side evaluation, four clinician raters ranked the quality of four
radiology reports, comparing the radiologist-provided reference report from
the MIMIC-CXR dataset with reports generated by different Med-PaLM M model
scales (12B, 84B, and 562B).

Figure 4(a) summarizes how often each rater ranked a report generated by one
of the three Med-PaLM M variants or the reference report as the best among
four candidate reports. Averaged over all four raters, the radiologist-
provided reference report was ranked best in 37.14% of cases, followed by Med-
PaLM M (84B) which was ranked best in 25.78% of cases, and the other two model
scales, 12B and 562B, which were ranked best in 19.49% and 17.59% of cases
respectively.

To enable a direct comparison of reports generated by each Med-PaLM M model
scale to the radiologist-provided reference report, we derived pairwise
preferences from the four-way ranking and provided a breakdown for each rater
and model scale in Figure 4(b). Averaged over all four raters, Med-PaLM M 84B
was preferred over the reference report in 40.50% of cases, followed by the
other two model scales, 12B and 562B, which were preferred over the reference
report in 34.05% and 32.00% of cases, respectively.

(a) Best-ranked report in four-way comparison

(b) Pairwise preference of each model scale compared to reference report

Figure 4: Side-by-side human evaluation. Four clinician raters ranked the
quality of four radiology reports in a side-by-side evaluation, comparing the
radiologist-provided reference report from MIMIC-CXR with reports generated by
different Med-PaLM M model scale variants (12B, 84B, 562B).

####  6.3.2 Independent evaluation

We report the rates of omissions and errors radiologists identified in
findings paragraphs generated by Med-PaLM M. Figure 5 provides breakdowns by
model scales (12B, 84B, 562B). We observed different trends for omissions and
errors. For omissions, we observed the lowest rate of 0.12 (95% CI, 0.10 -
0.15) omissions per report on average for both the Med-PaLM M 12B and 84B
models, followed by 0.13 (95% CI, 0.11 - 0.16) for the 562B model.

In contrast, we measured the lowest mean error rate of 0.25 (95% CI, 0.22 -
0.28) for Med-PaLM M 84B, followed by 0.28 (95% CI, 0.24 - 0.31) for Med-PaLM
M 12B and 0.29 (95% CI, 0.25 - 0.32) for the 562B model. Notably, this error
rate is comparable to those reported for human radiologists baselines on the
MIMIC-CXR dataset in a prior study [14].

It is important to mention that our analysis is limited to errors of clinical
relevance, ensuring a specific focus on clinical interpretation. This includes
those errors related to the presence, location or severity of a clinical
finding. Example of non-clinical errors are passages referring to views or
prior studies not present, which stem from training artifacts.

These trends across model scales were identical for the subset of omissions
and errors that were marked as significant by radiologist raters. We refer the
reader to Table A.8 for an overview of error and omission rates, including
non-clinical errors.

Figure 5: Independent human evaluation. Rates of omissions and clinical errors
identified by clinician raters in radiology reports generated by Med-PaLM M.
Clinical errors are those related to the presence, location or severity of a
clinical finding. Figure 6: Qualitative example of reference and Med-PaLM M
generated chest X-ray reports. We present a qualitative example of chest X-ray
reports generated by Med-PaLM M across model scales along with the target
reference report. In this example, a panel of radiologists adjudicated the
Med-PaLM M 12B report to have two clinically significant errors and one
omission, the Med-PaLM M 84B report to have zero errors and zero omissions,
and the Med-PaLM M 562B report to have one clinically insignificant error and
no omissions.

In Figure 6, we illustrate a qualitative example of chest X-ray reports
generated by Med-PaLM M across three model sizes along with the target
reference report. For this example, our panel of radiologists judged the Med-
PaLM M 12B report to have two clinically significant errors and one omission,
the Med-PaLM M 84B report to have zero errors and zero omissions, and the Med-
PaLM M 562B report to have one clinically insignificant errors and no
omissions.

##  7 Discussion

To the best of our knowledge, Med-PaLM M is the first demonstration of a
generalist biomedical AI system that can interpret a wide range of medical
modalities, perform competently (including near or exceeding prior SOTA) on a
diverse array of tasks, and generalize to unseen biomedical concepts and
tasks. This potentially opens up new possibilities in applications spanning
scientific discovery to care delivery. We elaborate on the implications of
this development as well as the challenges and limitations below.

##### Lack of benchmarks a key bottleneck for the development of generalist
biomedical AI

AI progress to date has largely been catalyzed by the development of high
quality benchmarks. While there exists several single-task biomedical AI
datasets, there have been limited attempts to unify them and create benchmarks
for the development of generalist biomedical AI systems. Our curation of
MultiMedBench is a step towards addressing this unmet need. However, the
benchmark has several important limitations including limited size of the
individual datasets (a cumulative size of 1̃ million samples) and limited
modality and task diversity (e.g., lacking life sciences such as
transcriptomics and proteomics). Another key barrier to developing models for
use across an even wider variety of biomedical data types is the lack of large
scale multimodal datasets, which would permit joint learning and alignment of
the modality-specific encoders with the decoder.

##### Importance of medical finetuning and specialization

PaLM-E is a highly capable generalist AI model as evidenced by its SOTA
performance on a wide range of vision-language and embodied robotics tasks.
Yet, its out-of-the-box performance on MultiMedBench was poor and Med-PaLM M
outperforms it by a wide margin across model scales. This result suggests that
finetuning with domain-specific biomedical data is critical to achieving good
performance on biomedical tasks, perhaps due to the distribution shift
presented by the domain overall compared to the plethora of non-medical tasks
and modalities.

##### Scaling multimodal AI models is challenging

In the language domain, scaling the model has led to leapfrog improvements in
performance and emergent capabilities. However, our preliminary experiments
suggest this is likely more challenging for multimodal generalist models in
the biomedical task domain due to the medical data scarcity. Given the wide
array of modalities and tasks such generalist models are expected to
understand and tackle, it is crucial that the encoders for such diverse
modalities are scaled jointly with the language model. Otherwise, for tasks
that require interpretation of data from a combination of modalities, the
performance will end up being bottlenecked by the weakest encoder. We see
evidence of this in medical image classification tasks such as mammography and
dermatology where scaling the language model component has little effect on
the task performance as the potential key bottleneck is the vision encoder. It
is possible that the small volume of medical data in MultiMedBench is not be
sufficient to effectively adapt a ViT pretrained on natural images to the
medical domain, thereby limiting the benefits of model scaling. As such, our
study only provides some initial insights on the effect of model scaling on
biomedical task performance. Future research is needed to fully understand the
effect of model scaling by teasing apart the scaling effect of the language
model from that of modality-specific encoders, with sufficient amounts of
biomedical data.

##### Technical considerations for generalist biomedical AI

Med-PaLM M builds on state-of-the-art vision and language components such as
ViT and PaLM. Yet, putting them together requires careful considerations
around token lengths allocated to visual encoder outputs, total context length
of the model, sampling strategies, training data mixtures and so forth.
Further, simple, yet important techniques such as the use of one-shot training
with dummy image tokens make an important difference in the quality and
compute efficiency of the final model. With increasing generality of the AI
system, the number of details requiring careful consideration tends to
increase as well. We also note that Med-PaLM M architecture as setup currently
is not optimal for few-shot in-context learning.

##### Progress in AI for radiology report generation

Our evaluation by radiologists of Med-PaLM M generated radiology reports
suggests encouraging performance of the model on a challenging multimodal
task. In up to 40.50% of the cases, a Med-PaLM M generated report was
preferred over the human-generated reference report. Further, the average
number of clinically significant errors within the model responses is
comparable to those reported for human-generated reports in prior studies [14]
on the same dataset. These promising results underpin rapid development in the
task of automatic radiology report generation and suggest the potential for
clinical utility in the future.

##### Generalist agents are not the only approach to multimodal biomedical AI

While generalist biomedical AI systems offer exciting possibilities [43],
there are other approaches to developing multimodal biomedical AI systems that
might be more applicable depending on data availability, pretrained models,
compute and application scenarios. These include leveraging frozen encoders
with adapter layers [75] to glue together a multimodal biomedical AI system or
developing LLMs that can interface with specialist biomedical encoders or
task-specific agents through tool use [76].

##### Considerations for real-world applications of generalist biomedical AI

While the development of generally capable biomedical AI systems is exciting,
for such systems to be useful in practice or opening the door to new
applications, they need to match or exceed specialized, single-task models or
otherwise reach clinically applicable levels of performance. While beyond the
scope of this work, the progress here necessitates careful considerations of
safety and equity in the development and validation of such systems.

##  8 Perspective on Generalist Biomedical AI

Reaching near or above SOTA on a diverse range of biomedical tasks with a
single set of model weights is a noteworthy milestone for the development of
generalist biomedical AI systems. While human clinicians can train for
“general practice” [77], helpful subspecialty-specific expertise is often
found in different experts [78], to whom non-specialist clinicians may refer
for specialist opinions in the course of care. It is also commonplace for
multiple physician specialities to work together in care delivery. We envisage
a similar future for biomedical AI where generalist and specialist AI systems
interact and collaborate together with expert clinicians and researchers in a
tight feedback loop to tackle grand challenges in biomedicine.

Our finding, of a single generalist biomedical AI that reaches compelling
performance across disparate tasks and contexts, hints at new frontiers for
impact in applications. This includes the potential for near zero-shot insight
in new domains, as a tool for discovery integrating insights from distinct
areas of biomedicine, and as a common point of assistance providing access to
expertise from many different fields.

##  9 Conclusion

Medicine is a multidisciplinary endeavour. Generalist biomedical AI systems
that effectively assimilate and encode multimodal medical data at scale and
rapidly adapt to new clinical contexts are likely to be the foundation of next
generation learning health systems and make healthcare more accessible,
efficient, equitable and humane. While further development and rigorous
validation is needed, we believe Med-PaLM M represents an important step
towards the development of such generalist biomedical AI.

#### Acknowledgments

This project was an extensive collaboration between many teams at Google
Research and Google DeepMind. We thank Andrew Sellergren, Yuan Liu, Michael
Howell, Julie Wang, Sho Kannan, Christine Kingsley, Roy Lee, Naama Hammel, Jay
Hartford, Preeti Singh, Kavita Kulkarni, Gavriel Goidel, Anil Palepu, Si Wai
Man, Amy Wang, Sami Lachgar, Lauren Winer, Maggie Shiels, Annisah Um’rani,
John Guilyard, Shravya Shetty and Evan Rapoport for their valuable insights
and feedback during our research. We are also grateful to Karen DeSalvo,
Zoubin Ghahramani, James Manyika, and Jeff Dean for their support during the
course of this project.

#### Data Availability

The benchmark used for training and evaluation in this study, MultiMedBench,
comprises de-identified datasets that are all open source. We present an
overview of datasets in Table 1.

#### Code Availability

We will not be able to open source the large language models (LLMs) used in
this study. We have provided comprehensive details regarding our underlying
methodology and build on previously detailed models [8, 10], so that similar
approaches can be tried with other classes of LLMs.

## References

  * [1] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau and Sebastian Thrun  “Dermatologist-level classification of skin cancer with deep neural networks”  In _nature_ 542.7639 Nature Publishing Group, 2017, pp. 115–118
  * [2] Varun Gulshan, Lily Peng, Marc Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams and Jorge Cuadros  “Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs”  In _Jama_ 316.22 American Medical Association, 2016, pp. 2402–2410
  * [3] Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri and Ivan Protsyuk  “A clinically applicable approach to continuous prediction of future acute kidney injury”  In _Nature_ 572.7767 Nature Publishing Group UK London, 2019, pp. 116–119
  * [4] Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S Corrado and Ara Darzi  “International evaluation of an AI system for breast cancer screening”  In _Nature_ 577.7788 Nature Publishing Group UK London, 2020, pp. 89–94
  * [5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut and Emma Brunskill  “On the opportunities and risks of foundation models”  In _arXiv preprint arXiv:2108.07258_ , 2021
  * [6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry and Amanda Askell  “Language models are few-shot learners”  In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [7] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Nenad Tomasev, Jovana Mitrović and Patricia Strachan  “Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging”  In _Nature Biomedical Engineering_ Nature Publishing Group UK London, 2023, pp. 1–24
  * [8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton and Sebastian Gehrmann  “PaLM: Scaling language modeling with pathways”  In _arXiv preprint arXiv:2204.02311_ , 2022
  * [9] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis and Stephen Pfohl  “Large Language Models Encode Clinical Knowledge”  In _arXiv preprint arXiv:2212.13138_ , 2022
  * [10] Danny Driess, Fei Xia, Mehdi S.. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch and Pete Florence  “PaLM-E: An Embodied Multimodal Language Model”  In _arXiv preprint arXiv:2303.03378_ , 2023
  * [11] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa and Lucas Beyer  “Pali: A jointly-scaled multilingual language-image model”  In _arXiv preprint arXiv:2209.06794_ , 2022
  * [12] Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang and Xiao-Ming Wu  “Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering”  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , 2021, pp. 1650–1654  IEEE
  * [13] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou and Donald Metzler  “Emergent abilities of large language models”  In _arXiv preprint arXiv:2206.07682_ , 2022
  * [14] Jaehwan Jeong, Katherine Tian, Andrew Li, Sina Hartung, Fardad Behzadi, Juan Calle, David Osayande, Michael Pohlen, Subathra Adithan and Pranav Rajpurkar  “Multimodal image-text matching improves retrieval-based chest X-ray report generation”  In _arXiv preprint arXiv:2303.17579_ , 2023
  * [15] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos and Ibrahim Alabdulmohsin  “Scaling vision transformers to 22 billion parameters”  In _arXiv preprint arXiv:2302.05442_ , 2023
  * [16] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier and Marco Tagliasacchi  “Audiolm: a language modeling approach to audio generation”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ IEEE, 2023
  * [17] Rich Caruana  “Multitask learning”  In _Machine learning_ 28.1 Springer, 1997, pp. 41–75
  * [18] Sebastian Thrun  “Lifelong learning algorithms”  In _Learning to learn_ Springer, 1998, pp. 181–209
  * [19] Geoffrey E Hinton, Simon Osindero and Yee-Whye Teh  “A fast learning algorithm for deep belief nets”  In _Neural computation_ 18.7 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2006, pp. 1527–1554
  * [20] Yoshua Bengio, Pascal Lamblin, Dan Popovici and Hugo Larochelle  “Greedy layer-wise training of deep networks”  In _Advances in neural information processing systems_ 19, 2006
  * [21] Pascal Vincent, Hugo Larochelle, Yoshua Bengio and Pierre-Antoine Manzagol  “Extracting and composing robust features with denoising autoencoders”  In _Proceedings of the 25th international conference on Machine learning_ , 2008, pp. 1096–1103
  * [22] Yoshua Bengio  “Deep learning of representations for unsupervised and transfer learning”  In _Proceedings of ICML workshop on unsupervised and transfer learning_ , 2012, pp. 17–36  JMLR WorkshopConference Proceedings
  * [23] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei  “Scaling laws for neural language models”  In _arXiv preprint arXiv:2001.08361_ , 2020
  * [24] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee and Andrew Y Ng  “Multimodal deep learning”  In _ICML_ , 2011
  * [25] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis and Mitchell Wortsman  “Laion-5b: An open large-scale dataset for training next generation image-text models”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 25278–25294
  * [26] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman and Joao Carreira  “Perceiver: General perception with iterative attention”  In _International conference on machine learning_ , 2021, pp. 4651–4664  PMLR
  * [27] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals and Felix Hill  “Multimodal few-shot learning with frozen language models”  In _Advances in Neural Information Processing Systems_ 34, 2021, pp. 200–212
  * [28] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican and Malcolm Reynolds  “Flamingo: a visual language model for few-shot learning”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 23716–23736
  * [29] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts and Marco Tagliasacchi  “Musiclm: Generating music from text”  In _arXiv preprint arXiv:2301.11325_ , 2023
  * [30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin and Jack Clark  “Learning transferable visual models from natural language supervision”  In _International conference on machine learning_ , 2021, pp. 8748–8763  PMLR
  * [31] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth and Li-Jia Li  “YFCC100M: The new data in multimedia research”  In _Communications of the ACM_ 59.2 ACM New York, NY, USA, 2016, pp. 64–73
  * [32] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  Ieee
  * [33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer  “High-Resolution Image Synthesis with Latent Diffusion Models”, 2021  arXiv:2112.10752 [cs.CV]
  * [34] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan and Tim Salimans  “Photorealistic text-to-image diffusion models with deep language understanding”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 36479–36494
  * [35] Prafulla Dhariwal and Alexander Nichol  “Diffusion models beat gans on image synthesis”  In _Advances in neural information processing systems_ 34, 2021, pp. 8780–8794
  * [36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever  “Language models are unsupervised multitask learners”  In _OpenAI blog_ 1.8, 2019, pp. 9
  * [37] Ting Chen, Saurabh Saxena, Lala Li, David J Fleet and Geoffrey Hinton  “Pix2seq: A language modeling framework for object detection”  In _arXiv preprint arXiv:2109.10852_ , 2021
  * [38] Ronan Collobert and Jason Weston  “A unified architecture for natural language processing: Deep neural networks with multitask learning”  In _Proceedings of the 25th international conference on Machine learning_ , 2008, pp. 160–167
  * [39] Sebastian Ruder  “An overview of multi-task learning in deep neural networks”  In _arXiv preprint arXiv:1706.05098_ , 2017
  * [40] Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang and Henryk Michalewski  “Multi-game decision transformers”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 27921–27936
  * [41] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi and Aniruddha Kembhavi  “Unified-io: A unified model for vision, language, and multi-modal tasks”  In _arXiv preprint arXiv:2206.08916_ , 2022
  * [42] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay and Jost Tobias Springenberg  “A generalist agent”  In _arXiv preprint arXiv:2205.06175_ , 2022
  * [43] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol and Pranav Rajpurkar  “Foundation models for generalist medical artificial intelligence”  In _Nature_ 616.7956 Nature Publishing Group UK London, 2023, pp. 259–265
  * [44] Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng and X Shirley Liu  “Transfer learning enables predictions in network biology”  In _Nature_ Nature Publishing Group UK London, 2023, pp. 1–9
  * [45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [46] Kai Zhang, Jun Yu, Zhiling Yan, Yixin Liu, Eashan Adhikarla, Sunyang Fu, Xun Chen, Chen Chen, Yuyin Zhou and Xiang Li  “BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks”  In _arXiv preprint arXiv:2305.17100_ , 2023
  * [47] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon and Jianfeng Gao  “LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day”  In _arXiv preprint arXiv:2306.00890_ , 2023
  * [48] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li and Scott Lundberg  “Sparks of artificial general intelligence: Early experiments with gpt-4”  In _arXiv preprint arXiv:2303.12712_ , 2023
  * [49] Kathryn Wantlin, Chenwei Wu, Shih-Cheng Huang, Oishi Banerjee, Farah Dadabhoy, Veeral Vipin Mehta, Ryan Wonhee Han, Fang Cao, Raja R Narayan and Errol Colak  “BenchMD: A Benchmark for Modality-Agnostic Learning on Medical Images and Sensors”  In _arXiv preprint arXiv:2304.08486_ , 2023
  * [50] Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Daniel Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang and Sudip Roy  “Pathways: Asynchronous distributed dataflow for ML”  In _Proceedings of Machine Learning and Systems_ 4, 2022, pp. 430–449
  * [51] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta and Adrià Garriga-Alonso  “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models”  In _arXiv preprint arXiv:2206.04615_ , 2022
  * [52] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold and Sylvain Gelly  “An image is worth 16x16 words: Transformers for image recognition at scale”  In _arXiv preprint arXiv:2010.11929_ , 2020
  * [53] Chen Sun, Abhinav Shrivastava, Saurabh Singh and Abhinav Gupta  “Revisiting unreasonable effectiveness of data in deep learning era”  In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 843–852
  * [54] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby and Lucas Beyer  “Scaling vision transformers”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12104–12113
  * [55] Kenneth Marino, Mohammad Rastegari, Ali Farhadi and Roozbeh Mottaghi  “Ok-vqa: A visual question answering benchmark requiring external knowledge”  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , 2019, pp. 3195–3204
  * [56] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra and Devi Parikh  “Making the v in vqa matter: Elevating the role of image understanding in visual question answering”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6904–6913
  * [57] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai and Quoc V Le  “Finetuned language models are zero-shot learners”  In _arXiv preprint arXiv:2109.01652_ , 2021
  * [58] Noam Shazeer and Mitchell Stern  “Adafactor: Adaptive learning rates with sublinear memory cost”  In _International Conference on Machine Learning_ , 2018, pp. 4596–4604  PMLR
  * [59] Stefan Jaeger, Sema Candemir, Sameer Antani, Yı̀-Xiáng J Wáng, Pu-Xuan Lu and George Thoma  “Two public chest X-ray datasets for computer-aided screening of pulmonary diseases”  In _Quantitative imaging in medicine and surgery_ 4.6 AME Publications, 2014, pp. 475
  * [60] Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad and Andrew Y Ng  “Evaluating progress in automatic chest x-ray radiology report generation”  In _medRxiv_ Cold Spring Harbor Laboratory Press, 2022, pp. 2022–08
  * [61] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis and Darlene Neal  “Towards expert-level medical question answering with large language models”  In _arXiv preprint arXiv:2305.09617_ , 2023
  * [62] Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Manuel Zambrano Chaves and Curtis P Langlotz  “RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models”  In _arXiv preprint arXiv:2305.01146_ , 2023
  * [63] Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Laila Bashmal and Mansour Zuair  “Vision–Language Model for Visual Question Answering in Medical Imagery”  In _Bioengineering_ 10.3 MDPI, 2023, pp. 380
  * [64] Tom Sonsbeek, Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees GM Snoek and Marcel Worring  “Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models”  In _arXiv preprint arXiv:2303.05977_ , 2023
  * [65] Aaron Nicolson, Jason Dowling and Bevan Koopman  “Improving chest X-Ray report generation by leveraging warm-starting”  In _arXiv preprint arXiv:2201.09405_ , 2022
  * [66] Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P Langlotz and Dan Jurafsky  “Improving factual completeness and consistency of image-to-text radiology report generation”  In _arXiv preprint arXiv:2010.10042_ , 2020
  * [67] Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid and Anja Thieme  “Learning to exploit temporal structure for biomedical vision-language processing”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 15016–15027
  * [68] Tim Tanida, Philip Müller, Georgios Kaissis and Daniel Rueckert  “Interactive and Explainable Region-guided Radiology Report Generation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7433–7442
  * [69] Ravidu Suien Rammuni Silva and Pumudu Fernando  “Effective utilization of multiple convolutional neural networks for chest X-ray classification”  In _SN Computer Science_ 3.6 Springer, 2022, pp. 492
  * [70] Adarsh Bhandary Panambur, Prathmesh Madhu and Andreas Maier  “Effect of Random Histogram Equalization on Breast Calcification Analysis Using Deep Learning”  In _Bildverarbeitung für die Medizin 2022: Proceedings, German Workshop on Medical Image Computing, Heidelberg, June 26-28, 2022_ , 2022, pp. 173–178  Springer
  * [71] Ryan Poplin, Pi-Chuan Chang, David Alexander, Scott Schwartz, Thomas Colthurst, Alexander Ku, Dan Newburger, Jojo Dijamco, Nam Nguyen, Pegah T Afshar, Sam S Gross, Lizzie Dorfman, Cory Y McLean and Mark A DePristo  “A universal SNP and small-indel variant caller using deep neural networks”  In _Nature Biotechnology_ 36.10 Springer ScienceBusiness Media LLC, 2018, pp. 983–987
  * [72] Seonghyeon Ye, Joel Jang, Doyoung Kim, Yongrae Jo and Minjoon Seo  “Retrieval of Soft Prompt Enhances Zero-Shot Task Generalization”  In _arXiv preprint arXiv:2210.03029_ , 2022
  * [73] Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y Ng and Pranav Rajpurkar  “Retrieval-based chest x-ray report generation using a pre-trained contrastive language-image model”  In _Machine Learning for Health_ , 2021, pp. 209–219  PMLR
  * [74] Mustapha Oloko-Oba and Serestina Viriri  “Ensemble of EfficientNets for the Diagnosis of Tuberculosis”  In _Computational Intelligence and Neuroscience_ 2021 Hindawi, 2021
  * [75] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao and Yu Qiao  “Llama-adapter: Efficient fine-tuning of language models with zero-init attention”  In _arXiv preprint arXiv:2303.16199_ , 2023
  * [76] Timo Schick, Jane Dwivedi-Yu, Roberto Dess, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda and Thomas Scialom  “Toolformer: Language models can teach themselves to use tools”  In _arXiv preprint arXiv:2302.04761_ , 2023
  * [77] Martin Marshall  “The future of general practice in England”  In _bmj_ 379 British Medical Journal Publishing Group, 2022
  * [78] Lindsay Blank, Susan Baxter, Helen Buckley Woods, Elizabeth Goyder, Andrew Lee, Nick Payne and Melanie Rimmer  “Referral interventions from primary to specialist care: a systematic review of international evidence”  In _British Journal of General Practice_ 64.629 British Journal of General Practice, 2014, pp. e765–e774
  * [79] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang and Peter Szolovits  “What disease does this patient have? a large-scale open domain question answering dataset from medical exams”  In _Applied Sciences_ 11.14 MDPI, 2021, pp. 6421
  * [80] Ankit Pal, Logesh Kumar Umapathi and Malaikannan Sankarasubbu  “MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”  In _Conference on Health, Inference, and Learning_ , 2022, pp. 248–260  PMLR
  * [81] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen and Xinghua Lu  “PubMedQA: A dataset for biomedical research question answering”  In _arXiv preprint arXiv:1909.06146_ , 2019
  * [82] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi and Roger G Mark  “MIMIC-III, a freely accessible critical care database”  In _Scientific data_ 3.1 Nature Publishing Group, 2016, pp. 1–9
  * [83] Jean-Benoit Delbrouck, Khaled Saab, Maya Varma, Sabri Eyuboglu, Pierre Chambon, Jared Dunnmon, Juan Zambrano, Akshay Chaudhari and Curtis Langlotz  “ViLMedic: a framework for research at the intersection of vision and language in medical AI”  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 2022, pp. 23–34
  * [84] Jean-Benoit Delbrouck, Maya Varma and Curtis P Langlotz  “Toward expanding the scope of radiology report summarization to multiple anatomies and modalities”  In _arXiv preprint arXiv:2211.08584_ , 2022
  * [85] Andre GC Pacheco, Gustavo R Lima, Amanda S Salomao, Breno Krohling, Igor P Biral, Gabriel G Angelo, Fábio CR Alves Jr, José GM Esgario, Alana C Simora and Pedro BC Castro  “PAD-UFES-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones”  In _Data in brief_ 32 Elsevier, 2020, pp. 106221
  * [86] Ekin D Cubuk, Barret Zoph, Jonathon Shlens and Quoc V Le  “Randaugment: Practical automated data augmentation with a reduced search space”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 702–703
  * [87] Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao and Van Vu  “VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography”  In _Scientific Data_ 10.1 Nature Publishing Group UK London, 2023, pp. 277
  * [88] Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy and Daniel L Rubin  “A curated mammography data set for use in computer-aided detection and diagnosis research”  In _Scientific data_ 4.1 Nature Publishing Group, 2017, pp. 1–9
  * [89] Nathan D. Olson, Justin Wagner, Jennifer McDaniel, Sarah H. Stephens, Samuel T. Westreich and al  “PrecisionFDA Truth Challenge V2: Calling variants from short and long reads in difficult-to-map regions”  In _Cell Genomics_ 2.5 Elsevier BV, 2022, pp. 100129
  * [90] Mark A DePristo, Eric Banks, Ryan Poplin, Kiran V Garimella, Jared R Maguire, Christopher Hartl and al  “A framework for variation discovery and genotyping using next-generation DNA sequencing data”  In _Nature Genetics_ 43.5 Springer ScienceBusiness Media LLC, 2011, pp. 491–498
  * [91] Saud H. AlDubayan, Jake R. Conway, Sabrina Y. Camp, Leora Witkowski, Eric Kofman, Brendan Reardon, Seunghun Han, Nicholas Moore, Haitham Elmarakeby, Keyan Salari, Hani Choudhry, Abdullah M. Al-Rubaish, Abdulsalam A. Al-Sulaiman, Amein K. Al-Ali, Amaro Taylor-Weiner and Eliezer M. Allen  “Detection of Pathogenic Variants With Germline Genetic Testing Using Deep Learning vs Standard Methods in Patients With Prostate Cancer and Melanoma”  In _JAMA_ 324.19 American Medical Association (AMA), 2020, pp. 1957
  * [92] Wen-Wei Liao, Mobin Asri, Jana Ebler, Daniel Doerr, Marina Haukness, Glenn Hickey and al  “A draft human pangenome reference”  In _Nature_ 617.7960 Springer ScienceBusiness Media LLC, 2023, pp. 312–324
  * [93] H. Thorvaldsdottir, J.. Robinson and J.. Mesirov  “Integrative Genomics Viewer (IGV): high-performance genomics data visualization and exploration”  In _Briefings in Bioinformatics_ 14.2 Oxford University Press (OUP), 2012, pp. 178–192
  * [94] Justin M. Zook, David Catoe, Jennifer McDaniel, Lindsay Vang, Noah Spies and al  “Extensive sequencing of seven human genomes to characterize benchmark reference materials”  In _Scientific Data_ 3.1 Springer ScienceBusiness Media LLC, 2016
  * [95] Jason J Lau, Soumya Gayen, Asma Ben Abacha and Dina Demner-Fushman  “A dataset of clinically generated visual questions and answers about radiology images”  In _Scientific data_ 5.1 Nature Publishing Group, 2018, pp. 1–10
  * [96] Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing and Pengtao Xie  “Pathvqa: 30000+ questions for medical visual question answering”  In _arXiv preprint arXiv:2003.10286_ , 2020
  * [97] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark and Steven Horng  “MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports”  In _Scientific data_ 6.1 Nature Publishing Group UK London, 2019, pp. 317
  * [98] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball and Katie Shpanskaya  “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison”  In _Proceedings of the AAAI conference on artificial intelligence_ 33.01, 2019, pp. 590–597
  * [99] Zhihong Chen, Yan Song, Tsung-Hui Chang and Xiang Wan  “Generating radiology reports via memory-driven transformer”  In _arXiv preprint arXiv:2010.16056_ , 2020
  * [100] Chin-Yew Lin  “Rouge: A package for automatic evaluation of summaries”  In _Text summarization branches out_ , 2004, pp. 74–81
  * [101] Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu  “Bleu: a method for automatic evaluation of machine translation”  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318
  * [102] Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren and Andrew Y Ng  “Radgraph: Extracting clinical entities and relations from radiology reports”  In _arXiv preprint arXiv:2106.14463_ , 2021
  * [103] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen  “Lora: Low-rank adaptation of large language models”  In _arXiv preprint arXiv:2106.09685_ , 2021
  * [104] Eric Lehman and Alistair Johnson  “Clinical-t5: Large language models built using mimic clinical text”  PhysioNet, 2023
  * [105] Daniel GP Petrini, Carlos Shimizu, Rosimeire A Roela, Gabriel Vansuita Valente, Maria Aparecida Azevedo Koike Folgueira and Hae Yong Kim  “Breast cancer diagnosis in two-view mammography using end-to-end trained efficientnet-based convolutional network”  In _Ieee access_ 10 IEEE, 2022, pp. 77723–77731
  * [106] Wei Dai, Rui Liu, Tianyi Wu, Min Wang, Jianqin Yin and Jun Liu  “Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attention”  In _arXiv preprint arXiv:2205.04326_ , 2022
  * [107] Leandro M Lima and Renato A Krohling  “Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets”  In _Intelligent Systems: 11th Brazilian Conference, BRACIS 2022, Campinas, Brazil, November 28–December 1, 2022, Proceedings, Part II_ , 2022, pp. 282–296  Springer
  * [108] Yunyi Liu, Zhanyu Wang, Dong Xu and Luping Zhou  “Q2atransformer: Improving medical vqa via an answer querying decoder”  In _International Conference on Information Processing in Medical Imaging_ , 2023, pp. 445–456  Springer
  * [109] Sedigheh Eslami, Gerard Melo and Christoph Meinel  “Does clip benefit visual question answering in the medical domain as much as it does in the general domain?”  In _arXiv preprint arXiv:2112.13906_ , 2021
  * [110] Ramakrishna Vedantam, C Lawrence Zitnick and Devi Parikh  “Cider: Consensus-based image description evaluation”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4566–4575
  * [111] Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng and Matthew P Lungren  “CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT”  In _arXiv preprint arXiv:2004.09167_ , 2020
  * [112] Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits and Marzyeh Ghassemi  “Clinically accurate chest x-ray report generation”  In _Machine Learning for Healthcare Conference_ , 2019, pp. 249–269  PMLR
  * [113] Yufei Xu, Qiming Zhang, Jing Zhang and Dacheng Tao  “Vitae: Vision transformer advanced by exploring intrinsic inductive bias”  In _Advances in neural information processing systems_ 34, 2021, pp. 28522–28535
  * [114] Vignav Ramesh, Nathan A Chi and Pranav Rajpurkar  “Improving radiology report generation systems by removing hallucinated references to non-existent priors”  In _Machine Learning for Health_ , 2022, pp. 456–473  PMLR

Appendix

In the following sections, we report additional experiments and detailed
analysis to further illustrate the performance of our proposed generalist
model, Med-PaLM M.

We provide details on:

  * •

Datasets and tasks in MultiMedBench

  * •

Med-PaLM M training procedure

  * •

Interpretations of Med-PaLM M performance by task type:

    * –

Performance analysis on language-only medical question answering

    * –

Performance analysis on radiology report summarization

    * –

Performance analysis on medical image classification tasks

    * –

Performance analysis on medical visual question answering

    * –

Performance analysis on chest X-ray report generation

  * •

Human evaluation of model-generated chest X-ray reports

  * •

Examples from MultiMedBench tasks

##  A.1 MultiMedBench

In this section, we offer a comprehensive overview of _MultiMedBench_ ,
including a detailed description of the datasets, data preprocessing, and task
setups. Figure A.1 summarizes MultiMedBench over its various biomedical tasks.

###  A.1.1 Language-only datasets

##### MultiMedQA

We used three of the multiple-choice medical question-answering datasets from
MultiMedQA [9]: the MedQA [79], MedMCQA [80], and PubMedQA [81] datasets for
training and evaluation of Med-PaLM M. These question answering tasks are
language-only and do not require the interpretation of additional modalities.
The training set consists of 10,178 questions from MedQA and 182,822 questions
from MedMCQA. The test set comprises 1,273 questions from MedQA, 4,183
questions from MedMCQA, and 500 questions from PubMedQA. Note that PubMedQA
was not included in the training data mixture and only used for evaluation.

##### MIMIC-III

is a large publicly-available medical database that contains medical records
of patients admitted to intensive care units [82]. It contains 79,790
radiology reports across two imaging modalities (CT and MRI) and seven
anatomic regions (head, abdomen, chest, head, neck, sinus, spine, pelvis). A
total of 78,875 reports were chosen based on criteria such as the length of
the report. We used the radiology report summarization dataset from [62],
which comprises six most common modality/anatomy pairs for training and
evaluation: CT head, CT abdomen, CT chest, MRI head, CT spine, and CT neck. To
evaluate out-of-distribution (OOD) performance we used five less common
modality/anatomy pairs: MRI spine, CT sinus, MRI abdomen, MRI pelvis, and MRI
neck. This resulted in a total of 58,405 reports for training, 7,413 reports
for validation, and 13,057 reports for testing. Note that chest X-ray reports
are excluded from this dataset to avoid data contamination with the MIMIC-CXR
dataset for the report generation task. For each report, we used the same
preprocessing functions as in [83, 84] to extract the findings and impression
sections. Specifically, we filtered out the reports whose findings section are
longer than 600 tokens. We performed a report summarization task by predicting
the impression section given the findings section as input, which is another
language-only task that does not require multi-modal input.

Figure A.1: MultiMedBench overview. MultiMedBench is a benchmark that covers
14 different biomedical tasks, including question answering, visual question
answering, image classification, radiology report generation and
summarization, and genomic variant calling. MultiMedBench comprises more than
1 million data samples from a diverse range of medical images, radiology
reports, medical question answers, and visual question answering pairs.

###  A.1.2 Multimodal datasets

##### PAD-UFES-20

consists of 2,298 clinical images of skin lesions collected from different
smartphone devices with varying resolutions, sizes, and lighting conditions
[85]. The data was collected through the Dermatological and Surgical
Assistance Program at the Federal University of Espírito Santo (UFES-Brazil),
a nonprofit program that provides free skin lesion treatment. The dataset
contains six different types of skin lesions including: Basal Cell Carcinoma
(BCC), Malignant Melanoma (MEL), Squamous Cell Carcinoma (SCC), Actinic
Keratosis (ACK), Melanocytic Nevus (NEV), and Seborrheic Keratosis (SEK). Each
image is associated with up to 21 patient clinical features such as patient
demographics, family cancer history lesion location, lesion size. We set up a
6-class classification task in a generative framework through a language
decoder using skin lesion images and the associated clinical textual features
as the multimodal input. Specifically, we selected 14 clinical attributes in
the metadata for each lesion including: age, gender, smoke, drink, skin cancer
history, cancer history, region, fitspatrick, horizontal and vertical
diameters, itch, grew, bleed, and elevation. The class ratio is approximately
16:1:4:14:5:4 over three skin cancers (BCC, MEL, and SCC) and three skin
disease (ACK, NEV, and SEK). Since there are no published official train/test
splits, we randomly split the dataset into a training set (80%) and a test
test (20%) using a stratified sampling to the preserve original class ratio.
We applied a series of image augmentation operations using RandAugment [86] to
the training set including: autoContrast, equalize, invert, rotate, posterize,
solarize, color, and contrast.

##### VinDr-Mammo

is a full-field digital mammography dataset which consists of 5000 breast
X-ray imaging studies and a total of 20,000 gray-scale images with extensive
breast-level assessment and lesion-level annotations, collected from two
hospitals in in Hanoi, Vietnam [87]. Each study contains four images where the
left and right breasts are imaged with mediolateral-oblique (MLO) and cranio-
caudal (CC) views. Each image has breast-level assessment following the Breast
Imaging Reporting and Data System (BI-RADS). BI-RADS assessment ranges from 1
(negative) to 5 (highly suggestive of malignancy). In addition to the BI-RADS
score, the breast density level is also provided as well as regional
abnormality finding annotations. We performed a breast-level 5-class BI-RADS
classification task similar to the setup in [49], except that the laterality
and view position of the image was provided as additional contextual features.
We used the official train/test splits where the train split contains 16,000
samples with a class ratio of 60:21:4:3:1 across BI-RADS 1-5, respectively and
the test split contains 4,000 samples with the same class ratio. We applied
the following transformations to the images in the training set: contrast,
equalize, rotate, shearX, shearY, translateX, and translateY. To mitigate the
class imbalance in the training data, we upsampled for each minority class
(BI-RADS 2-5) by a factor of 3.

##### CBIS-DDSM

is the Curated Breast Imaging Subset of Digital Database for Screening
Mammography [88]. This dataset contains 2,620 scanned film mammography
studies. Unlike VinDr-Mammo, CBIS-DDSM does not have breast-level BI-RADS
assessment. Annotations are provided at the lesion level including BI-RADS,
subtlety level, and pathology type. There are two types of lesions: mass and
calcification. Both of them are annotated with three possible pathology
labels: benign, benign without callback, and malignant. We performed a 3-class
abnormality (patch-level) pathology classification task on this dataset for
mass and calcification abnormalities separately. Abnormality image patch is
cropped by the bounding box of the region-of-interest (ROI) from the full
mammogram and used as the model input along with its view position (CC or MLO)
information. We used the official train/test splits for both abnormality
types. For mass cases, the training and test sets contain 1,318 and 378 images
(class ratio: 6:1:6), respectively. For calcification cases, the total number
of images in the training and test sets are 1,544 and 326 (class ratio:
1:1:1), respectively. For both cases, we applied the same image augmentation
as in VinDr-Mammo to the training set.

##### PrecisionFDA Truth Challenge V2

was developed for benchmarking the state-of-the-art of variant calling in
challenging genomics regions [89]. Genomic variant calling is a task aiming at
identifying genetic variants from sequencing data [90], which can identify
disease-causing mutations [91]. For variant calling, sequencing data is mapped
to the coordinates of a reference genome [92]. The mappings can be represented
as an image-like format that computational methods such as DeepVariant [71]
use to call variants, or in a human-friendly image format which experts use to
inspect and quality control variants of interest [93]. For this task, we used
an extensively characterized groundtruth set from the National Institute of
Standards and Technology (NIST) [94] for the HG002 sample. We generated
examples from sequencing from the PrecisionFDA Truth Challenge V2. For
training, we use 4% of the examples from the whole genome (except for
chromosome 20, 21, and 22). For evaluation, we used chromosome20, bases
3000001-9444417. This generated 197,038 candidate variants for training and
13,030 candidate variants for evaluation. For each example, the model predicts
three possible genotypes, corresponding to how many copies (0, 1, or 2) of the
given alternate allele are present. The training set consists of 45,011,
93,246, and 58,781 samples for classes 0, 1, 2, respectively. The evaluation
set contains 3,016, 6,169, and 3,845 for classes 0, 1, 2, respectively.

We used DeepVariant v1.3.0’s [71] example generation method to create image-
like examples suitable for machine classification. Specifically, input
examples to DeepVariant v1.3.0 have a shape of (100, 221, 6) corresponding to
(height, width, channels). Channels are shown in grey-scale below in the
following order:

  1. 1.

Read base: different intensities represent A, C, G, and T.

  2. 2.

Base quality: set by the sequencing machine. White is higher quality.

  3. 3.

Mapping quality: set by the aligner. White is higher quality.

  4. 4.

Strand of alignment: Black is forward; white is reverse.

  5. 5.

Read supports variant: White means the read supports the given alternate
allele, grey means it does not.

  6. 6.

Base differs from ref: White means the base is different from the reference,
dark grey means the base matches the reference.

To reshape the input example to be compatible with the Med-PaLM M input shape
of (224, 224, 3), we stacked up channels 1, 2, 3 with channels 4, 5, 6 such
that the original tensor of shape (100, 221, 6) became an RGB image of shape
(200, 221, 3). We then padded the image on the width and height dimensions to
give it a final shape of (224, 224, 3).

##### VQA-RAD

is a radiology visual question answering (VQA) dataset which consists of 315
radiology images and 3,515 question–answer pairs created and validated by
clinicians [95]. The radiology images are selected from three imaging
modalities (CT, MRI, and X-rays) and three anatomical regions (head,
abdominal, chest). The types of question fall into 11 categories including
modality, plane, organ system, abnormality, size, plane, positional reasoning,
color, counting, attribute and other. 58% of the question–answer (QA) pairs
are closed-ended (yes/no or limited choices) and the rest 42% are open-ended
(short answer). We adopted the official train/test splits, where the training
set contains 1,797 QA pairs (only free-form and paraphrased questions were
included) and the test set contains 451 QA pairs (not filtered).

##### Path-VQA

is a pathology VQA dataset, containing a total of 4,998 pathology images with
32,799 question-answer pairs [96]. Pathology images are extracted from medical
textbooks and online digital libraries. Each image is associated with multiple
QA pairs pertaining to different aspects of the pathology including color,
location, appearance, shape, etc. Open-ended questions account for 50.2% of
all questions, which are categorized into 7 categories: what, where, when,
whose, how, and how much/how many, accounting for 50.2% of all questions. The
rest are close-ended questions with simple "yes/no" answer. We adopted the
official data partitioning where the training, validation, and test sets
contain 19,755, 6,279, and 6,761 QA pairs, respectively.

##### Slake-VQA

is a semantically annotated and knowledge-enhanced bilingual (English and
Chinese) VQA dataset on radiology images [12]. It contains 642 annotated
images with 14,028 question-answer pairs covering 12 diseases, 39 organ
systems and 3 imaging modalities (CT, MRI, and chest X-rays). Questions are
either open-ended (free-form) or closed-ended (balanced yes/no) related to
various aspects of the image content including plane, quality, position,
organ, abnormality, size, color, shape, knowledge graph, etc. The training,
validation, and test sets contain 9,849, 2,109, and 2,070 samples,
respectively.

##### MIMIC-CXR

is a large dataset of chest radiographs with free-text radiology reports [97].
A total of 377,110 images are available in the dataset from 227,835 image
studies collected for 65,379 patients. Each patient may have multiple studies
and each study may contain one or more images associated with the same free-
text report. Images in MIMIC-CXR are collected from multiple view positions:
e.g., anterior-posterior (AP), posterior-anterior, and lateral (LA). Protected
health information (PHI) in radiology reports and images is removed, which
results in missing information in some sentences of the reports. Since this
dataset contains sequential imaging studies of an individual patient, a large
number of reports refer to information in prior studies of the same patient.
Each report is annotated with structured labels of 14 common radiological
observations using CheXpert labeler [98]. We performed two tasks using this
dataset: chest X-ray report generation and binary classification of
clinically-relevant pathology observations. We preprocessed the radiology
reports by extracting the indication, findings, and impression sections,
removing redundant white-spaces in the reports, following previous work [99].
We used the official train/validation/test splits. We discarded images without
reports and reports where the findings section can not be extracted across
train and test. We also filtered out the reports where the length of findings
section exceeds 800 characters. However, unlike most previous work using
focusing only on the frontal view, we treated images of different orientation
that are associated with the same report as independent samples (retaining the
patient-level train/test splits to avoid contamination of the test data). The
goal is to improve the image understanding capability of the model to process
images of different view positions. In a separate evaluation, we also studied
a subset of samples where reports are accompanied by both a front and lateral
view (two-view report generation).

For the report generation task, we combined the chest X-ray image with the
contextual information from the indication section (reason for the study) to
predict the findings section of the target report. The total number of samples
in the training, validation, and test sets are: 353,542, 2,866, and 4,834,
respectively.

For the binary classification task, we grouped negative and uncertain labels
as the negative class for 11 pathological conditions: no finding, atelectasis,
cardiomegaly, consolidation, edema, pleural effusion, lung opacity, enlarged
cardiomediastinum, fracture, pneumonia, and support devices. Atelectasis,
cardiomegaly, consolidation, edema, and pleural effusion are 5 major
conditions given their clinical relevance and prevalence. The "No finding"
label captures the cases without any pathology and therefore this
classification task simply helps the model to distinguish normal cases from
cases with any type of abnormality. Due to class imbalance, during training we
upsampled the positive class by a factor of 2 for the following conditions:
consolidation, enlarged cardiomediastinum, fracture, and pneumonia. These
binary classification tasks are auxiliary to the report generation task when
they are trained simultaneously since they help the model to distinguish among
different types of clinical observations in the chest X-ray images.

Figure A.2: Med-PaLM M data mixture overview. Summary of the mixture ratio in
the Med-PaLM M training data mixture across MultiMedBench datasets as detailed
in Table A.1.

##  A.2 Med-PaLM M Training Details

###  A.2.1 Training data mixture

Figures A.2 and A.1 show the mixture ratio and few-shot task setup of the
training data mixture. The majority of the data distribution is medical
vision-language tasks, with less than 15% consisting of language-only tasks.
While the majority of vision-language tasks were trained with a text-only
1-shot setup (without the corresponding image), the CBIS-DDSM classification
and genomic variant calling tasks were trained with a 0-shot setup.

Table A.1: Med-PaLM M data mixture. Summary of the task types, modalities, mixture ratios, and few-shot setups in Med-PaLM M training data mixture. Task | Modality | Dataset | Mixture ratio | Few-shot setup
---|---|---|---|---
Question Answering | Text | MedQA | 3.13% | 2-shot
MedMCQA | 6.25% | 2-shot
Report Summarization | Radiology | MIMIC-III | 3.13% | 0-shot
Visual Question Answering | Radiology | VQA-RAD | 0.15% | text-only 1-shot
Slake-VQA | 2.64% | text-only 1-shot
Pathology | Path-VQA | 1.90% | text-only 1-shot
Report Generation | Chest X-ray | MIMIC-CXR | 59.90% | text-only 1-shot
Medical Image Classification | Dermatology | PAD-UFES-20 | 6.25% | text-only 1-shot
Mammography | VinDr-Mammo | 1.56% | text-only 1-shot
CBIS-DDSM | 1.56% | 0-shot
Chest X-ray | MIMIC-CXR | 11.98% | text-only 1-shot
| Genomics
---
| PrecisionFDA
---
Truth Challenge V2 [89]
1.56% | 0-shot

###  A.2.2 Training hyperparameters

PaLM-E projects the multimodal inputs into the same language embedding space
as latent vectors such that continuous observations (e.g., images, time
series) can be processed the same way by a pre-trained LLM as the language
tokens, and thereby generates textual completions autoregressively given a
multimodal prompt. In our experiments, the ViT maps the visual input to a
fixed number of 256 tokens which are further processed by the LLM along with
the additional text/multimodal tokens [10]. Med-PaLM M was finetuned on the
pretrained PaLM-E checkpoints. Table A.2 shows the training hyperparameters
for Med-PaLM M 12B, 84B, and 562B, respectively.

Table A.2: Med-PaLM M finetuning hyperparameters. Summary of the finetuning hyperparameters for Med-PaLM M 12B, 84B, and 562B. Hyperparameter | Med-PaLM M (12B) | Med-PaLM M (84B) | Med-PaLM M (562B)
---|---|---|---
Learning rate | $5\times 10^{-5}$ | $5\times 10^{-5}$ | $2.5\times 10^{-5}$
Batch size | 128 | 128 | 256
Max token input length | 710 | 710 | 710
Max token output length | 256 | 256 | 256

##  A.3 Detailed Med-PaLM M Performance

##### Performance on text-only medical question answering

We report the few-shot performance of Med-PaLM M on MedQA, MedMCQA, and
PubMedQA in Table A.3. SOTA results were chosen from Med-PaLM 2 with ensemble
refinement prompting and PaLM 540B few-shot results reported in [9, 61]. Med-
PaLM M outperformed the baseline PaLM model (from which it inherits) by a
large margin on all three datasets, despite falling behind the Med-PaLM 2 best
results obtained with ensemble refinement. Scaling up the language model from
8B to 540B significantly improves the accuracy on the multiple-choice medical
question answering tasks, where strong capabilities to comprehend, recall, and
reason about medical knowledge are important. These results can be partly
explained by the improved base language model used for Med-PaLM 2.

Table A.3: Language-only medical question answering accuracy on MultiMedQA. Med-PaLM 2 results with ensemble refinement [61] and PaLM few-shot results [9] are presented for comparison. Few-shot Med-PaLM M outperforms the corresponding PaLM baseline by a large margin, despite falling short of the state-of-the-art Med-PaLM 2. Dataset | Med-PaLM 2 | PaLM | Med-PaLM M (12B) | Med-PaLM M (84B) | Med-PaLM M (562B)
---|---|---|---|---|---
MedQA (USMLE) | 86.50% | 58.90% | 29.22% | 46.11% | 69.68%
MedMCQA | 72.30% | 54.50% | 32.20% | 47.60% | 62.59%
PubMedQA | 81.80% | 55.00% | 48.60% | 71.40% | 80.00%

##### Performance on radiology report summarization

We report commonly used metrics such as ROUGE-L [100], BLEU [101], and
F1-RadGraph [102] scores on the radiology report summarization task as in [62]
in Table A.4. Med-PaLM M (562B) yielded the best overall performance compared
to the smaller model variants, consistent with our observations on medical
question answering tasks. Med-PaLM M performed worse than the SOTA results
which were obtained with a parameter-efficient finetuning method (low-rank
adaptation, LoRA [103]) on a 738M-parameter clinical-T5 model [104]. However,
as noted in [62], one caveat of clinical-T5 is that it is unclear if [104]
pretrained the model on the test set of MIMIC-III which led to potential data
leakage. Notably, Med-PaLM M compared favorably to the results in [62] based
on the T5 model which was not pretrained on clinical text, similar to the PaLM
model.

Table A.4: Med-PaLM M performance on MIMIC-III radiology report summarization. Dataset | Metric | SOTA | Med-PaLM M (12B) | Med-PaLM M (84B) | Med-PaLM M (562B)
---|---|---|---|---|---
MIMIC-III | ROUGE-L |  38.70% | 29.45% | 31.47% | 32.03%
BLEU |  16.20% | 12.14% | 15.36% | 15.21%
F1-RadGraph |  40.80% | 31.43% | 33.96% | 34.71%

##### Performance on medical image classification tasks

Table A.5 shows the performance of Med-PaLM M on a set of classification tasks
across multiple modalities including dermatology, radiology, and genomics.
Since these tasks all have imbalanced class distributions, we reported macro-
AUC (unweighted mean of all the per-class AUC scores) and macro-F1 scores
(unweighted mean of all the per-class F1 scores) as the classification metrics
except for the genomic variant calling task where the F1 scores for single
nucleotide polymorphisms (SNPs) and short insertions and deletions (indels) in
the context of variant discovery were used instead.

On VinDr-Mammo, all size variants of Med-PaLM M exceeded the prior SOTA using
a smaller ViT (9.7M) on macro-AUC [49]. On CBIS-DDSM, our model achieved the
best macro-F1 of 51.12% and 67.86% on the mass and calcification
classification, respectively, behind the SOTA F1 of 70.71% reported on the
calcification classification [70]. Note that most previous works on CBIS-DDSM
focused on a two-class patch-level classification (benign versus malignant)
problem in contrast to our 3-class setup as discussed in [105]. On Pad-
UFES-20, since no official train/test splits are available, our results are
not directly comparable with prior studies. Med-PaLM M 84B achieved a macro-
AUC of 97.27%, on par with previous reported results (94% - 98%) obtained
using CNN and ViT variants [106, 107]. On MIMIC-CXR, we reported the macro-
average of F1 scores across the binary classification of 5 major conditions:
atelectasis, cardiomegaly, consolidation, edema, and pleural effusion. Med-
PaLM M (562B) achieved a macro-AUC of 79.09%, slightly lower than the SOTA
result of 81.27% obtained from ParallelXNet [69], which used a parallelization
of various CNN Architectures. On the variant calling task, DeepVariant model
[71] outperformed Med-PaLM M on both Indel-F1 and SNP-F1 scores. The SOTA
DeepVariant model was trained with 2,633-fold more training examples. Training
with the same examples resulted in a narrower advantage for DeepVariant for
SNP (Med-PaLM M 99.35% versus DeepVariant 99.63%) and Indel (Med-PaLM M 97.04%
versus DeepVariant 98.55%. Notably, Med-PaLM M outperformed the accuracy of
the widely used GATK4 method [90] for SNP calling (Med-PaLM M 99.35% versus
GATK4 99.29%) but not Indel calling (Med-PaLM M 97.04% versus GATK4 99.32%).

Taken together, Med-PaLM M achieved competitive results on a variety of
classification tasks using a single model compared to highly specialized SOTA
models. It is worth noting that we did not perform any fine-grained task-
specific customization and hyperparameter tuning beyond data augmentation and
class balancing. It is expected that scaling up the language model does not
significantly benefit the classification tasks where the vision encoder is
likely the bottleneck for the model performance. There is no overall evidence
to suggest that larger vision model outperforms the small one across all our
experiments, suggesting that more domain-specific pretraining may be more
important for improving vision encoder performance. It is also likely that
relatively small-scale datasets we explored here are not sufficient to
establish such a robust scaling relationship between the model size and task
performance, as results were generally close to each other across model
scales.

Table A.5: Med-PaLM M performance on medical image classification. We report macro-averaged AUC and F1 for all tasks. For MIMIC-CXR, metrics are averaged over 5 major pathological conditions. Dataset | # Classes | Metric | SOTA |  | Med-PaLM M
---
(12B)
| Med-PaLM M
---
(84B)
| Med-PaLM M
---
(562B)
MIMIC-CXR (5 conditions) | 2-class | Macro-AUC | 81.27% | 76.67% | 78.35% | 79.09%
Macro-F1 | N/A | 38.33% | 36.83% | 41.57%
PAD-UFES-20 | 6-class | Macro-AUC | N/A | 95.57% | 97.27% | 96.08%
Macro-F1 | N/A | 78.42% | 84.32% | 77.03%
Variant Calling | 3-class | Indel-F1 |  99.40% | 96.42% | 97.04% | 95.46%
SNP-F1 |  99.70% | 99.35% | 99.32% | 99.16%
VinDr-Mammo | 5-class | Macro-AUC | 64.50% | 66.29% | 71.76% | 71.42%
Macro-F1 | N/A | 29.81% | 35.7% | 33.90%
CBIS-DDSM (mass) | 3-class | Macro-AUC | N/A | 70.11% | 73.09% | 73.31%
Macro-F1 | N/A | 47.23% | 49.98% | 51.12%
CBIS-DDSM (calcification) | 3-class | Macro-AUC | N/A | 81.40% | 82.22% | 80.90%
Macro-F1 | 70.71% | 67.86% | 63.81% | 63.03%

##### Performance on medical visual question answering

Since we formulated both close-end and open-end QA pairs in three VQA datasets
as an open-ended language decoding task conditioned on visual input, we used
BLEU-1 and token-level F1 scores to assess the performance of Med-PaLM M. This
is in contrast with many prior works which used a string-level accuracy
evaluation metric as they often considered VQA as a classification task on a
set of pre-defined fixed-number answer candidates [108, 109]. This accuracy
metric has the weakness of failing to capture "near misses" of groundtruth
answers, particularly in our open-ended generative setup. We also noted that
only human validation by experts can provide additional insights on the
quality of model answers beyond token-level or string-level matching metrics.
As shown in Table A.6, Med-PaLM M surpassed previous SOTA using a similar
generative approach across all three datasets and metrics [64, 63]. In
particular, model performance increased with scaling up the language model on
VQA-RAD and Path-VQA. On Slake-VQA, the best performance was achieved with the
medium size model variant. These results suggest that scaling up language
models is beneficial for visual-language tasks where language reasoning is
conditioned on visual understanding.

Table A.6: Med-PaLM M performance on medical visual question answering. Med-PaLM exceeds prior SOTA on all three VQA tasks. Dataset | Metric | SOTA | Med-PaLM M (12B) | Med-PaLM M (84B) | Med-PaLM M (562B)
---|---|---|---|---|---
VQA-RAD | BLEU-1 | 71.03% | 64.02% | 69.38% | 71.27%
F1 | N/A | 50.66% | 59.90% | 62.06%
Path-VQA | BLEU-1 | 70.30% | 68.97% | 70.16% | 72.27%
F1 | 58.40% | 57.24% | 59.51% | 62.69%
Slake-VQA | BLEU-1 | 78.60% | 90.77% | 92.7% | 91.64%
F1 | 78.10% | 86.22% | 89.28% | 87.50%

##### Performance on chest X-ray report generation

To measure the quality of generated chest X-ray reports using automatic
metrics, we computed common natural language generation metrics such as
BLEU-1, BLEU-4, ROUGE-L, CIDEr-D [110], in addition to the clinical efficacy
(CE) metrics and F1-RadGraph which were designed to capture the factuality and
diagnostic accuracy in the generated reports. Specifically, we used CheXbert
[111], an automatic radiology report labeller based on a BERT model improved
with expert annotation, to extract the 14 CheXpert pathological observations
from a given report. For each observation, the predicted label was compared
against the groundtruth label to compute CE metrics. F1-RadGraph generalizes
over CheXbert labeller to more observation categories by measuring the
overlapping clinical entities and relations between a generated report and the
reference report [60]. In line with previous studies [112, 99, 66, 65, 67, 68,
14], we reported the macro-F1 and micro-F1 scores averaged over 5 major
observations and all 14 observations for CE metrics, respectively. As shown in
Table A.7, Med-PaLM M achieved a new SOTA on all CE metrics and F1-RadGraph,
with a substantial increase of about 9 points on macro-F1-14 and micro-F1-14
averaged across all clinical relevant observations over previous best SOTA
results in [65, 14]. The macro-average F1 resulted in a lower score than the
micro-average F1 over 14 observation categories because of the worse model
performance on some categories with very low representation in the training
data. Notably, improvements on F1 scores were more prominent across all 14
categories than over the 5 major categories for Med-PaLM M. This is likely due
to the benefit of jointly training with the classification tasks on those
minority conditions. We consider such positive task transfer as one of the
main advantages of a generalist multi-task model over a specialized single-
task model. On text overlap based natural language generation metrics, Med-
PaLM M did not outperform existing SOTA results. However, the pitfalls of
these automatic metrics have been raised by many studies, particularly in that
they fail to capture the factual correctness and do not align well with
radiologist judgements [112, 66, 60, 68].

Interestingly, our largest model Med-PaLM M (562B) did not achieve the best
performance, falling slightly short of Med-PaLM M (84B). Furthermore, the gap
in performance across three model sizes is relatively small across all types
of metrics. The diminishing return of increasing the size of the language
model is likely because the output space for chest X-ray report generation is
fairly confined to a set of template sentences and limited number of
conditions. It is also possible that the task performance is primarily limited
by the vision encoder, particularly in how well it is adapted for this domain.
As noted by [113], ViT lacks inductive bias for modeling local visual features
which are often crucial for interpreting medical images. To overcome this
limitation, large-scale medical training data may be required to enable
benefit from size scaling. Additionally, the input image size $224\times
224\times 3$ we used cause loss in resolution, which is a tradeoff we made to
shorten the length of embedded image tokens to fit within the context length
limit of the language model.

Table A.7: Med-PaLM M performance on chest X-ray report generation. We use text-overlap based and clinical factuality based automatic metrics to evaluate the quality of model-generated reports. Med-PaLM M sets new SOTA on all metrics designed to capture clinical efficacy and correctness. Across three Med-PaLM M variants, the medium-sized model achieves the best performance. Metric | SOTA | Med-PaLM M (12B) | Med-PaLM M (84B) | Med-PaLM M (562B)
---|---|---|---|---
Micro-F1-14 | 44.20% | 51.41% | 53.56% | 51.60%
Macro-F1-14 | 30.70% | 37.31% | 39.83% | 37.81%
Micro-F1-5 | 56.70% | 56.54% | 57.88% | 56.28%
Macro-F1-5 | N/A | 50.57% | 51.60% | 49.86%
F1-RadGraph | 24.40% | 25.20% | 26.71% | 26.06%
BLEU-1 | 39.48% | 30.90% | 32.31% | 31.73%
BLEU-4 | 13.30% | 10.43% | 11.31% | 11.50%
ROUGE-L | 29.60% | 26.16% | 27.29% | 27.49%
CIDEr-D | 49.50% | 23.43% | 26.17% | 25.27%

##  A.4 Details on Human Evaluation

Figures A.3 and A.4 depict the task interfaces used for side-by-side and
independent radiologist evaluations, including the task input and annotation
prompts presented to radiologist raters. For ease of a detailed inspection
(e.g., identification of subtle structures), the built-in medical image viewer
provided tools for raters to adjust the chest X-ray image, including zoom,
gamma, and blend controls.