# Towards Expert-Level
Medical Question Answering
with Large Language Models

Karan Singhal  Google Research,  Tao Tu  Google Research,  Juraj Gottweis
Google Research,  Rory Sayres  Google Research,
Ellery Wulczyn  Google Research,  Le Hou  Google Research,  Kevin Clark
Google Research,  Stephen Pfohl  Google Research,  Heather Cole-Lewis  Google
Research,  Darlene Neal  Google Research,
Mike Schaekermann  Google Research,  Amy Wang  Google Research,  Mohamed Amin
Google Research,  Sami Lachgar  Google Research,
Philip Mansfield  Google Research,  Sushant Prakash  Google Research,  Bradley
Green  Google Research,  Ewa Dominowska  Google Research,  Blaise Aguera y
Arcas  Google Research,  Nenad Tomasev  DeepMind,  Yun Liu  Google Research,
Renee Wong  Google Research,  Christopher Semturs  Google Research,  S. Sara
Mahdavi  Google Research,
Joelle Barral  Google Research,  Dale Webster  Google Research,  Greg S.
Corrado  Google Research,  Yossi Matias  Google Research,
Shekoofeh Azizi  Google Research,  Alan Karthikesalingam  Google Research,
Vivek Natarajan  Google Research,

###### Abstract

Recent artificial intelligence (AI) systems have reached milestones in “grand
challenges” ranging from Go to protein-folding. The capability to retrieve
medical knowledge, reason over it, and answer medical questions comparably to
physicians has long been viewed as one such grand challenge.

Large language models (LLMs) have catalyzed significant progress in medical
question answering; Med-PaLM was the first model to exceed a “passing” score
in US Medical Licensing Examination (USMLE) style questions with a score of
67.2% on the MedQA dataset. However, this and other prior work suggested
significant room for improvement, especially when models’ answers were
compared to clinicians’ answers. Here we present Med-PaLM 2, which bridges
these gaps by leveraging a combination of base LLM improvements (PaLM 2),
medical domain finetuning, and prompting strategies including a novel ensemble
refinement approach.

Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM by
over 19% and setting a new state-of-the-art. We also observed performance
approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU
clinical topics datasets.

We performed detailed human evaluations on long-form questions along multiple
axes relevant to clinical applications. In pairwise comparative ranking of
1066 consumer medical questions, physicians preferred Med-PaLM 2 answers to
those produced by physicians on eight of nine axes pertaining to clinical
utility ($p$ < 0.001). We also observed significant improvements compared to
Med-PaLM on every evaluation axis ($p$ < 0.001) on newly introduced datasets
of 240 long-form “adversarial” questions to probe LLM limitations.

While further studies are necessary to validate the efficacy of these models
in real-world settings, these results highlight rapid progress towards
physician-level performance in medical question answering.

##  1 Introduction

Figure 1: Med-PaLM 2 performance on MultiMedQA Left: Med-PaLM 2 achieved an
accuracy of 86.5% on USMLE-style questions in the MedQA dataset. Right: In a
pairwise ranking study on 1066 consumer medical questions, Med-PaLM 2 answers
were preferred over physician answers by a panel of physicians across eight of
nine axes in our evaluation framework.

Language is at the heart of health and medicine, underpinning interactions
between people and care providers. Progress in Large Language Models (LLMs)
has enabled the exploration of medical-domain capabilities in artificial
intelligence (AI) systems that can understand and communicate using language,
promising richer human-AI interaction and collaboration. In particular, these
models have demonstrated impressive capabilities on multiple-choice research
benchmarks [53, 2, 3].

In our prior work on Med-PaLM, we demonstrated the importance of a
comprehensive benchmark for medical question-answering, human evaluation of
model answers, and alignment strategies in the medical domain [53]. We
introduced MultiMedQA, a diverse benchmark for medical question-answering
spanning medical exams, consumer health, and medical research. We proposed a
human evaluation rubric enabling physicians and lay-people to perform detailed
assessment of model answers. Our initial model, Flan-PaLM, was the first to
exceed the commonly quoted passmark on the MedQA dataset comprising questions
in the style of the US Medical Licensing Exam (USMLE). However, human
evaluation revealed that further work was needed to ensure the AI output,
including long-form answers to open-ended questions, are safe and aligned with
human values and expectations in this safety-critical domain (a process
generally referred to as "alignment"). To bridge this, we leveraged
instruction prompt-tuning to develop Med-PaLM, resulting in substantially
improved physician evaluations over Flan-PaLM. However, there remained key
shortfalls in the quality of model answers compared to physicians. Similarly,
although Med-PaLM achieved state-of-the-art on every multiple-choice benchmark
in MultiMedQA, these scores left room for improvement.

Here, we bridge these gaps and further advance LLM capabilities in medicine
with Med-PaLM 2. We developed this model using a combination of an improved
base LLM (PaLM 2 [4]), medical domain-specific finetuning and a novel
prompting strategy that enabled improved medical reasoning. Med-PaLM 2
improves upon Med-PaLM by over 19% on MedQA as depicted in Figure 1 (left).
The model also approached or exceeded state-of-the-art performance on MedMCQA,
PubMedQA, and MMLU clinical topics datasets.

While these benchmarks are a useful measure of the knowledge encoded in LLMs,
they do not capture the model’s ability to generate factual, safe responses to
questions that require nuanced answers, typical in real-world medical
question-answering. We study this by applying our previously published rubric
for evaluation by physicians and lay-people [53]. Further, we introduce two
additional human evaluations: first, a pairwise ranking evaluation of model
and physician answers to consumer medical questions along nine clinically
relevant axes; second, a physician assessment of model responses on two newly
introduced adversarial testing datasets designed to probe the limits of LLMs.

Our key contributions are summarized as follows:

  * •

We developed Med-PaLM 2, a new medical LLM trained using a new base model
(PaLM 2 [4]) and targeted medical domain-specific finetuning (Section 3.2).

  * •

We introduced _ensemble refinement_ as a new prompting strategy to improve LLM
reasoning (Section 3.3).

  * •

Med-PaLM 2 achieved state-of-the-art results on several MultiMedQA benchmarks,
including MedQA USMLE-style questions (Section 4.1).

  * •

Human evaluation of long-form answers to consumer medical questions showed
that Med-PaLM 2’s answers were preferred to physician and Med-PaLM answers
across eight of nine axes relevant to clinical utility, such as factuality,
medical reasoning capability, and low likelihood of harm. For example, Med-
PaLM 2 answers were judged to better reflect medical consensus 72.9% of the
time compared to physician answers (Sections 4.2 and 1).

  * •

Finally, we introduced two adversarial question datasets to probe the safety
and limitations of these models. We found that Med-PaLM 2 performed
significantly better than Med-PaLM across every axis, further reinforcing the
importance of comprehensive evaluation. For instance, answers were rated as
having low risk of harm for 90.6% of Med-PaLM 2 answers, compared to 79.4% for
Med-PaLM. (Sections 4.2, 5 and LABEL:tab-sup:3).

##  2 Related Work

The advent of transformers [5] and large language models (LLMs) [6, 7] has
renewed interest in the possibilities of AI for medical question-answering
tasks–a long-standing “grand challenge” [8, 9, 10]. A majority of these
approaches involve smaller language models trained using domain specific data
(BioLinkBert [11], DRAGON [12], PubMedGPT [13], PubMedBERT [14], BioGPT [15]),
resulting in a steady improvement in state-of-the-art performance on benchmark
datasets such as MedQA (USMLE) [54], MedMCQA [55], and PubMedQA [56].

However, with the rise of larger general-purpose LLMs such as GPT-3 [19] and
Flan-PaLM [20, 21] trained on internet-scale corpora with massive compute, we
have seen leapfrog improvements on such benchmarks, all in a span of a few
months (Figure 1). In particular, GPT 3.5 [3] reached an accuracy of 60.2% on
the MedQA (USMLE) dataset, Flan-PaLM reached an accuracy of 67.6%, and
GPT-4-base [2] achieved 86.1%.

In parallel, API access to the GPT family of models has spurred several
studies evaluating the specialized clinical knowledge in these models, without
specific alignment to the medical domain. [22] evaluated the diagnostic and
triage accuracies of GPT-3 for 48 validated case vignettes of both common and
severe conditions and compared to lay-people and physicians. GPT-3’s
diagnostic ability was found to be better than lay-people and close to
physicians. On triage, the performance was less impressive and closer to lay-
people. On a similar note, [23], [24], and [25] studied GPT-3 performance in
genetics, surgery, and ophthalmology, respectively. More recently, Ayers et
al. [26] compared ChatGPT and physician responses on 195 randomly drawn
patient questions from a social media forum and found ChatGPT responses to be
rated higher in both quality and empathy.

With Med-PaLM and Med-PaLM 2, we take a “best of both worlds” approach: we
harness the strong out-of-the-box potential of the latest general-purpose LLMs
and then use publicly available medical question-answering data and physician-
written responses to align the model to the safety-critical requirements of
the medical domain. We introduce the ensemble refinement prompting strategy to
improve the reasoning capabilities of the LLM. This approach is closely
related to self-consistency [27], recitation-augmentation [28], self-refine
[29], and dialogue enabled reasoning [30]. It involves contextualizing model
responses by conditioning on multiple reasoning paths generated by the same
model in a prior step as described further in Section 3.3.

In this work, we not only evaluate our model on multiple-choice medical
benchmarks but also provide a rubric for how physicians and lay-people can
rigorously assess multiple nuanced aspects of the model’s long-form answers to
medical questions with independent and pairwise evaluation. This approach
allows us to develop and evaluate models more holistically in anticipation of
future real-world use.

##  3 Methods

Table 1: Multiple-choice question evaluation datasets. Name | Count | Description
---|---|---
MedQA (USMLE) | 1273 | General medical knowledge in US medical licensing exam
PubMedQA | 500 | Closed-domain question answering given PubMed abstract
MedMCQA | 4183 | General medical knowledge in Indian medical entrance exams
MMLU-Clinical knowledge | 265 | Clinical knowledge multiple-choice questions
MMLU Medical genetics | 100 | Medical genetics multiple-choice questions
MMLU-Anatomy | 135 | Anatomy multiple-choice questions
MMLU-Professional medicine | 272 | Professional medicine multiple-choice questions
MMLU-College biology | 144 | College biology multiple-choice questions
MMLU-College medicine | 173 | College medicine multiple-choice questions

###  3.1 Datasets

We evaluated Med-PaLM 2 on multiple-choice and long-form medical question-
answering datasets from MultiMedQA [53] and two new adversarial long-form
datasets introduced below.

##### Multiple-choice questions

For evaluation on multiple-choice questions, we used the MedQA [54], MedMCQA
[55], PubMedQA [56] and MMLU clinical topics [57] datasets (Table 1).

##### Long-form questions

For evaluation on long-form questions, we used two sets of questions sampled
from MultiMedQA (Table 2). The first set (MultiMedQA 140) consists of 140
questions curated from the HealthSearchQA, LiveQA [32], MedicationQA [33]
datasets, matching the set used by [53]. The second set (MultiMedQA 1066), is
an expanded sample of 1066 questions sampled from the same sources.

##### Adversarial questions

We also curated two new datasets of adversarial questions designed to elicit
model answers with potential for harm and bias: a general adversarial set and
health equity focused adversarial set (Table 2). The first set (Adversarial -
General) broadly covers issues related to health equity, drug use, alcohol,
mental health, COVID-19, obesity, suicide, and medical misinformation. Health
equity topics covered in this dataset include health disparities, the effects
of structural and social determinants on health outcomes, and racial bias in
clinical calculators for renal function [34, 35, 36]. The second set
(Adversarial - Health equity) prioritizes use cases, health topics, and
sensitive characteristics based on relevance to health equity considerations
in the domains of healthcare access (e.g., health insurance, access to
hospitals or primary care provider), quality (e.g., patient experiences,
hospital care and coordination), and social and environmental factors (e.g.,
working and living conditions, food access, and transportation). The dataset
was curated to draw on insights from literature on health equity in AI/ML and
define a set of implicit and explicit adversarial queries that cover a range
of patient experiences and health conditions [37, 38, 39, 40, 41].

Table 2: Long-form question evaluation datasets. Name | Count | Description
---|---|---
MultiMedQA 140 | 140 |  |  Sample from HealthSearchQA, LiveQA, MedicationQA [53]
---
MultiMedQA 1066 | 1066 |  |  Sample from HealthSearchQA, LiveQA, MedicationQA (Extended from [53])
---
Adversarial (General) | 58 | General adversarial dataset
Adversarial (Health equity) | 182 | Health equity adversarial dataset

###  3.2 Modeling

##### Base LLM

For Med-PaLM, the base LLM was PaLM [20]. Med-PaLM 2 builds upon PaLM 2 [4], a
new iteration of Google’s large language model with substantial performance
improvements on multiple LLM benchmark tasks.

##### Instruction finetuning

We applied instruction finetuning to the base LLM following the protocol used
by [21]. The datasets used included the training splits of MultiMedQA–namely
MedQA, MedMCQA, HealthSearchQA, LiveQA and MedicationQA. We trained a
“unified” model, which is optimized for performance across all datasets in
MultiMedQA using dataset mixture ratios (proportions of each dataset) reported
in Table 3. These mixture ratios and the inclusion of these particular
datasets were empirically determined. Unless otherwise specified, Med-PaLM 2
refers to this unified model. For comparison purposes, we also created a
variant of Med-PaLM 2 obtained by finetuning exclusively on multiple-choice
questions which led to improved results on these benchmarks.

Table 3: Instruction finetuning data mixture. Summary of the number of training examples and percent representation in the data mixture for the different MultiMedQA datasets used for instruction finetuning of the unified Med-PaLM 2 model. Dataset |  Count |  Mixture ratio
---|---|---
MedQA | 10,178 | 37.5%
MedMCQA | 182,822 | 37.5%
LiveQA | 10 | 3.9%
MedicationQA | 9 | 3.5%
HealthSearchQA | 45 | 17.6%

###  3.3 Multiple-choice evaluation

We describe below prompting strategies used to evaluate Med-PaLM 2 on
multiple-choice benchmarks.

##### Few-shot prompting

Few-shot prompting [19] involves prompting an LLM by prepending example inputs
and outputs before the final input. Few-shot prompting remains a strong
baseline for prompting LLMs, which we evaluate and build on in this work. We
use the same few-shot prompts as used by [53].

##### Chain-of-thought

Chain-of-thought (CoT), introduced by [52], involves augmenting each few-shot
example in a prompt with a step-by-step explanation towards the final answer.
The approach enables an LLM to condition on its own intermediate outputs in
multi-step problems. As noted in [53], the medical questions explored in this
study often involve complex multi-step reasoning, making them a good fit for
CoT prompting. We crafted CoT prompts to provide clear demonstrations on how
to appropriately answer the given medical questions (provided in LABEL:sec-
app:chain-of-thought-prompts).

##### Self-consistency

Self-consistency (SC) is a strategy introduced by [43] to improve performance
on multiple-choice benchmarks by sampling multiple explanations and answers
from the model. The final answer is the one with the majority (or plurality)
vote. For a domain such as medicine with complex reasoning paths, there might
be multiple potential routes to the correct answer. Marginalizing over the
reasoning paths can lead to the most accurate answer. The self-consistency
prompting strategy led to particularly strong improvements for [44]. In this
work, we performed self-consistency with 11 samplings using COT prompting, as
in [53].

##### Ensemble refinement

Building on chain-of-thought and self-consistency, we developed a simple
prompting strategy we refer to as ensemble refinement (ER). ER builds on other
techniques that involve conditioning an LLM on its own generations before
producing a final answer, including chain-of-thought prompting and self-Refine
[29].

ER involves a two-stage process: first, given a (few-shot) chain-of-thought
prompt and a question, the model produces multiple possible generations
stochastically via temperature sampling. In this case, each generation
involves an explanation and an answer for a multiple-choice question. Then,
the model is conditioned on the original prompt, question, and the
concatenated generations from the previous step, and is prompted to produce a
refined explanation and answer. This can be interpreted as a generalization of
self-consistency, where the LLM is aggregating over answers from the first
stage instead of a simple vote, enabling the LLM to take into account the
strengths and weaknesses of the explanations it generated. Here, to improve
performance we perform the second stage multiple times, and then finally do a
plurality vote over these generated answers to determine the final answer.
Ensemble refinement is depicted in Figure 2.

Unlike self-consistency, ensemble refinement may be used to aggregate answers
beyond questions with a small set of possible answers (e.g., multiple-choice
questions). For example, ensemble refinement can be used to produce improved
long-form generations by having an LLM condition on multiple possible
responses to generate a refined final answer. Given the resource cost of
approaches requiring repeated samplings from a model, we apply ensemble
refinement only for multiple-choice evaluation in this work, with 11 samplings
for the first stage and 33 samplings for the second stage.

###  3.4 Overlap analysis

An increasingly important concern given recent advances in large models
pretrained on web-scale data is the potential for overlap between evaluation
benchmarks and training data. To evaluate the potential impact of test set
contamination on our evaluation results, we searched for overlapping text
segments between multiple-choice questions in MultiMedQA and the corpus used
to train the base LLM underlying Med-PaLM 2. Specifically, we defined a
question as overlapping if either the entire question or at least 512
contiguous characters overlap with any document in the training corpus. For
purposes of this analysis, multiple-choice options or answers were not
included as part of the query, since inclusion could lead to underestimation
of the number of overlapping questions due to heterogeneity in formatting and
ordering options. As a result, this analysis will also treat questions without
answers in the training data as overlapping. We believe this methodology is
both simple and conservative, and when possible we recommend it over blackbox
memorization testing techniques [2], which do not conclusively measure test
set contamination.

Figure 2: Illustration of Ensemble Refinement (ER) with Med-PaLM 2. In this
approach, an LLM is conditioned on multiple possible reasoning paths that it
generates to enable it to refine and improves its answer.

###  3.5 Long-form evaluation

To assess the performance of Med-PaLM 2 on long-form consumer medical
question-answering, we conducted a series of human evaluations.

##### Model answers

To elicit answers to long-form questions from Med-PaLM models, we used the
prompts provided in LABEL:sec-app:long-form-question-prompts. We did this
consistently across Med-PaLM and Med-PaLM 2. We sampled from models with
temperature 0.0 as in [53].

##### Physician answers

Physician answers were generated as described in [53]. Physicians were not
time-limited in generating answers and were permitted access to reference
materials. Physicians were instructed that the audience for their answers to
consumer health questions would be a lay-person of average reading
comprehension. Tasks were not anchored to a specific environmental context or
clinical scenario.

##### Physician and lay-person raters

Human evaluations were performed by physician and lay-person raters. Physician
raters were drawn from a pool of 15 individuals: six based in the US, four
based in the UK, and five based in India. Specialty expertise spanned family
medicine and general practice, internal medicine, cardiology, respiratory,
pediatrics and surgery. Although three physician raters had previously
generated physician answers to MultiMedQA questions in prior work [53], none
of the physician raters evaluated their own answers and eight to ten weeks
elapsed between the task of answer generation and answer evaluation. Lay-
person raters were drawn from a pool of six raters (four female, two male,
18-44 years old) based in India, all without a medical background. Lay-person
raters’ educational background breakdown was: two with high school diploma,
three with graduate degrees, one with postgraduate experience.

##### Individual evaluation of long-form answers

Individual long-form answers from physicians, Med-PaLM, and Med-PaLM 2 were
rated independently by physician and lay-person raters using rubrics
introduced in [53]. Raters were blinded to the source of the answer and
performed ratings in isolation without conferring with other raters.
Experiments were conducted using the MultiMedQA 140, Adversarial (General),
and Adversarial (Health equity) datasets. Ratings for MultiMedQA 140 for Med-
PaLM were taken from [53]. For all new rating experiments, each response was
evaluated by three independent raters randomly drawn from the respective pool
of raters (lay-person or physician). Answers in MultiMedQA 140 were triple-
rated, while answers to Adversarial questions were quadruple rated. Inter-
rater reliability analysis of MultiMedQA 140 answers indicated that raters
were in very good ($\kappa>0.8$) agreement for 10 out of 12 alignment
questions, and good ($\kappa>0.6$) agreement for the remaining two questions,
including whether answers misses important content, or contain unnecessary
additional information (LABEL:fig:inter-rater). Triplicate rating enabled
inter-rater reliability analyses shown in LABEL:sec-sup:inter-rater-
reliability.

##### Pairwise ranking evaluation of long-form answers

In addition to independent evaluation of each response, a pairwise preference
analysis was performed to directly rank preference between two alternative
answers to a given question. Raters were presented with a pair of answers from
different sources (e.g., physician vs Med-PaLM 2) for a given question. This
intuitively reduces inter-rater variability in ratings across questions.

For each pair of answers, raters were asked to select the preferred response
or indicate a tie along the following axes (with exact instruction text in
quotes):

  * •

Alignment with medical consensus: “Which answer better reflects the current
consensus of the scientific and clinical community?”

  * •

Reading comprehension: “Which answer demonstrates better reading
comprehension? (indication the question has been understood)”

  * •

Knowledge recall: “Which answer demonstrates better recall of knowledge?
(mention of a relevant and/or correct fact for answering the question)”

  * •

Reasoning: “Which answer demonstrates better reasoning step(s)? (correct
rationale or manipulation of knowledge for answering the question)”

  * •

Inclusion of irrelevant content: “Which answer contains more content that it
shouldn’t? (either because it is inaccurate or irrelevant)”

  * •

Omission of important information: “Which answer omits more important
information?”

  * •

Potential for demographic bias: “Which answer provides information that is
biased for any demographic groups? For example, is the answer applicable only
to patients of a particular sex where patients of another sex might require
different information?”

  * •

Possible harm extent: “Which answer has a greater severity/extent of possible
harm? (which answer could cause more severe harm)”

  * •

Possible harm likelihood: “Which answer has a greater likelihood of possible
harm? (more likely to cause harm)”

Note that for three of the axes (reading comprehension, knowledge recall, and
reasoning), the pairwise ranking evaluation differed from the long-form
individual answer evaluation. Specifically, in individual answer evaluation we
separately examine whether a response contains evidence of correctly and
incorrectly retrieved facts; the pairwise ranking evaluation consolidates
these two questions to understand which response is felt by raters to
demonstrate greater quality for this property in aggregate. These evaluations
were performed on the MultiMedQA 1066 and Adversarial dataset. Raters were
blinded as to the source of each answer, and the order in which answers were
shown was randomized. Due to technical issues in the display of answers,
raters were unable to review 8 / 1066 answers for the Med-PaLM 2 vs Physician
comparison, and 11 / 1066 answers for the Med-PaLM 2 vs Med-PaLM comparison;
these answers were excluded from analysis in Figures 1, 5, LABEL:tab-
sup:pairwise-mp2mp and LABEL:tab-sup:pairwise-mp2phys.

##### Statistical analyses

Confidence intervals were computed via bootstrapping (10,000 iterations). Two-
tailed permutation tests were used for hypothesis testing (10,000 iterations);
for multiple-rated answers, permutations were blocked by answer. For
statistical analysis on the MultiMedQA dataset, where Med-PaLM and physician
answers were single rated, Med-PaLM 2 ratings were randomly sub-sampled to one
rating per answer during bootstrapping and permutation testing.

##  4 Results

Table 4: Comparison of Med-PaLM 2 results to reported results from GPT-4. Med-PaLM 2 achieves state-of-the-art accuracy on several multiple-choice benchmarks and was first announced on March 14, 2023. GPT-4 results were released on March 20, 2023, and GPT-4-base (non-production) results were released on April 12, 2023 [2]. We include Flan-PaLM results from December 2022 for comparison [53]. ER stands for Ensemble Refinement. Best results are across prompting strategies. Dataset |  | Flan-PaLM
---
(best)
| Med-PaLM 2
---
(ER)
| Med-PaLM 2
---
(best)
| GPT-4
---
(5-shot)
| GPT-4-base
---
(5-shot)
MedQA (USMLE) | 67.6 | 85.4 | 86.5 | 81.4 | 86.1
PubMedQA | 79.0 | 75.0 | 81.8 | 75.2 | 80.4
MedMCQA | 57.6 | 72.3 | 72.3 | 72.4 | 73.7
MMLU Clinical knowledge | 80.4 | 88.7 | 88.7 | 86.4 | 88.7
MMLU Medical genetics | 75.0 | 92.0 | 92.0 | 92.0 | 97.0
MMLU Anatomy | 63.7 | 84.4 | 84.4 | 80.0 | 85.2
MMLU Professional medicine | 83.8 | 92.3 | 95.2 | 93.8 | 93.8
MMLU College biology | 88.9 | 95.8 | 95.8 | 95.1 | 97.2
MMLU College medicine | 76.3 | 83.2 | 83.2 | 76.9 | 80.9

###  4.1 Multiple-choice evaluation

Tables 4 and 5 summarize Med-PaLM 2 results on MultiMedQA multiple-choice
benchmarks. Unless specified otherwise, Med-PaLM 2 refers to the unified model
trained on the mixture in Table 3. We also include comparisons to GPT-4 [45,
2].

##### MedQA

Our unified Med-PaLM 2 model reaches an accuracy of 85.4% using ensemble
refinement (ER) as a prompting strategy. Our best result on this dataset is
86.5% obtained from a version of Med-PaLM 2 not aligned for consumer medical
question answering, but instead instruction finetuned only on MedQA, setting a
new state-of-art for MedQA performance.

##### MedMCQA

On MedMCQA, Med-PaLM 2 obtains a score of 72.3%, exceeding Flan-PaLM
performance by over 14% but slightly short of state-of-the-art (73.66 from
GPT-4-base [45]).

##### PubMedQA

On PubMedQA, Med-PaLM 2 obtains a score of 75.0%. This is below the state-of-
the-art performance (81.0 from BioGPT-Large [15]) and is likely because no
data was included for this dataset for instruction finetuning. However, after
further exploring prompting strategies for PubMedQA on the development set
(see LABEL:sec-sup:pubmedqa-extra-prompting), the unified model reached an
accuracy of 79.8% with a single run and 81.8% using self-consistency (11x).
The latter result is state-of-the-art, although we caution that PubMedQA’s
test set is small (500 examples), and remaining failures of Med-PaLM 2 and
other strong models appear to be largely attributable to label noise intrinsic
in the dataset (especially given human performance is 78.0% [56]).

##### MMLU clinical topics

On MMLU clinical topics, Med-PaLM 2 significantly improves over previously
reported results in Med-PaLM [53] and is the state-of-the-art on 3 out 6
topics, with GPT-4-base reporting better numbers in the other three. We note
that the test set for each of these topics is small, as reported in Table 1.

Interestingly, we see a drop in performance between GPT-4-base and the aligned
(production) GPT-4 model on these multiple-choice benchmarks (Table 4). Med-
PaLM 2, on the other hand, demonstrates strong performance on multiple-choice
benchmarks while being specifically aligned to the requirements of long-form
medical question answering. While multiple-choice benchmarks are a useful
measure of the knowledge encoded in these models, we believe human evaluations
of model answers along clinically relevant axes as detailed further in Section
4.2 are necessary to assess their utility in real-world clinical applications.

We also see in Table 5 that ensemble refinement improves on few-shot and self-
consistency prompting strategies in eliciting strong model performance across
these benchmarks.

Table 5: Med-PaLM 2 performance with different prompting strategies including few-shot, chain-of-thought (CoT), self-consistency (SC), and ensemble refinement (ER). Dataset |  | Med-PaLM 2
---
(5-shot)
| Med-PaLM 2
---
(COT+SC)
| Med-PaLM 2
---
(ER)
MedQA (USMLE) | 79.7 | 83.7 | 85.4
PubMedQA | 79.2 | 74.0 | 75.0
MedMCQA | 71.3 | 71.5 | 72.3
MMLU Clinical knowledge | 88.3 | 88.3 | 88.7
MMLU Medical genetics | 90.0 | 89.0 | 92.0
MMLU Anatomy | 77.8 | 80.0 | 84.4
MMLU Professional medicine | 95.2 | 93.4 | 92.3
MMLU College biology | 94.4 | 95.1 | 95.8
MMLU College medicine | 80.9 | 81.5 | 83.2

##### Overlap analysis

Using the methodology described in Section 3.4, overlap percentages ranged
from 0.9% for MedQA to 48.0% on MMLU Medical Genetics. Performance of Med-PaLM
2 was slightly higher on questions with overlap for 6 out of 9 datasets,
though the difference was only statistically significant for MedMCQA (accuracy
difference 4.6%, [1.3, 7.7]) due to the relatively small number of questions
with overlap in most datasets (Table 6). When we reduced the overlap segment
length from 512 to 120 characters (see Section 3.4), overlap percentages
increased (11.15% for MedQA to 56.00% on MMLU Medical Genetics), but
performance differences on questions with overlap were similar (LABEL:tab-
sup:1), and the difference was still statistically significant for just one
dataset. These results are similar to those observed by [20], who also saw
minimal performance difference from testing on overlapping data. A limitation
of this analysis is that we were not able to exhaustively identify the subset
of overlapping questions where the correct answer is also explicitly provided
due to heterogeneity in how correct answers can be presented across different
documents. Restricting the overlap analysis to questions with answers would
reduce the overlap percentages while perhaps leading to larger observed
performance differences.

Table 6: Med-PaLM 2 performance on multiple-choice questions with and without overlap. We define a question as overlapping if either the entire question or up to 512 characters overlap with any document in the training corpus of the LLM underlying Med-PaLM 2. Dataset | Overlap Fraction |  | Performance
---
(without Overlap)
| Performance
---
(with Overlap)
Delta
MedQA (USMLE) |  | 12/1273
---
(0.9%)
| 85.3
---
[83.4, 87.3]
| 91.7
---
[76.0, 100.0]
| -6.3
---
[-13.5, 20.8]
PubMedQA |  | 6/500
---
(1.2%)
| 74.1
---
[70.2, 78.0]
| 66.7
---
[28.9, 100.0]
| 7.4
---
[-16.6, 44.3]
MedMCQA |  | 893/4183
---
(21.4%)
| 70.5
---
[68.9, 72.0]
| 75.0
---
[72.2, 77.9]
| -4.6
---
[-7.7, -1.3]
MMLU Clinical knowledge |  | 55/265
---
(20.8%)
| 88.6
---
[84.3, 92.9]
| 87.3
---
[78.5, 96.1]
| 1.3
---
[-6.8, 13.2]
MMLU Medical genetics |  | 48/100
---
(48.0%)
| 92.3
---
[85.1, 99.6]
| 91.7
---
[83.8, 99.5]
| 0.6
---
[-11.0, 12.8]
MMLU Anatomy |  | 37/135
---
(27.4%)
| 82.7
---
[75.2, 90.1]
| 89.2
---
[79.2, 99.2]
| -6.5
---
[-17.4, 8.7]
MMLU Professional medicine |  | 79/272
---
(29.0%)
| 89.1
---
[84.7, 93.5]
| 92.4
---
[86.6, 98.2]
| -3.3
---
[-9.9, 5.5]
MMLU College biology |  | 60/144
---
(41.7%)
| 95.2
---
[90.7, 99.8]
| 96.7
---
[92.1, 100.0]
| -1.4
---
[-8.7, 7.1]
MMLU College medicine |  | 47/173
---
(27.2%)
| 78.6
---
[71.4, 85.7]
| 91.5
---
[83.5, 99.5]
| -12.9
---
[-22.4, 0.1]
Figure 3: Independent long-form evaluation with physician raters Values are
the proportion of ratings across answers where each axis was rated in the
highest-quality bin. (For instance, "Possible harm extent = No harm" reflects
the proportion of answers where the extent of possible harm was rated "No
harm".) Left: Independent evaluation of long-form answers from Med-PaLM, Med-
PaLM 2 and physicians on the MultiMedQA 140 dataset. Right: Independent
evaluation of long-form answers from Med-PaLM and Med-PaLM 2 on the combined
adversarial datasets (General and Health equity). Detailed breakdowns are
presented in LABEL:tab-sup:2 and LABEL:tab-sup:3. (*) designates 0.01 < $p$ <
0.05 between Med-PaLM and Med-PaLM 2.

###  4.2 Long-form evaluation

##### Independent evaluation

On the MultiMedQA 140 dataset, physicians rated Med-PaLM 2 answers as
generally comparable to physician-generated and Med-PaLM-generated answers
along the axes we evaluated (Figures 3 and LABEL:tab-sup:2). However, the
relative performance of each varied across the axes of alignment that we
explored, and the analysis was largely underpowered for the effect sizes
(differences) observed. This motivated the pairwise ranking analysis presented
below on an expanded sample (MultiMedQA 1066). The only significant
differences observed were in favor of Med-PaLM 2 over Med-PaLM ($p$ < 0.05)
for the following 3 axes: evidence of reasoning, incorrect knowledge recall,
and incorrect reasoning.

On the adversarial datasets, physicians rated Med-PaLM 2 answers as
significantly higher quality than Med-PaLM answers across all axes ($p$ <
0.001 for all axes, Figures 3 and LABEL:tab-sup:3). This pattern held for both
the general and health equity-focused subsets of the Adversarial dataset
(LABEL:tab-sup:3).

Figure 4: Independent evaluation of long-form answers with lay-person raters
Med-PaLM 2 answers were rated as more directly relevant and helpful than Med-
PaLM answers on the MultiMedQA 140 dataset.

Finally, lay-people rated Med-PaLM 2 answers to questions in the MultiMedQA
140 dataset as more helpful and relevant than Med-PaLM answers
($p\leavevmode\nobreak\ \leq\leavevmode\nobreak\ 0.002$ for both dimensions,
Figures 4 and LABEL:tab-sup:4).

Notably, Med-PaLM 2 answers were longer than Med-PaLM and physician answers
(LABEL:tab-sup:8). On MultiMedQA 140, for instance, the median answer length
for Med-PaLM 2 was 794 characters, compared to 565.5 for Med-PaLM and 337.5
for physicians. Answer lengths to adversarial questions tended to be longer in
general, with median answer length of 964 characters for Med-PaLM 2 and 518
characters for Med-PaLM, possibly reflecting the greater complexity of these
questions.

(a) MultiMedQA

(b) Adversarial Question Sets

Figure 5: Ranking comparison of long-form answers  Med-PaLM 2 answers are
consistently preferred over Med-PaLM answers by physician raters across all
ratings dimensions, in both MultiMedQA and Adversarial question sets. Each row
shows the distribution of side-by-side ratings for which either Med-PaLM 2
(yellow) or Med-PaLM (green)’s answer were preferred; gray shade indicates
cases rated as ties along a dimension. Error bars are binomial confidence
intervals for the Med-PaLM 2 and Med-PaLM selection rates. Detailed breakdowns
for adversarial questions are presented in Supplemental Table 3.

##### Pairwise ranking evaluation

Pairwise ranking evaluation more explicitly assessed the relative performance
of Med-PaLM 2, Med-PaLM, and physicians. This ranking evaluation was over an
expanded set, MultiMedQA 1066 and the Adversarial sets. Qualitative examples
and their rankings are included in LABEL:tab-sup:ranking-example-ratings and
LABEL:tab-sup:ranking-examples, respectively, to provide indicative examples
and insight.

On MultiMedQA, for eight of the nine axes, Med-PaLM 2 answers were more often
rated as being higher quality compared to physician answers (p < 0.001,
Figures 1 and LABEL:tab-sup:pairwise-mp2phys). For instance, they were more
often rated as better reflecting medical consensus, or indicating better
reading comprehension; and less often rated as omitting important information
or representing a risk of harm. However, for one of the axes, including
inaccurate or irrelevant information, Med-PaLM 2 answers were not as favorable
as physician answers. Med-PaLM 2 answers were rated as higher quality than
Med-PaLM axes on the same eight axes (Figures 5 and LABEL:tab-sup:pairwise-
mp2mp); Med-PaLM 2 answers were marked as having more inaccurate or irrelevant
information less often than Med-PaLM answers (18.4% Med-PaLM 2 vs. 21.5% Med-
PaLM), but the difference was not significant (p = 0.12, LABEL:tab-
sup:pairwise-mp2mp).

On Adversarial questions, Med-PaLM 2 was ranked more favorably than Med-PaLM
across every axis (Figure 5), often by substantial margins.

##  5 Discussion

We show that Med-PaLM 2 exhibits strong performance in both multiple-choice
and long-form medical question answering, including popular benchmarks and
challenging new adversarial datasets. We demonstrate performance approaching
or exceeding state-of-the-art on every MultiMedQA multiple-choice benchmark,
including MedQA, PubMedQA, MedMCQA, and MMLU clinical topics. We show
substantial gains in long-form answers over Med-PaLM, as assessed by
physicians and lay-people on multiple axes of quality and safety. Furthermore,
we observe that Med-PaLM 2 answers were preferred over physician-generated
answers in multiple axes of evaluation across both consumer medical questions
and adversarial questions.

As LLMs become increasingly proficient at structured tests of knowledge, it is
becoming more important to delineate and assess their capabilities along
clinically relevant dimensions [26, 22]. Our evaluation framework examines the
alignment of long-form model outputs to human expectations of high-quality
medical answers. Our use of adversarial question sets also enables explicit
study of LLM performance in difficult cases. The substantial improvements of
Med-PaLM 2 relative to Med-PaLM suggest that careful development and
evaluation of challenging question-answering tasks is needed to ensure robust
model performance.

Using a multi-dimensional evaluation framework lets us understand tradeoffs in
more detail. For instance, Med-PaLM 2 answers significantly improved
performance on “missing important content” (LABEL:tab-sup:2) and were longer
on average (LABEL:tab-sup:8) than Med-PaLM or physician answers. This may
provide benefits for many use cases, but may also impact tradeoffs such as
including unnecessary additional details vs. omitting important information.
The optimal length of an answer may depend upon additional context outside the
scope of a question. For instance, questions around whether a set of symptoms
are concerning depend upon a person’s medical history; in these cases, the
more appropriate response of an LLM may be to request more information, rather
than comprehensively listing all possible causes. Our evaluation did not
consider multi-turn dialogue [46], nor frameworks for active information
acquisition [47].

Our individual evaluation did not clearly distinguish performance of Med-PaLM
2 answers from physician-generated answers, motivating more granular
evaluation, including pairwise evaluation and adversarial evaluation. In
pairwise evaluation, we saw that Med-PaLM 2 answers were preferred over
physician answers along several axes pertaining to clinical utility such as
factuality, medical reasoning capability, and likelihood of harm. These
results indicate that as the field progress towards physician-level
performance, improved evaluation frameworks will be crucial for further
measuring progress.

##  6 Limitations

Given the broad and complex space of medical information needs, methods to
measure alignment of model outputs will need continued development. For
instance, additional dimensions to those we measure here are likely to be
important, such as the empathy conveyed by answers [26]. As we have previously
noted, our rating rubric is not a formally validated qualitative instrument,
although our observed inter-rater reliability was high (LABEL:fig:inter-
rater). Further research is required to develop the rigor of rubrics enabling
human evaluation of LLM performance in medical question answering.

Likewise, a robust understanding of how LLM outputs compare to physician
answers is a broad, highly significant question meriting much future work; the
results we report here represent one step in this research direction. For our
current study, physicians generating answers were prompted to provide useful
answers to lay-people but were not provided with specific clinical scenarios
or nuanced details of the communication requirements of their audience. While
this may be reflective of real-world performance for some settings, it is
preferable to ground evaluations in highly specific workflows and clinical
scenarios. We note that our results cannot be considered generalizable to
every medical question-answering setting and audience. Model answers are also
often longer than physician answers, which may contribute to improved
independent and pairwise evaluations, as suggested by other work [26]. The
instructions provided to physicians did not include examples of outputs
perceived as higher or lower quality in preference ranking, which might have
impacted our evaluation. Furthermore, we did not explicitly assess inter-rater
variation in preference rankings or explore how variation in preference
rankings might relate to the lived experience, expectations or assumptions of
our raters.

Physicians were also asked to only produce one answer per question, so this
provides a limited assessment of the range of possible physician-produced
answers. Future improvements to this methodology could provide a more explicit
clinical scenario with recipient and environmental context for answer
generation. It could also assess multiple possible physician answers to each
question, alongside inter-physician variation. Moreover, for a more principled
comparison of LLM answers to medical questions, the medical expertise, lived
experience and background, and specialization of physicians providing answers,
and evaluating those answers, should be more explicitly explored. It would
also be desirable to explore intra- and inter-physician variation in the
generation of answers under multiple scenarios as well as contextualize LLM
performance by comparison to the range of approaches that might be expected
among physicians.

Finally, the current evaluation with adversarial data is relatively limited in
scope and should not be interpreted as a comprehensive assessment of safety,
bias, and equity considerations. In future work, the adversarial data could be
systematically expanded to increase coverage of health equity topics and
facilitate disaggregated evaluation over sensitive characteristics [48, 49,
50] .

##  7 Conclusion

These results demonstrate the rapid progress LLMs are making towards
physician-level medical question answering. However, further work on
validation, safety and ethics is necessary as the technology finds broader
uptake in real-world applications. Careful and rigorous evaluation and
refinement of LLMs in different contexts for medical question-answering and
real world workflows will be needed to ensure this technology has a positive
impact on medicine and health.

#### Acknowledgments

This project was an extensive collaboration between many teams at Google
Research. We thank Michael Howell, Boris Babenko, and Naama Hammel for their
valuable insights and feedback during our research. We are also grateful to
Jeff Dean, James Manyika, Karen DeSalvo, Zoubin Ghahramani, David Fleet,
Douglas Eck, and Simon Kornblith for their support during the course of this
project. We also want to thank Brett Hatfield, SiWai Man, Sudhanshu Sharma,
Gary Parakkal, Gordon Turner, Jukka Zitting, Evan Rappaport, Dave Steiner,
Jonas Kemp, Jimmy Hu, Yuan Liu, Jonathan Krause, Kavita Kulkarni, Susan
Thomas, Kate Weber, Annisah Um’rani, Anna Iurchenko, Will Vaughan, Julie Wang,
Maggie Shiels, and Lauren Winer for their assistance.

## References

  * [1] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis and Stephen Pfohl  “Large Language Models Encode Clinical Knowledge”  In _arXiv preprint arXiv:2212.13138_ , 2022
  * [2] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan and Eric Horvitz  “Capabilities of gpt-4 on medical challenge problems”  In _arXiv preprint arXiv:2303.13375_ , 2023
  * [3] Valentin Liévin, Christoffer Egeberg Hother and Ole Winther  “Can large language models reason about medical questions?”  In _arXiv preprint arXiv:2207.08143_ , 2022
  * [4] Google  “PaLM 2 Technical Report”, https://ai.google/static/documents/palm2techreport.pdf, 2023
  * [5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “Bert: Pre-training of deep bidirectional transformers for language understanding”  In _arXiv preprint arXiv:1810.04805_ , 2018
  * [7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J Liu  “Exploring the limits of transfer learning with a unified text-to-text transformer”  In _The Journal of Machine Learning Research_ 21.1 JMLRORG, 2020, pp. 5485–5551
  * [8] Edward H Shortliffe  “Computer programs to support clinical decision making”  In _Jama_ 258.1 American Medical Association, 1987, pp. 61–66
  * [9] William B Schwartz  “Medicine and the computer: the promise and problems of change”  In _Use and impact of computers in clinical medicine_ Springer, 1987, pp. 321–335
  * [10] Daniel G Bobrow  “Categorical and probabilistic reasoning in medicine revisited”  MIT Press, 1994
  * [11] Michihiro Yasunaga, Jure Leskovec and Percy Liang  “LinkBERT: Pretraining Language Models with Document Links”  In _arXiv preprint arXiv:2203.15827_ , 2022
  * [12] Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy Liang and Jure Leskovec  “Deep bidirectional language-knowledge graph pretraining”  In _arXiv preprint arXiv:2210.09338_ , 2022
  * [13] Elliot Bolton, David Hall, Michihiro Yasunaga, Tony Lee, Chris Manning and Percy Liang  “Stanford CRFM Introduces PubMedGPT 2.7B”, https://hai.stanford.edu/news/stanford-crfm-introduces-pubmedgpt-27b, 2022
  * [14] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao and Hoifung Poon  “Domain-specific language model pretraining for biomedical natural language processing”  In _ACM Transactions on Computing for Healthcare (HEALTH)_ 3.1 ACM New York, NY, 2021, pp. 1–23
  * [15] Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon and Tie-Yan Liu  “BioGPT: generative pre-trained transformer for biomedical text generation and mining”  In _Briefings in Bioinformatics_ 23.6 Oxford Academic, 2022
  * [16] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang and Peter Szolovits  “What disease does this patient have? a large-scale open domain question answering dataset from medical exams”  In _Applied Sciences_ 11.14 MDPI, 2021, pp. 6421
  * [17] Ankit Pal, Logesh Kumar Umapathi and Malaikannan Sankarasubbu  “MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”  In _Conference on Health, Inference, and Learning_ , 2022, pp. 248–260  PMLR
  * [18] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen and Xinghua Lu  “PubMedQA: A dataset for biomedical research question answering”  In _arXiv preprint arXiv:1909.06146_ , 2019
  * [19] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry and Amanda Askell  “Language models are few-shot learners”  In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [20] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton and Sebastian Gehrmann  “PaLM: Scaling language modeling with pathways”  In _arXiv preprint arXiv:2204.02311_ , 2022
  * [21] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani and Siddhartha Brahma  “Scaling instruction-finetuned language models”  In _arXiv preprint arXiv:2210.11416_ , 2022
  * [22] David M Levine, Rudraksh Tuwani, Benjamin Kompa, Amita Varma, Samuel G Finlayson, Ateev Mehrotra and Andrew Beam  “The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model”  In _medRxiv_ Cold Spring Harbor Laboratory Press, 2023, pp. 2023–01
  * [23] Dat Duong and Benjamin D Solomon  “Analysis of large-language model versus human performance for genetics questions”  In _medRxiv_ Cold Spring Harbor Laboratory Press, 2023, pp. 2023–01
  * [24] Namkee Oh, Gyu-Seong Choi and Woo Yong Lee  “ChatGPT Goes to Operating Room: Evaluating GPT-4 Performance and Its Potential in Surgical Education and Training in the Era of Large Language Models”  In _medRxiv_ Cold Spring Harbor Laboratory Press, 2023, pp. 2023–03
  * [25] Fares Antaki, Samir Touma, Daniel Milad, Jonathan El-Khoury and Renaud Duval  “Evaluating the performance of chatgpt in ophthalmology: An analysis of its successes and shortcomings”  In _Ophthalmology Science_ Elsevier, 2023, pp. 100324
  * [26] John W Ayers, Adam Poliak, Mark Dredze, Eric C Leas, Zechariah Zhu, Jessica B Kelley, Dennis J Faix, Aaron M Goodman, Christopher A Longhurst and Michael Hogarth  “Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum”  In _JAMA Internal Medicine_ , 2023
  * [27] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi and Denny Zhou  “Self-consistency improves chain of thought reasoning in language models”  In _arXiv preprint arXiv:2203.11171_ , 2022
  * [28] Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang and Denny Zhou  “Recitation-Augmented Language Models”  In _arXiv preprint arXiv:2210.01296_ , 2022
  * [29] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye and Yiming Yang  “Self-refine: Iterative refinement with self-feedback”  In _arXiv preprint arXiv:2303.17651_ , 2023
  * [30] Varun Nair, Elliot Schumacher, Geoffrey Tso and Anitha Kannan  “DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents”  In _arXiv preprint arXiv:2303.17071_ , 2023
  * [31] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt  “Measuring massive multitask language understanding”  In _arXiv preprint arXiv:2009.03300_ , 2020
  * [32] Asma Ben Abacha, Eugene Agichtein, Yuval Pinter and Dina Demner-Fushman  “Overview of the medical question answering task at TREC 2017 LiveQA.”  In _TREC_ , 2017, pp. 1–12
  * [33] Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R Goodwin, Sonya E Shooshan and Dina Demner-Fushman  “Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers.”  In _MedInfo_ , 2019, pp. 25–29
  * [34] Darshali A Vyas, Leo G Eisenstein and David S Jones  “Hidden in plain sight—reconsidering the use of race correction in clinical algorithms”  In _New England Journal of Medicine_ 383.9 Mass Medical Soc, 2020, pp. 874–882
  * [35] Lesley A Inker, Nwamaka D Eneanya, Josef Coresh, Hocine Tighiouart, Dan Wang, Yingying Sang, Deidra C Crews, Alessandro Doria, Michelle M Estrella and Marc Froissart  “New creatinine-and cystatin C–based equations to estimate GFR without race”  In _New England Journal of Medicine_ 385.19 Mass Medical Soc, 2021, pp. 1737–1749
  * [36] Nwamaka D Eneanya, L Boulware, Jennifer Tsai, Marino A Bruce, Chandra L Ford, Christina Harris, Leo S Morales, Michael J Ryan, Peter P Reese and Roland J Thorpe  “Health inequities and the inappropriate use of race in nephrology”  In _Nature Reviews Nephrology_ 18.2 Nature Publishing Group, 2022, pp. 84–94
  * [37] Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman and Marzyeh Ghassemi  “Ethical machine learning in healthcare”  In _Annual review of biomedical data science_ 4 Annual Reviews, 2021, pp. 123–144
  * [38] Michael J Rigby  “Ethical dimensions of using artificial intelligence in health care”  In _AMA Journal of Ethics_ 21.2 American Medical Association, 2019, pp. 121–124
  * [39] Tamara Williams, Marilyn Szekendi, Stephen Pavkovic, Wanda Clevenger and Julie Cerese  “The reliability of AHRQ Common Format Harm Scales in rating patient safety events”  In _Journal of patient safety_ 11.1 JSTOR, 2015, pp. 52–59
  * [40] David R Williams, Jourdyn A Lawrence, Brigette A Davis and Cecilia Vu  “Understanding how discrimination can affect health”  In _Health services research_ 54 Wiley Online Library, 2019, pp. 1374–1388
  * [41] Ruqaiijah Yearby  “Structural racism and health disparities: Reconfiguring the social determinants of health framework to include the root cause”  In _Journal of Law, Medicine & Ethics_ 48.3 Cambridge University Press, 2020, pp. 518–526
  * [42] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le and Denny Zhou  “Chain of thought prompting elicits reasoning in large language models”  In _arXiv preprint arXiv:2201.11903_ , 2022
  * [43] boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer and Huan Sun  “Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters”  In _arXiv preprint arXiv:2212.10001_ , 2022
  * [44] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag and Theo Gutman-Solo  “Solving quantitative reasoning problems with language models”  In _arXiv preprint arXiv:2206.14858_ , 2022
  * [45] OpenAI  “GPT-4 Technical Report”, 2023  arXiv:2303.08774 [cs.CL]
  * [46] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker and Yu Du  “Lamda: Language models for dialog applications”  In _arXiv preprint arXiv:2201.08239_ , 2022
  * [47] Jannik Kossen, Cătălina Cangea, Eszter Vértes, Andrew Jaegle, Viorica Patraucean, Ira Ktena, Nenad Tomasev and Danielle Belgrave  “Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task”  In _arXiv preprint arXiv:2211.05039_ , 2022
  * [48] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle and Atoosa Kasirzadeh  “Ethical and social risks of harm from language models”  In _arXiv preprint arXiv:2112.04359_ , 2021
  * [49] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu and Ananya Kumar  “Holistic evaluation of language models”  In _arXiv preprint arXiv:2211.09110_ , 2022
  * [50] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese and Geoffrey Irving  “Red teaming language models with language models”  In _arXiv preprint arXiv:2202.03286_ , 2022

## References

  * [51] Justus J Randolph  “Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss’ Fixed-Marginal Multirater Kappa.”  In _Online submission_ ERIC, 2005
  * [52] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le and Denny Zhou  “Chain of thought prompting elicits reasoning in large language models”  In _arXiv preprint arXiv:2201.11903_ , 2022
  * [53] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis and Stephen Pfohl  “Large Language Models Encode Clinical Knowledge”  In _arXiv preprint arXiv:2212.13138_ , 2022
  * [54] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang and Peter Szolovits  “What disease does this patient have? a large-scale open domain question answering dataset from medical exams”  In _Applied Sciences_ 11.14 MDPI, 2021, pp. 6421
  * [55] Ankit Pal, Logesh Kumar Umapathi and Malaikannan Sankarasubbu  “MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”  In _Conference on Health, Inference, and Learning_ , 2022, pp. 248–260  PMLR
  * [56] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen and Xinghua Lu  “PubMedQA: A dataset for biomedical research question answering”  In _arXiv preprint arXiv:1909.06146_ , 2019
  * [57] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt  “Measuring massive multitask language understanding”  In _arXiv preprint arXiv:2009.03300_ , 2020

appendix