# Hidden Flaws Behind Expert-Level Accuracy of GPT-4 Vision in Medicine

Qiao Jin  National Library of Medicine, National Institutes of Health,
Bethesda, MD, USA.  M.D  National Library of Medicine, National Institutes of
Health, Bethesda, MD, USA.  Fangyuan Chen  University of Pittsburgh,
Pittsburgh, PA, USA.  Yiliang Zhou  Department of Population Health Sciences,
Weill Cornell Medicine, New York, NY, USA.  M.S  Department of Population
Health Sciences, Weill Cornell Medicine, New York, NY, USA.  Ziyang Xu  Ronald
O. Perelman Department of Dermatology, New York University Grossman School of
Medicine, New York City, NY, USA.  M.D  Ronald O. Perelman Department of
Dermatology, New York University Grossman School of Medicine, New York City,
NY, USA.  Ph.D  Ronald O. Perelman Department of Dermatology, New York
University Grossman School of Medicine, New York City, NY, USA.  Justin M.
Cheung  Department of Medicine, Harvard Medical School and Massachusetts
General Hospital, Boston, MA, USA.  M.D  Department of Medicine, Harvard
Medical School and Massachusetts General Hospital, Boston, MA, USA.  Robert
Chen  Pathology & Laboratory Medicine, Weill Cornell Medicine, New York, NY,
USA.  M.D  Pathology & Laboratory Medicine, Weill Cornell Medicine, New York,
NY, USA.  Ronald M. Summers  Imaging Biomarkers and Computer-Aided Diagnosis
Laboratory, Department of Radiology and Imaging Sciences, National Institutes
of Health Clinical Center, Bethesda, MD, USA.  M.D  Imaging Biomarkers and
Computer-Aided Diagnosis Laboratory, Department of Radiology and Imaging
Sciences, National Institutes of Health Clinical Center, Bethesda, MD, USA.
Ph.D  Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Department
of Radiology and Imaging Sciences, National Institutes of Health Clinical
Center, Bethesda, MD, USA.  Justin F. Rousseau  Department of Neurology, Peter
O’Donnell Jr. Brain Institute, UT Southwestern Medical Center, Dallas, TX,
USA.  M.D  Department of Neurology, Peter O’Donnell Jr. Brain Institute, UT
Southwestern Medical Center, Dallas, TX, USA.  M.M.Sc  Department of
Neurology, Peter O’Donnell Jr. Brain Institute, UT Southwestern Medical
Center, Dallas, TX, USA.  Peiyun Ni  Division of Gastroenterology, Department
of Medicine, Harvard Medical School and Massachusetts General Hospital,
Boston, MA, USA.  M.D  Division of Gastroenterology, Department of Medicine,
Harvard Medical School and Massachusetts General Hospital, Boston, MA, USA.
Marc J Landsman  Division of Gastroenterology, Department of Medicine,
Metrohealth Medical Center, Cleveland, OH, USA. Case Western Reserve
University School of Medicine, Cleveland, OH, USA.  M.D  Division of
Gastroenterology, Department of Medicine, Metrohealth Medical Center,
Cleveland, OH, USA. Case Western Reserve University School of Medicine,
Cleveland, OH, USA.  Sally L. Baxter  Division of Ophthalmology Informatics
and Data Science, Viterbi Family Department of Ophthalmology and Shiley Eye
Institute, University of California San Diego, La Jolla, CA, USA.  M.D
Division of Ophthalmology Informatics and Data Science, Viterbi Family
Department of Ophthalmology and Shiley Eye Institute, University of California
San Diego, La Jolla, CA, USA.  M.Sc  Division of Ophthalmology Informatics and
Data Science, Viterbi Family Department of Ophthalmology and Shiley Eye
Institute, University of California San Diego, La Jolla, CA, USA.  Subhi J.
Al’Aref  Division of Cardiology, Department of Internal Medicine, University
of Arkansas for Medical Sciences, Little Rock, AR, USA.  M.D  Division of
Cardiology, Department of Internal Medicine, University of Arkansas for
Medical Sciences, Little Rock, AR, USA.  Yijia Li  University of Pittsburgh
Medical Center, Pittsburgh, PA, USA.  M.D  University of Pittsburgh Medical
Center, Pittsburgh, PA, USA.  Michael F. Chiang  National Eye Institute,
National Institutes of Health, Bethesda, MD, USA.  National Library of
Medicine, National Institutes of Health, Bethesda, MD, USA.  M.D  National Eye
Institute, National Institutes of Health, Bethesda, MD, USA.  National Library
of Medicine, National Institutes of Health, Bethesda, MD, USA.  Yifan Peng
Department of Population Health Sciences, Weill Cornell Medicine, New York,
NY, USA.  Corresponding author(s<EMAIL_ADDRESS><EMAIL_ADDRESS>Ph.D  Department of Population Health Sciences, Weill Cornell Medicine, New
York, NY, USA.  Corresponding author(s<EMAIL_ADDRESS><EMAIL_ADDRESS>Zhiyong Lu  National Library of Medicine, National
Institutes of Health, Bethesda, MD, USA.  Corresponding author(s):
<EMAIL_ADDRESS><EMAIL_ADDRESS>Ph.D  National Library of
Medicine, National Institutes of Health, Bethesda, MD, USA.  Corresponding
author(s<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent studies indicate that Generative Pre-trained Transformer 4 with Vision
(GPT-4V) outperforms human physicians in medical challenge tasks. However,
these evaluations primarily focused on the accuracy of multi-choice questions
alone. Our study extends the current scope by conducting a comprehensive
analysis of GPT-4V’s rationales of image comprehension, recall of medical
knowledge, and step-by-step multimodal reasoning when solving _New England
Journal of Medicine_ (NEJM) Image Challenges – an imaging quiz designed to
test the knowledge and diagnostic capabilities of medical professionals.
Evaluation results confirmed that GPT-4V outperforms human physicians
regarding multi-choice accuracy (88.0% vs. 77.0%, p=0.034). GPT-4V also
performs well in cases where physicians incorrectly answer, with over 80%
accuracy. However, we discovered that GPT-4V frequently presents flawed
rationales in cases where it makes the correct final choices (27.3%), most
prominent in image comprehension (21.6%). Regardless of GPT-4V’s high accuracy
in multi-choice questions, our findings emphasize the necessity for further
in-depth evaluations of its rationales before integrating such models into
clinical workflows.

## Main

Large language models (LLMs) exemplified by Generative Pre-trained Transformer
4 (GPT-4) 1 have achieved remarkable performance on a wide range of biomedical
tasks 2, including answering medical license examination questions 3, 4,
assisting literature search 5, matching patients to clinical trials 6, and
summarizing medical evidence 7. However, most of these LLMs are unimodal,
utilizing only the context in natural language, while clinical tasks most
often require the integration of narrative clinical descriptions and multiple
types of imaging tests 8, 9. Recently, OpenAI released GPT-4 with Vision
(GPT-4V), a state-of-the-art multimodal LLM that allows users to instruct
GPT-4 to analyze both images and texts together. Subsequent pilot studies have
been conducted to analyze the performance of GPT-4V in the medical domain 10,
11, 12 (summarized in Supplementary Table 1). These evaluations mainly focused
on the accuracy of GPT-4V in answering multi-choice medical questions, and in
some cases, GPT-4V outperformed medical students and even physicians in
closed-book settings. However, the multi-choice accuracy might not reflect the
actual competence of GPT-4V, and there is no guarantee that correct final
choices are based on accurate underlying rationales. Therefore, a thorough
analysis is imperative to assess whether the decision-making of GPT-4V is
based on sound rationales, rather than arbitrary conjecture.

To bridge this gap, we used 100 multiple-choice questions with single correct
answers from _New England Journal of Medicine_ (NEJM) Image Challenge (between
November 11, 2021, and October 5, 2023) as this task is non-trivial (see
details with a prior vision-language foundation model in Methods).
Specifically, we concentrated on evaluating the GPT-4V model, assessing its
proficiency in generating both the final answer and the rationales with
respect to three essential capabilities – (1) Image comprehension, where the
model describes the provided patient image(s); (2) Recall of medical
knowledge, where the model generates relevant medical knowledge required to
solve the question, such as outlining the radiological characteristics
associated with each possible choice; and (3) Step-by-step reasoning, where
the model demonstrates detailed multimodal reasoning to answer the given
question, utilizing the generated content from both image comprehension and
recall of medical knowledge. The Institutional Review Board (IRB) has
determined that this study does not involve human subjects. Therefore, IRB
review and approval are not required.

Fig. 1 presents the overall design of this study. A senior medical student
collected and answered the questions, establishing a student baseline. We then
used a specifically designed prompt to ask GPT-4V to generate rationales in
separate sections, which facilitates easier localization of the involved
capability (described in Online Methods). GPT-4V responses were manually
recorded in independent chat sessions. Each question in the dataset was then
categorized into a medical specialty and was annotated by one clinician in
that field. A multidisciplinary cohort of nine attending and resident
physicians from different specialties was recruited to answer the questions
and evaluate the rationales of GPT-4V based on their expertise, with reference
to the official correct answers and explanations provided by NEJM Image
Challenge.

Figure 1: Evaluation Procedure for GPT-4 with Vision (GPT-4V). This figure
illustrates the evaluation workflow for GPT-4V using 100 NEJM Image Challenges
collected between November 11, 2021, and October 5, 2023. a, A senior medical
student answered all questions and triaged them into specialties. b, A cohort
of nine resident and attending physicians provided their answers to the
questions in their specialty. c, GPT-4V is prompted to answer challenge
questions with a final choice and structured responses reflecting three
specific capabilities (Image Comprehension, Recall of Medical Knowledge, and
Step-by-Step Reasoning). d, The human physicians then appraised the validity
of each component of GPT-4V’s responses based on the ground-truth
explanations.

The evaluation results are shown in Fig. 2. Human performance is evaluated in
two settings: the closed-book setting (without using external tools such as a
literature search engine or the Internet), and the open-book setting (with
external resources). GPT-4V can be considered using a closed-book setting
because web browsing is disabled. First and foremost, GPT-4V surpassed
physicians in closed-book setting per final choice selection (Fig. 2a). GPT-4V
achieved 88% overall accuracy (CI: 82-94%), significantly higher than that of
physicians (77%, CI: 62-85%). GPT-4V also largely outperforms the senior
medical student, who achieved an average accuracy of 60% (CI: 50-70%) under
the closed-book setting, representing a human passing score. The best
performance under the open-book setting is achieved by human physicians (95%,
CI: 91-99%), though not significantly different from GPT-4V. Our findings,
therefore, align with the previous ones, which show the superior performance
of GPT-4V in the closed-book setting13, 12.

Figure 2: Evaluation results. a, Average multi-choice accuracies achieved by
various models and individuals, segmented by question difficulty. Statistical
significance is computed by two-sided paired t-test. b, Confusion matrices
showing the intersection of errors made by GPT-4V and human physicians. c, Bar
graphs representing the percentage of GPT-4V’s rationales in each capability
area as evaluated by human physicians for accuracy. *: $p<0.05$, **: $p<0.01$,
***: $p<0.001$, n.s.: not significant.

To investigate performance in relation to question difficulty, we classified
the questions into three levels containing similar numbers of questions based
on the percentage of correct answers chosen by the users from the NEJM website
– easy (33 questions), medium (34 questions), and hard (33 questions). Overall
performance correlates with question difficulty – almost all respondent groups
showed non-inferior performance in easy questions compared to the other two
levels. Differences between the studied groups are not significant for easy
questions. For the medium-level questions, GPT-4V significantly outperforms
all human groups in the closed-book setting, but there is no significant
difference between the performance of GPT-4V and the open-book human
physicians. Interestingly, for hard questions, human physicians with the open-
book setting achieved a significantly higher score than GPT-4V, suggesting the
assistance of GPT-4V to physicians in complex clinical scenarios could be
fairly limited.

Fig. 2b displays the confusion matrices of GPT-4V and human physicians.
Overall, 19 out of 23 (82.6%) questions that physicians failed to answer in
the closed-book setting were correctly answered by GPT-4V. Similarly, 4 out of
5 (80.0%) questions incorrectly answered by physicians in the open-book
setting were correctly answered by GPT-4V. This suggests that GPT-4V holds
potential in decision support for physicians, particularly in closed-book
scenarios. Such potential utility can be illustrated via Question 96
(Supplementary Data), which all human groups answered incorrectly. In
comparison, GPT-4V successfully deduced tongue ulceration as a rare
complication in the context of other manifestations of giant cell arteritis.
Overall, only 1 out of 100 questions was answered incorrectly by both
physicians (open-book) and GPT-4V, indicating a promising synergy between the
current open-book tools and GPT-4V.

We next evaluated the rationales of GPT-4V in three dimensions – image
comprehension, recall of medical knowledge, and step-by-step reasoning (Fig.
2c). We found that image comprehension is the most problematic, with more than
20% of cases (21.3%-29.4% in different difficulty levels) containing flawed
rationales. For example, GPT-4V mistakenly counted the input image containing
three CT images, while there are only two provided in Question 12
(Supplementary Data). In contrast, medical knowledge recall is the most
reliable, with error rates ranging from 12.1% to 15.2%. Step-by-step reasoning
has an intermediate error rate (19.2%-21.2%). Across all types of rationales,
GPT-4V reached accurate rationales in the majority (70.6%-87.9%) of all
difficulty groups.

Surprisingly, despite overall satisfactory performance, a closer investigation
showed that GPT-4V can still be erroneous in one or multiple rationales when
the final answer is correct – these mistakes predominantly occur in image
comprehension (21.6%), as opposed to knowledge recall (10.2%) and reasoning
(11.4%). For instance, in image comprehension of Question 21 (Supplementary
Data), GPT-4V correctly identified malignant syphilis with multiple evidence,
but it failed to recognize that the two skin lesions presenting at different
angles actually arise from the same pathology. In another case of Question 89
(Supplementary Data), GPT-4V correctly linked Lisch nodules with features of
the iris surface; however, medical knowledge recall of other options was
mistaken – e.g., it wrongly stated that iris nodule is uncommon in sarcoid-
associated uveitis. GPT-4V could also be logically incomplete while guessing
right – in Question 95 (Supplementary Data), it failed to exclude Argyll
Roberson pupil with a sound reason, a condition which also presents with
light-near dissociation (the correct answer) but has drastically different
etiology than the truth. This showed its incompetence to distinguish similar
manifestations. This finding is consistent with the design of the multi-choice
question format, as one does not need to understand all related knowledge to
predict the correct choice. However, a solid and comprehensive understanding
of real-world cases is expected from clinicians, which is crucial in making
valid real-world decisions.

Our evaluation has several limitations. First, we studied a relatively small
number of 100 questions as each GPT-4V output requires human examination which
is costly and time consuming. Like other similar studies, we use challenge
questions with single correct answers. However, clinicians routinely encounter
cases with incomplete information, where multiple diagnoses are possible. This
requires listing rationales for each differential diagnosis with supportive or
excluding evidence, and proposals of further testing or treatment. In future
studies, we plan to also evaluate the rationales of human physicians in
answering medical questions for comparison.

In summary, we present a comprehensive evaluation of GPT-4V’s rationales in
multimodal medical challenge tasks. Although GPT-4V demonstrates superior
multi-choice accuracy compared to human physicians in closed-book settings,
physicians remain superior with open-book tools, especially in hard questions.
Moreover, among correctly answered questions, GPT-4V may fail to understand or
interpret medical scenarios correctly at individual rationales. Our research
also identified image comprehension as the greatest challenge for GPT-4V, with
an error rate of over 20%, while medical knowledge recall was the most
reliable. This suggests that comprehensive evaluations beyond mere multi-
choice accuracy are needed before these models can be integrated into clinical
practices.

## Online Methods

### Collecting NEJM Image Challenge

For the collection of NEJM Image Challenges, we assembled 100 most recent
questions (between November 11, 2021, and October 5, 2023) along with their
ground-truth explanations and answers at https://www.nejm.org/image-challenge.
The proportion of correct answers from NEJM users, which varied between 31%
and 83%, was employed to indicate question difficulty. Consequently, the
challenges were categorized into three difficulty tiers: “easy” for a 57%-83%
correct answer rate, “medium” for 44%-57%, and “hard” for 31%-44%. The medical
specialty and imaging modality distributions are shown in Supplementary Fig.
1.

### Prompting GPT-4V

We used the Web version of GPT-4V on 10/05/2023 through
https://chat.openai.com/, the pre-training of which was completed in 2022. We
observed consistent performance, with no significant difference between 2023
and earlier questions (2021-2022), indicating minimal impact from potential
dataset leakage. The prompt we used to evaluate GPT-4V is shown below:

> _{ image}_
> _{ question}_
> _{ choices}_
> _Please first describe the image in a section named ”Image comprehension”._
> _Then, recall relevant medical knowledge that is useful for answering the
> question but is not explicitly mentioned in a section named ”Recall of
> medical knowledge”._
> _Finally, based on the first two sections, provide your step-by-step
> reasoning and answer the question in a section named ”Step-by-step
> reasoning”._
> _Please be concise._

Here {_image_}, {_question_}, and {_choices_} represent the actual image,
question, and the set of possible answers for each NEJM Image Challenge,
respectively.

### BiomedCLIP

To assess the difficulty of the NEJM Image Challenge for vision-language
foundation models, we tested the performance of BiomedCLIP 14, a multimodal
LLM that is contrastively pre-trained on a dataset of 15 million figure-
caption pairs extracted from biomedical literature. We used BiomedCLIP in a
zero-shot setting to predict the correct choice for each question.
Specifically, let $E_{i}$ be the pre-trained image encoder and $E_{t}$ be the
pre-trained text encoder. Both the image and the text encoders are accessed
from Hugging Face via https://huggingface.co/microsoft/BiomedCLIP-
PubMedBERT_256-vit_base_patch16_224. Each NEJM Image Challenge contains an
image $I$ and five free-text choices $C_{1},\ \ldots,\ C_{5}$. We first
generated the embeddings of the image and all choices with their corresponding
encoders, and then computed the logit for each choice by its dot product with
the image representation:

$Logit(C_{i})=E_{i}(I)^{T}E_{t}(C_{i})\in R$ (1)

The choice with the highest logit will be the predicted answer by BiomedCLIP.

$\hat{i}=Logit(C_{i})$ (2)

Overall, BiomedCLIP achieved the lowest performance of 26% accuracy (CI:
17-35%), only slightly better than chance (20%). This suggests the difficulty
of the NEJM Image Challenge for vision-language foundation models of smaller
sizes.

### Annotations on MTurk

The challenge questions are first triaged into nine medical specialties,
including dermatology (34 cases), pathology (10 cases), pulmonology (8 cases),
gastroenterology (7 cases), neurology (7 cases), ophthalmology (6 cases),
cardiology (6 cases), infectious diseases (5 cases), and other internal
medicine (17 cases). A senior medical student provided a closed-book answer
for each question as the student baseline. For each specialty, a resident or
attending physician was recruited to perform a two-stage annotation. In the
initial stage, the physicians were asked to answer the questions of their
specialty both without (closed-book) and with (open-book) the use of external
resources such as internet searches.

In the second stage of annotation, the human physicians review GPT-4V’s
responses given the ground-truth explanations and answers provided by the NEJM
website, evaluating the presence of errors within each segment of GPT-4V’s
rationale (Image Comprehension, Recall of Medical Knowledge, Step-by-step
Reasoning) and the accuracy of GPT-4V’s final answers. GPT-4V’s rationale for
each capability is labeled as either “Correct”, “Partially Correct”, or
“Incorrect”. When “Partially Correct” or “Incorrect” are chosen, the
physicians are also required to explain the reasons. Both stages of
annotations were conducted on the Amazon Mechanical Turk (MTurk) platform at
https://workersandbox.mturk.com/. The annotation interfaces of the first and
second stages of annotations are shown in Supplementary Figures 2 and 3,
respectively. The full annotation reports are shown in the Supplementary Data.

### Related work

The related studies on evaluating GPT-4V are summarized in Supplementary Table
1 and are compared to this study. Importantly, our evaluation is the only one
that includes both quantitative performance with physicians and systematic
rationale evaluations for all answers generated by GPT-4V.

## Acknowledgment

This research was supported by the NIH Intramural Research Program, National
Library of Medicine, Clinical Center, and National Eye Institute. This project
was also sponsored by the National Library of Medicine grant R01LM014344.

## Author Contributions

Study concepts/study design, Q.J., Y.P., Z.L.; manuscript drafting or
manuscript revision for important intellectual content, all authors; approval
of the final version of the submitted manuscript, all authors; agrees to
ensure any questions related to the work are appropriately resolved, all
authors; literature research, Q.J., F.C., Y.P., Z.L.; experimental studies,
Q.J., F.C., Y.Z.; human evaluation, Z.X., J.C., R.C., R.S., J.R., M.L., S.B.,
S.A., Y.L.; data interpretation and statistical analysis, Q.J., F.C.; and
manuscript editing, all authors.

## Competing Interests

R.S. receives royalties for patents or software licenses from iCAD, Philips,
ScanMed, PingAn, Translation Holdings, and MGB. R.S. received research support
from PingAn. Other authors declare no competing interests.

## Code Availability

The BiomedCLIP models are available at
https://huggingface.co/microsoft/BiomedCLIP-
PubMedBERT_256-vit_base_patch16_224. We accessed GPT-4 Vision through the Web
interface (https://chat.openai.com/) and manually collected its responses.

## References

  * OpenAI et al. 2023 OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, C J Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , March 2023.
  * Tian et al. 2023 Shubo Tian, Qiao Jin, Lana Yeganova, Po-Ting Lai, Qingqing Zhu, Xiuying Chen, Yifan Yang, Qingyu Chen, Won Kim, Donald C Comeau, Rezarta Islamaj, Aadit Kapoor, Xin Gao, and Zhiyong Lu.  Opportunities and challenges for ChatGPT and large language models in biomedicine and health.  _Brief. Bioinform._ , 25(1), November 2023.  ISSN 1467-5463, 1477-4054.  10.1093/bib/bbad493.
  * Singhal et al. 2023 Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera Y Arcas, Dale Webster, Greg S Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan.  Large language models encode clinical knowledge.  _Nature_ , 620(7972):172–180, August 2023.  ISSN 0028-0836, 1476-4687.  10.1038/s41586-023-06291-2.
  * Nori et al. 2023 Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz.  Capabilities of GPT-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_ , March 2023.
  * Jin et al. 2023a Qiao Jin, Robert Leaman, and Zhiyong Lu.  Retrieve, summarize, and verify: How will ChatGPT affect information seeking from the medical literature?  _J. Am. Soc. Nephrol._ , 34(8):1302–1304, August 2023a.  ISSN 1046-6673, 1533-3450.  10.1681/ASN.0000000000000166.
  * Jin et al. 2023b Qiao Jin, Zifeng Wang, Charalampos S Floudas, Jimeng Sun, and Zhiyong Lu.  Matching patients to clinical trials with large language models.  _ArXiv_ , July 2023b.  ISSN 2331-8422.  10.4103/2229-3485.184820.
  * Tang et al. 2023 Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, Chunhua Weng, and Yifan Peng.  Evaluating large language models on medical evidence summarization.  _NPJ Digit Med_ , 6(1):158, August 2023.  ISSN 2398-6352.  10.1038/s41746-023-00896-7.
  * Acosta et al. 2022 Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol.  Multimodal biomedical AI.  _Nat. Med._ , 28(9):1773–1784, September 2022.  ISSN 1078-8956, 1546-170X.  10.1038/s41591-022-01981-2.
  * Topol 2023 Eric J Topol.  As artificial intelligence goes multimodal, medical applications multiply.  _Science_ , 381(6663):adk6139, September 2023.  ISSN 0036-8075, 1095-9203.  10.1126/science.adk6139.
  * Wu et al. 2023 Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Can GPT-4V(ision) serve medical applications? case studies on GPT-4V for multimodal medical diagnosis.  _arXiv preprint arXiv:2310.09909_ , October 2023.
  * Yan et al. 2023 Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xiang Li, and Lichao Sun.  Multimodal ChatGPT for medical applications: an experimental study of GPT-4V.  _arXiv preprint arXiv:2310.19061_ , October 2023.
  * Buckley et al. 2023 Thomas Buckley, James A Diao, Adam Rodman, and Arjun K Manrai.  Accuracy of a Vision-Language model on challenging medical cases.  _arXiv preprint arXiv:2311.05591_ , November 2023.
  * Yang et al. 2023 Zhichao Yang, Zonghai Yao, Mahbuba Tasmin, Parth Vashisht, Won Seok Jang, Feiyun Ouyang, Beining Wang, Dan Berlowitz, and Hong Yu.  Performance of multimodal GPT-4V on USMLE with image: Potential for imaging diagnostic support with explanations.  _bioRxiv_ , October 2023.  10.1101/2023.10.26.23297629.
  * Zhang et al. 2023 Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Matthew P Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon.  BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs.  _arXiv preprint arXiv:2303.00915_ , March 2023.

## Supplementary materials

Supplementary Figure 1: The medical specialty and imaging modality
distributions of the 100 NEJM Image Challenges used in this study. (derm:
dermatology, other: other internal medicine, pulm: pulmonology, neuro:
neurology, gi: gastroenterology, oph: ophthalmology, card: cardiology, inf:
infectious diseases; gross: macroscopic clinical images, patho: microscopic
pathology, CT: computed tomography, XR: X-ray, MRI: magnetic resonance
imaging; other: other image types including electrocardiogram, echocardiogram,
etc.).

Supplementary Figure 2: Annotation interface for human physicians to provide
their closed-book and open-book answers to NEJM Image Challenge.

Supplementary Figure 3: Annotation interface for human physicians to provide
their judgments on whether each section (image comprehension, recall of
medical knowledge, step-by-step reasoning, and the final answer) of the GPT-4V
response contains errors.

Supplementary Table 1: Summary of relevant studies in comparison to our evaluation Evaluation | Multi-choice Accuracy |  GPT-4V Rationale Evaluation
---|---|---
|  GPT-4V |  Human Performance
92 cases from radiology and pathology websites10 |  NA |  NA |  Yes (all) & qualitative by ad-hoc comments
133 from medical visual question answering datasets11 |  29.9% (pathology), 50.0% (radiology) |  NA |  Partial (through case studies in the error analysis of GPT-4V)
226 USMLE examination questions13 |  84.2%-88.9% |  NA |  Partial (through case studies in the error analysis of GPT-4V)
934 NEJM Image Challenge12 |  61% (scored by GPT-4) |  49% (NEJM online anonymous respondents) |  NA
100 NEJM Image Challenge (this study) |  88% (scored by physicians) |  60% (medical student), 77% (physicians) |  Yes (all 100 cases) & systematically on three distinct capabilities

See pages - of GPT4V_annotation.pdf