* Schick et al. (2024) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom.  Toolformer: Language models can teach themselves to use tools.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * Sieferd et al. (2019) E. Sieferd, N. Mohanty, and R. J. Holden.  After visit summary: Not an afterthought.  In _Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care_ , volume 8, pages 85–89. SAGE Publications Sage CA: Los Angeles, CA, 2019.
  * Singhal et al. (2023a) K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al.  Large language models encode clinical knowledge.  _Nature_ , 620(7972):172–180, 2023a.
  * Singhal et al. (2023b) K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, et al.  Towards expert-level medical question answering with large language models.  _arXiv preprint arXiv:2305.09617_ , 2023b.
  * Srinivasan and Bisk (2021) T. Srinivasan and Y. Bisk.  Worst of both worlds: Biases compound in pre-trained vision-and-language models.  _arXiv preprint arXiv:2104.08666_ , 2021.
  * Steptoe and Zaninotto (2020) A. Steptoe and P. Zaninotto.  Lower socioeconomic status and the acceleration of aging: An outcome-wide analysis.  _Proceedings of the National Academy of Sciences_ , 117(26):14911–14917, 2020.
  * Strasberg and Brunt (2010) S. M. Strasberg and M. L. Brunt.  Rationale and use of the critical view of safety in laparoscopic cholecystectomy.  _Journal of the American College of Surgeons_ , 211(1):132–138, 2010.
  * Stutz et al. (2023) D. Stutz, A. T. Cemgil, A. G. Roy, T. Matejovicova, M. Barsbey, P. Strachan, M. Schaekermann, J. Freyberg, R. Rikhye, B. Freeman, J. P. Matos, U. Telang, D. R. Webster, Y. Liu, G. S. Corrado, Y. Matias, P. Kohli, Y. Liu, A. Doucet, and A. Karthikesalingam.  Evaluating AI systems under uncertain ground truth: a case study in dermatology, 2023.
  * Tanno et al. (2024) R. Tanno, D. Barrett, A. Sellergren, S. Ghaisas, S. Dathathri, A. See, J. Welbl, K. Singhal, S. Azizi, T. Tu, et al.  Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation.  2024\.
  * The New England Journal of Medicine (2024) The New England Journal of Medicine.  Image challenge.  https://www.nejm.org/image-challenge, 2024.
  * Toma et al. (2023) A. Toma, P. R. Lawler, J. Ba, R. G. Krishnan, B. B. Rubin, and B. Wang.  Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding.  _arXiv preprint arXiv:2305.12031_ , 2023.
  * Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Tu et al. (2024a) T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P.-C. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al.  Towards generalist biomedical AI.  _NEJM AI_ , 1(3):AIoa2300138, 2024a.
  * Tu et al. (2024b) T. Tu, A. Palepu, M. Schaekermann, K. Saab, J. Freyberg, R. Tanno, A. Wang, B. Li, M. Amin, N. Tomasev, et al.  Towards conversational diagnostic AI.  _arXiv preprint arXiv:2401.05654_ , 2024b.
  * Twinanda et al. (2016) A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. De Mathelin, and N. Padoy.  Endonet: a deep architecture for recognition tasks on laparoscopic videos.  _IEEE transactions on medical imaging_ , 36(1):86–97, 2016.
  * Umapathi et al. (2023) L. K. Umapathi, A. Pal, and M. Sankarasubbu.  Med-halt: Medical domain hallucination test for large language models.  _arXiv preprint arXiv:2307.15343_ , 2023.
  * Varshney et al. (2023) N. Varshney, W. Yao, H. Zhang, J. Chen, and D. Yu.  A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation.  _arXiv preprint arXiv:2307.03987_ , 2023.
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wagner et al. (2020) P. Wagner, N. Strodthoff, R.-D. Bousseljot, D. Kreiseler, F. I. Lunze, W. Samek, and T. Schaeffter.  PTB-XL, a large publicly available electrocardiography dataset.  _Scientific data_ , 7(1):1–15, 2020.
  * Wan et al. (2024) Z. Wan, C. Liu, X. Wang, C. Tao, H. Shen, Z. Peng, J. Fu, R. Arcucci, H. Yao, and M. Zhang.  Electrocardiogram instruction tuning for report generation, 2024.
  * Wang et al. (2022a) A. Wang, V. V. Ramaswamy, and O. Russakovsky.  Towards intersectionality in machine learning: Including more identities, handling underrepresentation, and performing evaluation.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 336–349, 2022a.
  * Wang et al. (2022b) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022b.
  * Wang et al. (2024) Y. Wang, W. Chen, X. Han, X. Lin, H. Zhao, Y. Liu, B. Zhai, J. Yuan, Q. You, and H. Yang.  Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning, 2024\.
  * Ward et al. (2024) A. Ward, J. Li, J. Wang, S. Lakshminarasimhan, A. Carrick, B. Campana, J. Hartford, T. Tiyasirichokchai, S. Virmani, R. Wong, et al.  Crowdsourcing dermatology images with google search ads: Creating a real-world skin condition dataset.  _arXiv preprint arXiv:2402.18545_ , 2024.
  * Way et al. (2003) L. W. Way, L. Stewart, W. Gantert, K. Liu, C. M. Lee, K. Whang, and J. G. Hunter.  Causes and prevention of laparoscopic bile duct injuries: analysis of 252 cases from a human factors and cognitive psychology perspective.  _Annals of surgery_ , 237(4):460–469, 2003.
  * Wei et al. (2021) J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * Weng and Li (2023) Y. Weng and B. Li.  Visual answer localization with cross-modal mutual knowledge transfer.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE, 2023.
  * Williams and Wyatt (2015) D. R. Williams and R. Wyatt.  Racial bias in health care and health: challenges and opportunities.  _Jama_ , 314(6):555–556, 2015.
  * Wu et al. (2024) K. Wu, E. Wu, A. Cassasola, A. Zhang, K. Wei, T. Nguyen, S. Riantawan, P. S. Riantawan, D. E. Ho, and J. Zou.  How well do llms cite relevant medical references? an evaluation framework and analyses.  _arXiv preprint arXiv:2402.02008_ , 2024.
  * Xu et al. (2023) S. Xu, L. Yang, C. Kelly, M. Sieniek, T. Kohlberger, M. Ma, W.-H. Weng, A. Kiraly, S. Kazemzadeh, Z. Melamed, et al.  ELIXR: Towards a general purpose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders.  _arXiv preprint arXiv:2308.01317_ , 2023.
  * Yang et al. (2020) F. Yang, M. Cisse, and S. Koyejo.  Fairness with overlapping groups; a probabilistic perspective.  _Advances in neural information processing systems_ , 33:4067–4078, 2020.
  * Yao et al. (2023) S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models, 2023.
  * Yu et al. (2022) J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Yue et al. (2023) X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al.  MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  _arXiv preprint arXiv:2311.16502_ , 2023.
  * Zakka et al. (2024) C. Zakka, R. Shad, A. Chaurasia, A. R. Dalal, J. L. Kim, M. Moor, R. Fong, C. Phillips, K. Alexander, E. Ashley, et al.  Almanac—retrieval-augmented language models for clinical medicine.  _NEJM AI_ , 1(2):AIoa2300068, 2024.
  * Zambrano Chaves et al. (2024) J. M. Zambrano Chaves, S.-C. Huang, Y. Xu, H. Xu, N. Usuyama, S. Zhang, F. Wang, Y. Xie, M. Khademi, Z. Yang, et al.  Training small multimodal models to bridge biomedical competency gap: A case study in radiology imaging.  _arXiv preprint arXiv:2403.08002_ , 2024.
  * Zelikman et al. (2022) E. Zelikman, J. Mu, N. D. Goodman, and Y. T. Wu.  Star: Self-taught reasoner bootstrapping reasoning with reasoning.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022\.
  * Zhang et al. (2024) T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez.  Raft: Adapting language model to domain specific rag.  _arXiv preprint arXiv:2403.10131_ , 2024.
  * Zhou et al. (2023) D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, and E. Chi.  Least-to-most prompting enables complex reasoning in large language models, 2023.

## Appendix

##  Appendix A Supplementary Table for Figure 1

Capability | Task | Metric | Med-Gemini | Previous SoTA | Best GPT-4 Method | SoTA Reference
---|---|---|---|---|---|---
Advanced Text Reasoning | NEJM CPC | Top-10 Accuracy | 72.3 | 59.1 | 50.0 | McDuff et al. (2023)
| GeneTuring | Averaged accuracy | 53.3 | 48.6 | 48.6 | Hou and Ji (2023)
| MedQA | Accuracy | 91.1 | 90.2 | 90.2 | Nori et al. (2023)
Multimodal Understanding | NEJM Image | Accuracy | 69.7 | 61.0 | 61.0 | Buckley et al. (2023)
| USMLE-MM | Accuracy | 93.5 | 80.4 | 80.4 | Reproduced
| ECG-QA | Accuracy | 57.7 | 51.6 | 51.6 | Oh et al. (2023)
| MMMU-HM | Accuracy | 67.3 | 64.7 | 64.7 | Yue et al. (2023)
| Path-VQA | Token F1 | 64.7 | 62.7 | 36.0 (Reproduced) | Tu et al. (2024a)
| PAD-UFES-20 | Accuracy | 85.9 | 88.0 | 50.0 (Reproduced) | Tu et al. (2024a)
| Slake-VQA | Token F1 | 87.5 | 89.3 | 41.0 (Reproduced) | Tu et al. (2024a)
Long-context Processing | MedVidQA | mIoU | 43.4 | 27.5 | N/A | Li et al. (2022)
| MedVidQA w/ subtitles | mIoU | 65.8 | 58.3 | N/A | Weng and Li (2023)
| Long EHR | F1 | 0.77 | 0.78 | N/A | Feder et al. (2022)
| Surgery Video CVS Assessment | Accuracy | 50.0 | 67.0 | 0.290 | Reproduced

Table A1: Performance results of bar plot in Figure 1. We display our
aggregated results comparing Med-Gemini to the previous state-of-the-art
(SoTA) and the best GPT-4 methods across text-based, multimodal, and long-
context tasks. For benchmarks where we could not find GPT-4 (or GPT-4V)
reported numbers in literature, we run evaluations on the same test sets using
public APIs for a head-to-head comparison, using same few-shot prompts as the
corresponding Med-Gemini model including instructions to ensure outputs are
correctly formatted. Note that GPT-4 results are not available (N/A) for three
long context tasks due to limitations of the context window of the public
GPT-4 / GPT-4V APIs

##  Appendix B Related Works

##### Overview of large language model in medicine

Large language models (LLMs) have revolutionized machine learning and
artificial intelligence. Researchers have employed novel network
architectures, such as transformers (Vaswani et al., 2017) and pathways
(Barham et al., 2022), to train these models on massive datasets. This self-
supervised training across diverse domains includes models like BERT (Devlin
et al., 2018), GPT (Radford et al., 2018), T5 (Raffel et al., 2020), FLAN (Wei
et al., 2021), BLOOM (Le Scao et al., 2022), Flamingo (Alayrac et al., 2022),
PaLM and PaLM2 (Chowdhery et al., 2023; Anil et al., 2023), LLaMA (Touvron et
al., 2023), PaLI (Chen et al., 2022), PaLM-E (Driess et al., 2023), and the
recent Gemini models (Gemini Team, Google, 2023, 2024). By processing text or
multimodal information, these pretrained models develop a robust understanding
of language, patterns, and relationships with remarkable adaptibility.

Minimal fine-tuning allows these models to adapt to diverse downstream tasks.
In the medical domain, Med-PaLM (Singhal et al., 2023a) and Med-PaLM 2
(Singhal et al., 2023b) represent pioneering medical LLMs fine-tuned on EHRs,
exam questions, and research literature. To achieve the goal of generalist
medical AI (GMAI) Moor et al. (2023a), researchers use general LLMs with
prompting strategies [e.g., GPT-4 with Medprompt (Nori et al., 2023)], or
refine them with multimodal data for enhanced medical understanding [e.g.,
Med-PaLM-M (Tu et al., 2024a)]. These models show promise in diagnosis
assistance (McDuff et al., 2023), risk prediction, drug discovery, diagnostic
dialogue (Tu et al., 2024b) and assessing psychiatric functioning (Galatzer-
Levy et al., 2023). Our work leverages the latest Gemini models, using either
direct instruction prompting or further fine-tuning for specialized medical
tasks. Below, we discuss related works across the areas of language,
multimodal learning, and long-context modeling.

##### Model reasoning and tool-use for language-based tasks

Reasoning is a process of logical thinking that leads to a conclusion, which
can be significantly enhanced by recent advances in LLMs and large multimodal
models (LMMs). These improvements stem from a combination of better models and
methods that directly imitate human reasoning. Language model based reasoning
techniques have been surveyed in prior works (Qiao et al., 2023; Huang and
Chang, 2023), with such surveys extended into multimodal reasoning (Wang et
al., 2024). Strategies to enhance language reasoning include prompt
engineering, improved processes, and enhancing reasoning with access to
external elements such as tools or knowledge. Prompt engineering is
exemplified by approaches such as Chain-of-Thought (CoT) prompting (Wei et
al., 2022), which involves generating a series of intermediate reasoning
steps, Least-to-Most prompting, which involves breaking down a problem into
smaller subproblems and then sequentially solving them (Zhou et al., 2023),
and other methods that explore different reasoning paths to arrive at a
conclusion (Yao et al., 2023; Besta et al., 2024). Improved processes arise
from methods such as model updates via self-improvement (Zelikman et al.,
2022) or ensemble-based approaches (Wang et al., 2022b).

Access to external elements such as tools (Schick et al., 2024; Hao et al.,
2024) or external knowledge bases through the use of retrieval augmented
generation (RAG) (Gao et al., 2024; Zhang et al., 2024) has also demonstrated
improvements in language model reasoning. Recently LLMs have also evolved to
interact with information and web tools. For tool-use, LLMs can learn to
execute external tools or application programming interface (APIs), enabling
them to perform actions in the real world like searching, calendar use, or
using translation service via APIs (Schick et al., 2024; Qin et al., 2023).
For web search specifically, LLMs incorporate traditional search engines by
understanding complex queries and providing summaries that synthesize
information from multiple sources (Nakano et al., 2021; Varshney et al.,
2023). Furthermore, LLMs are able to not only retrieve information but also
utilize tools and create ones based on user-defined needs (Cai et al., 2023).
Zakka et al. (2024) have demonstrated that search tool-use can be particularly
useful in medical guideline and treatment recommendations. In this work, we
integrate a strategy of self-training with search to improve Med-Gemini’s
capabilities for model reasoning.

##### Large multimodal models in medicine

Medical practice often requires integration of multiple modalities to deliver
effective care, for example, integrating data sources from patient history,
medical imaging, genetic testing and lab results. Models that can integrate
such modalities may provide a more comprehensive picture of a patient’s
condition. Existing approaches fall into two broad categories: specialist and
generalist. Specialist models excel at specific tasks within a medical
discipline. Examples include models optimized for radiology report generation
(Tanno et al., 2024; Zambrano Chaves et al., 2024), pathology question
answering or histopathology captioning (Lu et al., 2023), radiology-related
tasks (Xu et al., 2023), and cardiology electrocardiogram captioning (Wan et
al., 2024).

Conversely, “generalist medical AI” (GMAI) systems (Moor et al., 2023a), such
as Med-PaLM M (Tu et al., 2024a) and LLaVA-Med (Li et al., 2024), tackle a
wider range of tasks across multiple specialties, aiming for broader
applicability in clinical settings. The diversity of tasks performed by
systems such as Med-PaLM M performance remains noteworthy as one of the
earliest examples of generalist multimodal models in medicine, capable of
addressing radiology, pathology, dermatology, and genomics tasks with
competitive performance or exceeding SoTA across different specialties using a
strong pretrained LLM with appropriate fine-tuning strategies. In this report,
we further advance the evidence that AI systems can deliver strong generalist
multimodal capabilities in medicine with Med-Gemini but the primary focus is
on developing a model family considering application specific trade-offs.

##### Long-context capability of large language models

Prior works addressing tasks with long-context windows have been limited by
the capabilities of LLMs to effectively utilize large spans of text due to the
memory and computation limitation of the Transformer-based models (Liu et al.,
2024; Vaswani et al., 2017). Initial efforts used hierarchical approaches to
derive representations of clinical text that could not fit into a model’s
limited context window (Dai et al., 2022). Subsequent work such as Clinical-
Longformer and Clinical-BigBird (Li et al., 2023) focused on extending context
lengths from 512 to 4096 tokens, enabling improvements in performance in
question answering, document classification and information retrieval tasks.
Subsequent approaches explored the use of such models in combination with
imaging encoders to tackle multimodal tasks such as medical visual question
answering (Gupta and Demner-Fushman, 2022). With the advancements of hardware
and efficient algorithms, researchers have developed LLMs with larger context
windows toward 100K tokens (Dai et al., 2019; Poli et al., 2023). Recently,
Gemini further advanced the boundary of long-context capability to one million
tokens (Gemini Team, Google, 2024).

However, in the domain of medicine the majority of LLMs continue to be
evaluated on relatively short texts (Parmar et al., 2023) and single images.
Despite their importance to medicine and clinical practice, long-context
capabilities in medicine, especially in multimodal settings, are
underexplored. We address this unmet need and investigate the potential of
Med-Gemini on different long-context use cases, including video and long EHR-
related tasks.

##  Appendix C Additional details on advanced reasoning text-based tasks

###  C.1 Text-based fine-tuning & evaluation datasets

Task type | Datasets | Sample size | Description | Reference
---|---|---|---|---
Multiple-choice question answering | MedQA | 10177 | Multiple-choice questions from MedQA | Jin et al. (2021)
| Multiple-choice question answering
---
with reasoning
MedQA-R, MedQA-RS | 20354 |  | Multiple-choice questions from MedQA with
---
synthetically generated reasoning examples
Novel
Long-form question answering | HealthSearchQA, LiveQA, MedicationQA | 260 | Clinician-written long-form responses | Singhal et al. (2023a)
Summarization | MIMIC-summaries | 65 | Clinician-written summaries of medical notes | Tu et al. (2024b)

Table C1: Overview of datasets used for text-based instruction fine-tuning.
The dataset mixture and synthetic data are curated to improve Med-Gemini-L
1.0’s reasoning and ability to make use of web search.

Task Type | Modality | Dataset | Test sample size | Description | Reference
---|---|---|---|---|---
Close-ended QA | Text | MedQA | 1273 | US medical licensing exam-style, multiple-choice | Jin et al. (2021)
Open-ended QA | Text | NEJM CPC | 303 | Complex diagnostic challenging in NEJM | McDuff et al. (2023)
Open/Close-ended QA | Text | GeneTuring | 600 | Commonly seen tasks in genomics research | Hou and Ji (2023)
Long-form generation | Text | Clinical Abstraction | 81 | Meaningful summarization in clinical practice and research | Section C.4

Table C2: Overview of the evaluation benchmarks used for text-based reasoning
tasks.

###  C.2 MedQA (USMLE) Relabeling

The main objectives of this rater study are to identify (a) unanswerable
questions due to missing information, (b) potential label errors, and (c)
potentially ambiguous questions (Stutz et al., 2023). To this end, we
carefully design a two-step study as follows:

  * •

Step 1: Given the MedQA (USMLE) question and all four answer options:

    * –

(Q1) We ask “Are any of the options appropriate to answer this question?”

    * –

(Q2) If yes, “Select one or more options to answer the question.” (Multi-
select)

    * –

(Q3) We ask “Is there any additional information (such as figures, plots, lab
results, or similar) referenced in the question that is missing?”

    * –

(Q4) If yes, we ask “Do you think having access to the missing information
would change your answer?”

  * •

Step 2: After the rater completes step 1, they are presented with the ground
truth answer from MedQA:

    * –

(Q1) We ask “Having revealed the question bank’s answer key, does your answer
from before change?”

    * –

(Q2) If yes, we repeat the first two questions from above.

A key consideration that leads to this two-step approach is to reveal the
MedQA (USMLE) ground truth at the right time to avoid biasing the rater with
the ground truth when answering questions about potentially missing
information in the question (Q3 and Q4). For properly identifying label
errors, however, we present the raters with the ground truth so they can
decide to disagree (Q1 and Q2 in step 2). To identify potentially ambiguous
questions (allowing multiple “good” or true answers), we further allow raters
to select multiple options as answers111The additional question asking whether
any option is appropriate (before revealing the multi-select) is due to
technical constraints.. When asking about potentially missing information, we
aim to identify whether this missing information is critical to answer the
question.

Rater agreement in % against vote
---
|  | All | Med-Gemini incorrect | Med-Gemini correct
info missing | majority | 95.6 | 95.6 | 95.6
info missing | unanimous | 94.0 | 94.4 | 94.0
label errors | majority | 89.6 | 80.8 | 90.5
label errors | unanimous | 87.6 | 74.6 | 88.8
ambiguous | majority | 94.9 | 92.9 | 95.0
ambiguous | unanimous | 94.6 | 92.3 | 94.9
Rater agreement of answers as average overlap in %
|  | All | Med-Gemini incorrect | Med-Gemini correct
before | majority | 49.9 | 36.4 | 51.2
before | unanimous | 49.1 | 36.0 | 50.4
after | majority | 75.9 | 54.8 | 77.9
after | unanimous | 74.6 | 53.9 | 76.6
Table C3: Annotation agreement. Top: Agreement of individual ratings against
the majority or unanimous vote for various rating tasks of interest. Bottom:
Agreement of raters’ answers in terms of average overlap before and after
having revealed the MedQA ground truth.

We recruit a total of 18 primary care physicians (PCPs) from the US to
participate in the study. We select PCPs located in the US because MedQA
comprises USMLE-style questions across many specialties. For each MedQA
(USMLE) question, we collect at least three ratings from independent raters.
While the original MedQA work in (Jin et al., 2021) evaluated expert
performance with access to additional text material, our raters are not
instructed to use any material. However, we do not explicitly control for
this. PCPs take an average of 255 seconds to complete one question; 98% take
less than 10 minutes.

For each question, we aggregate the ratings in order to identify, e.g., label
errors with high certainty. First, we evaluate the agreement of each rater
against the majority or unanimous vote in Table C3. Specifically, we consider
the rater agreement for four rating tasks of interest: whether information is
missing, whether there is a label error (i.e., the rater’s answer after
revealing the MedQA (USMLE) ground truth does _not_ include the ground truth
answer from MedQA (USMLE)), whether a question is ambiguous (i.e., the rater’s
answer includes more than one option even after revealing the MedQA (USMLE)
ground truth) and agreement between raw answer options selected in terms of
average overlap (each rater can select none or multiple options). For the
former three, agreement is generally high (>87%), although it is usually lower
on questions where Med-Gemini-L 1.0 makes mistakes. For the third, in
contrast, agreement in terms of average overlap between all pairs of answers
is significantly lower: typically around 75% when raters have seen the MedQA
(USMLE) ground truth, but agreement drops to around 50% if the ground truth is
not revealed to raters.

Figure C1: MedQA (USMLE) results after re-annotation. Results complementary to
Figure 3b showing accuracy (blue) and remaining MedQA (USMLE) questions (red)
after filtering questions with missing information, label errors or questions
deemed ambiguous when aggregating ratings using majority voting (left) or
unanimous voting (right).

To measure the impact of filtering MedQA (USMLE) questions with missing
information or label errors on evaluation while taking into account annotation
uncertainty, we perform a bootstrapping experiment. Specifically, we
repeatedly sample a committee of three raters per question (with replacement).
For each committee of raters, we perform majority or unanimous voting to
identify questions with missing information or label errors to be filtered for
evaluation. This can be seen as an instance of the evaluation framework
proposed in (Stutz et al., 2023). The advantage of bootstrapping over simple
voting is that we get reliable uncertainty estimates that make sure we can
identify performance changes as statistically significant. We repeat this
experiment $1000$ times and report average and standard deviation of accuracy
and fraction of remaining questions in Figure C1.

While questions with missing information or label errors can be identified
with high confidence due to high agreement, it is more difficult to judge
whether a question is _ambiguous_. Here, we define a question as being
ambiguous if it allows for multiple answer options to be correct. Most
questions in the MedQA (USMLE) test set specifically ask for the “best”, “most
likely” or “most appropriate” option. However, it is largely unclear whether
answers do indeed only allow for one option to be e.g. the “best next step in
management” of a case. After excluding questions with missing information and
label errors using majority voting, raters selected on average 1.065 options,
indicating that some questions might indeed be ambiguous. This increases to
1.119 after revealing the ground truth. To take this into account during
evaluation, we define a rating as ambiguous if the rater selects more than one
option after revealing the ground truth. We then follow the same analysis as
above and show results in Figure C1. Overall, we find that filtering label
errors has the biggest impact on Med-Gemini-L 1.0’s performance, while
filtering for missing information or ambiguous questions can reduce the number
of questions but does not change accuracy significantly.

###  C.3 Additional results on NEJM clinico-pathological conference dataset

We present Top-1 and Top-10 performance on the NEJM clinical pathology case
studies as broken down by the primary speciality of the cases (as identified
by NEJM) in Table C4, for all specialties with at least 10 cases. In most
specialties, Internal Medicine, Pediatrics, and Psychiatry, the best Top-1 and
Top-10 performance is achieved by either Med-Gemini-L 1.0 without search or
the model with search.

| Previous SoTA | Clinician | Med-Gemini
---|---|---|---
| AMIE | Without Search | with Search | Without Search | With Search
Metrics |  Top-1↑ |  Top-10↑ |  Top-1↑ |  Top-10↑ |  Top-1↑ |  Top-10↑ |  Top-1↑ |  Top-10↑ |  Top-1↑ |  Top-10↑
Internal Med (159 Cases) | 27.7% | 61.6% | 15.5% | 34.6% | 24.5% | 47.8% | 24.5% | 64.8% | 31.4% | 74.8%
Neurology (42 Cases) | 26.8% | 56.1% | 17.1% | 31.7% | 22.0% | 36.6% | 31.0% | 66.7% | 26.2% | 61.9%
Pediatrics (33 Cases) | 30.3% | 45.5% | 6.1% | 22.7% | 12.1% | 33.3% | 21.2% | 45.5% | 12.1% | 48.5%
Psychiatry (10 Cases) | 50.0% | 70.0% | 20.0% | 50.0% | 20.0% | 60.0% | 50.0% | 100.0% | 60.0% | 90.0%

Table C4: Breakdown of performance on NEJM case studies by speciality.
Performance is reported for the specialities with at least 10 cases.

###  C.4 Real-world use cases for advanced reasoning on text-based tasks

We instruction fine-tune and evaluate Med-Gemini-M 1.0 on three challenging
real-world tasks requiring long-form text generation. Summary results are
shown in Figure 5. Detailed results of additional evaluation axes are shown in
Table C5. Datasets and evaluation procedures for each task are described in
more detail below.

Task | Num. Examples | Evaluation-Axis | Expert Preferred | Tied | Med-Gemini | p-value
---|---|---|---|---|---|---
After-Visit Summary | 31 | Accuracy | 19% | 55% | 26% | p < 0.001
|  | Coverage | 48% | 16% | 35% | p = 0.861
|  | Succinctness | 29% | 10% | 61% | p = 0.017
|  | Coherence | 29% | 13% | 58% | p = 0.017
|  | Overall | 32% | 3% | 65% | p = 0.046
Referral Letter | 25 | Overall | 0% | 8% | 92% | p < 0.001
Cochrane Simplification | 25 | Accuracy | 52% | 8% | 40% | p = 0.846
|  | Coverage | 12% | 12% | 76% | p < 0.001
|  | Succinctness | 4% | 4% | 92% | p < 0.001
|  | Reading Level | 0% | 0% | 100% | p < 0.001
|  | Overall | 12% | 0% | 88% | p < 0.001

Table C5: Evaluation of Med-Gemini on long-form text-based tasks via side-by-
side comparison with experts. Tasks include generation of after-visit
summaries, referral letters, and simplified summaries of systematic biomedical
reviews. Evaluation is performed by clinician raters. P-values are for whether
the rate at which Med-Gemini-M 1.0 is preferred or tied with experts is $0.5$
(two-sided t-test).

##### Medical summarization evaluation

This task involves generating an after-visit summary (AVS) from a de-
identified history and physical (H&P) note. An H&P note is a detailed document
in which a healthcare provider records the findings of a patient visit,
including a patient’s health background, their current symptoms, and the
findings of a physical examination. It is largely written for other healthcare
providers to ensure coordinated care. An AVS, on the other hand, is a
structured report that patients receive at the end of a medical visit
summarizing the most important aspect of the visit and their health status.

A set of 31 de-identified H&P notes is sampled from a dataset of de-identified
medical notes from outpatient visits to family medicine or internal medicine
providers. The expert after-visit summaries are written by U.S based
clinicians following guidelines based on (Sieferd et al., 2019), and further
refined by a second round of clinicians to further increase quality.

Med-Gemini-M 1.0 is prompted to generate an after-visit summary given the de-
identified H&P note as follows:

Please read through the provided medical note describing an outpatient visit
and extract the relevant information for each of the following 12 fields:
\- Patient name/age/gender: This should summarize the patient’s name, age and
gender. It should use the format: “[Patient name], [age] year old [gender]”.
If the name is not mentioned in the note, please answer "Not available".
\- Today I was seen by: This field should provide the name of the provider. If
the provider seen for the note being summarized is not mentioned, please
answer “Not available”.
\- I came in today for: This field should indicate the chief complaint or
complaints that caused the visit.
\- New health issues identified today are: This field should indicate any new
diagnoses or other issues identified as a result of the visit being
summarized. If the issue is a pre-existing condition identified in the past,
please answer "No new diagnosis".
\- Other health issues I have are: This field should indicate any pre-existing
health issues identified in notes.
\- Today we accomplished: This field should summarize the main topics of
discussion and results of any procedures performed during the current visit.
The summary could be a short list of procedures, or could be a text
description of the patient’s experience. Please be as brief as possible when
providing details, such as test results or medication names. Describing the
experience from the patient’s point of view, using phrases like “my visit”,
“my condition”.
\- My important numbers: This field should provide the results of any
measurements relevant to the visit, including vitals. Provide the results of
any numeric measurements relevant to the visit, including vitals, laboratory
studies, or pain scores. Please include the numbers that should be monitored.
Do not fabricate numbers that are not presented in the note.
\- Changes to my medications are: This field should specify any medications
that were added, for which the doses were updated, or which are no longer
needed after the visit. Please specify both newly added and stopped
medications when possible. If no changes are apparent from the note, please
answer “no changes”.
\- Other medications I have are: If the note indicates any existing
medications for the patient that the patient should continue taking without
changes, list them here. If no medications are indicated in the note, please
“Not specified”.
\- My next steps are: This field should document the patient’s next steps,
including any actions they should take, test results they should expect, and
follow-up visits they should schedule, along with the appropriate time frames
for each.
\- I should seek immediate medical attention if: If the note specifies any
conditions for which the patient should immediately seek care, specify it
here. Be sure to only include conditions that are mentioned in the note. If no
conditions are mentioned, write "Not specified".
\- Other comments from my provider: This is an optional extra field that
captures any additional relevant information the provider indicated in the
notes that it would be useful for the patient to know. Do not include
information that is already listed in the previous field.

For each field, write at a sixth-grade reading level and avoid using
abbreviations or jargon.
Note: {MEDICAL_NOTE}
After Visit Summary:

Physician raters are presented with the H&P note, the clinician generated AVS
and our model’s generated AVS. Each example is evaluated once by one of three
different U.S.-based physicians across the following axes:

Accuracy:
Which summary is more accurate? (Are all statements in the summary correct?)
\- A - B - Tie
Coverage:
Which summary has better coverage? (Does it include all relevant aspects of
the note?)
\- A - B - Tie
Coherence:
Which summary is easier to read? (Is the summary comprehensible to a consumer
with no specific medical knowledge at a 6th-grade reading level?)
\- A - B - Tie
Succinctness:
Which summary is more succinct? (Is the summary longer than it needs to be?)
\- A - B - Tie
Overall:
Which summary feels higher quality to you? (Beyond these metrics, is there a
gut feeling about the quality of the summary?)
\- A - B - Tie

##### Referral letter generation evaluation

This task involves generating a referral letter to another healthcare provider
given a de-identified outpatient medical note that contains a recommendation
for a referral. A medical referral letter is a formal document written by a
healthcare professional that requests another healthcare professional to
evaluate or treat a patient. It serves as a communication tool between
healthcare providers, ensuring continuity of care and facilitating appropriate
treatment for the patient.

A set of de-identified medical notes requiring inter-specialty evaluation are
manually selected by clinicians from a de-identified electronic healthcare
record dataset. They then generate referral letters, which are further
reviewed for quality by a U.S. board-certified clinician.

Med-Gemini-M 1.0 is prompted to generate a referral letter given the medical
note as follows:

You will be provided with a medical note describing a patient visit. The
medical note will contain a recommendation that the patient be referred to
another healthcare provider. Your task is to generate the medical referral
letter for this healthcare provider.

A medical referral letter is a formal document written by a healthcare
professional that requests another healthcare professional to evaluate or
treat a patient. It serves as a communication tool between healthcare
providers, ensuring continuity of care and facilitating appropriate treatment
for the patient.

Medical Note: {MEDICAL_NOTE}
Referral Letter:

Physician raters are presented with the outpatient note, the clinician
generated referral letter and our model’s generated referral letter. They are
blinded to the source of each referral letter and asked to perform the
following comparison:

Instructions:
You are given a medical note that mentions a referral to another healthcare
provider. Imagine you need to write a referral letter based on the information
in the note. You are provided with draft referral letters written by two
different assistants. Which draft do you prefer as a starting point for
editing into a final version? Please also provide a brief justification for
your preference in the ‘Notes‘ column.
WARNING: Unfortunately, it is not guaranteed that the draft letters accurately
reflect the referral reason or patient history. This will need to be
ascertained based on the provided medical note and should heavily factor into
your preference.

Options:
\- A Strongly Preferred - A Somewhat Preferred - Tied - B Somewhat Preferred -
B Strongly Preferred

Three different U.S. board certified physicians are recruited and each of them
evaluates all 25 examples. Ratings are aggregated by mapping the Likert scales
to a numerical range ([-2,2]) and taking the sign of the median value.

##### Medical simplification evaluation

This task involves generating a plain language summary (PLS) from a technical
abstract from a biomedical systematic review. A PLS is a version of the
technical abstract that is written in plain English and meant to be understood
by most readers without a university education (Cochrane, 2014).

A set of 25 technical abstracts and plain language summaries from systematic
reviews conducted by Cochrane is sampled from the test split of the dataset
introduced by Devaraj et al. (2021). The expert plain language summaries are
written by the original authors of the Cochrane systematic reviews.

Med-Gemini-M 1.0 is prompted to generate a PLS given the technical abstract as
follows:

Please read through the provided technical summary of a body of medical
research and provide a simplified summary that is accessible to a lay audience
without medical expertise.

Technical Summary: {TECHNICAL_ABSTRACT}
Simplified Summary:

Clinicians are presented with the technical abstract, the original PLS and our
model’s generated PLS. They are blinded to the source of each PLS and asked to
perform the following comparisons:

Grounding:
Is all the information in the simple summary factually supported by the
technical summary?
\- A Strongly Preferred - A Somewhat Preferred - Tied - B Somewhat Preferred -
B Strongly Preferred

Coverage:
Are the most important takeaways for a lay audience included in the simple
summary?
\- A Strongly Preferred - A Somewhat Preferred - Tied - B Somewhat Preferred -
B Strongly Preferred
Succinctness:
Does the simple summary only contain the most important takeaways for a lay
audience?
\- A Strongly Preferred - A Somewhat Preferred - Tied - B Somewhat Preferred -
B Strongly Preferred
Reading Level:
Is the reading-level of the simple summary appropriate for a lay audience?
\- A Strongly Preferred - A Somewhat Preferred - Tied - B Somewhat Preferred -
B Strongly Preferred
Overall:
What is the overall quality of the simple summary for a lay audience?
\- A Strongly Preferred - A Somewhat Preferred - Tied - B Somewhat Preferred -
B Strongly Preferred

Three different U.S. board certified physicians each evaluates all 25
examples. Ratings are aggregated similar to the referral letter task.

##  Appendix D Additional details on multimodal understanding tasks

###  D.1 Multimodal fine-tuning datasets

For Med-Gemini-M 1.5’s multimodal fine-tuning, we use four image-text datasets
from MultiMedBench (Tu et al., 2024a; Tanno et al., 2024) including Slake-VQA
(Liu et al., 2021), Path-VQA (He et al., 2020), MIMIC-CXR (Johnson et al.,
2019a, b), PAD-UFES-20 (Pacheco et al., 2020), in addition to the Radiology
Objects in COntext (ROCO) dataset (Pelka et al., 2018). We further use a
subset of ECG-QA (Oh et al., 2023) to develop the health signal encoder for
encoding sensor input in Med-Gemini-S 1.0. We describe the datasets in details
below:

Task type | Dataset |  | Fine-tuning
---
sample size
Dataset Description | Reference
VQA | Slake-VQA | 9849 | Close/open-ended English/Chinese VQA of radiology images | Liu et al. (2021)
VQA | Path-VQA | 19755 | Close/open-ended VQA of pathology images | He et al. (2020)
VQA | ROCO | 29907 | Close/open-ended VQA of radiology and non-radiology images | Pelka et al. (2018)
Classification | PAD-UFES-20 | 1838 | Close-ended, multiple-choice, 6-class dermatology condition | Pacheco et al. (2020)
Classification | MIMIC-CXR | 164512 | Close-ended, multiple-choice, 13-class CXR condition | Johnson et al. (2019a, b)
Classification | MIMIC-CXR | 237962 | Close-ended, binary-choice, normal vs. abnormal classification | Johnson et al. (2019a, b)
Text Report Generation | MIMIC-CXR | 90968 | Open-ended, predicting CXR finding given image and indication | Johnson et al. (2019a, b)
Signal QA | ECG-QA | 159306 | Close-ended signal QA of electrocardiograms. | Oh et al. (2023)

Table D1: Overview of the datasets used for multimodal instruction fine-
tuning.

  * •

MIMIC-CXR is a CXR dataset with free-text reports (Johnson et al., 2019a, b),
consisting of 377110 chest X-ray images along with the corresponding protected
health information (PHI)-removed text reports from 65379 patients (227835
image studies, with one or more image view positions). Each report is
annotated with 13 common radiological conditions using the CheXpert labelling
software (Irvin et al., 2019). We use the official train/test split as
described in the MIMIC-CXR for all tasks. We consider four fine-tuning tasks
using MIMIC-CXR: (1) normal vs. abnormal binary classification, (2) CXR
abnormality condition VQA, (3) synthetic CXR VQA, and (4) text report
generation. For the normal vs. abnormal binary classification task, we
classify each image into either normal or abnoraml category based on the
CheXpert “no finding” label using all frontal view images [anterior-posterior
(AP) and posterioranterior (PA) views] with the task prompt listed in Figure
D1. For CXR abnormality condition VQA, we exclude all images with normal
findings, and group positive and uncertain labels as positive class for 13
abnormal conditions: atelectasis, cardiomegaly, consolidation, edema, enlarged
cardiomediastinum, fracture, lung lesion, lung opacity, pleural effusion,
pleural other, pneumonia, pneumothorax, and support devices. Then we frame the
abnormality detection problem into a close-ended multi-class multiple-choice
question setup as shown in Figure D1. To further enrich these VQA tasks, we
generate an collection of synthetic question-and-answer pairs from radiology
reports by querying Gemini base models. We specifically prompt the LLM to
extract pairs of yes-or-no question and the corresponding answer from each
report such that they are independent of the presence of the above 13
conditions. We ensure that for each question, the number of “yes” and “no” are
the same to avoid introducing spurious correlation. All VQA tasks are added as
the auxiliary tasks for the report generation task which combines the image
with the contextual information from the INDICATION section (reason for the
study) as the model input to generate the FINDINGS and IMPRESSION sections of
the report as the target, similar to prior works (Hyland et al., 2023; Tu et
al., 2024a). Furthermore, following the procedure proposed in Tanno et al.
(2024), we filter out the training examples whose reports reference prior
studies and only keep examples where the report only refers to findings
present in the input image. This aims to mitigate hallucination of references
to non-existing prior reports, a common issue raised by multiple lines of
research Ramesh et al. (2022) and Hyland et al. (2023). The evaluation of
MIMIC-CXR will be reported in a subsequent paper.

  * •

PAD-UFES-20 includes 2298 clinical skin lesion images collected from various
smartphone devices with different resolutions, sizes, and lighting conditions
through the Dermatological and Surgical Assistance Program at the Federal
University of Espírito Santo (UFES-Brazil) (Pacheco et al., 2020). Six types
of skin lesions are included in the dataset: basal cell carcinoma, melanoma,
squamous cell carcinoma, actinic keratosis, melanocytic nevus, and seborrheic
keratosis. Each image is correlated with up to 21 clinical features (e.g.,
patient demographics, family cancer history lesion location, lesion size).
Given no published official splits, we adopt two PAD-UFES-20 split setup. We
use Med-PaLM M split (the image-level split) for a direct, fair evaluation and
comparison against the previous SoTA method. We also evaluate on a new split,
which is a split at the patient level (Table 2). We set up three
classification tasks for fine-tuning: (1) 6-class classification using the
original label distribution and 14 clinical features (age, gender, smoke,
drink, skin cancer history, cancer history, region, Fitspatrick, horizontal
and vertical diameters, itch, grew, bleed, and elevation); (2) 6-class
classification using images and clinical features as the previous task, but
with image augmentation on the training set using 8 RandAugment (Cubuk et al.,
2020) operations: autoContrast, equalize, invert, rotate, posterize, solarize,
color, and contrast; (3) 6-class classification the same as previous task, but
using an upsampled subset for four minor skin conditions (melanoma, squamous
cell carcinoma, seborrheic keratosis, and nevus) with image augmentation
during training to mitigate the class imbalance problem. The latter two
auxiliary tasks are included in the training mixture to help the model to
distinguish among different types of clinical observations. We also formulate
the skin condition classification problem as a close-ended multiple-choice
question setup as shown in Figure D1, and report the prediction accuracy for
this task.

  * •

Path-VQA is a pathology VQA dataset, which consists of 998 pathology images
with 32799 QA pairs (He et al., 2020). All images are extracted from medical
textbooks and online digital libraries. Each image is associated one or more
questions regarding different aspects of the pathology imaging including
color, location, appearance, shape, etc. 50.2% of the QA pairs are open-ended
questions (divided into 7 categories: what, where, when, whose, how, and how
much/how many). 49.8% of the QA pairs are close-ended questions with simple
"yes/no" answer. We adopt the official splits where the
training/validation/testing splits contain 19755, 6279, and 6761 QA pairs,
respectively.

  * •

Slake-VQA is a bilingual (English and Chinese) radiology image VQA dataset
(Liu et al., 2021), containing 642 annotated images with 14028 question-answer
pairs covering three imaging modalities (CT, MRI, and chest X-Rays), 39 organ
systems, and 12 diseases. Questions are either open-ended or closed-ended
related to various aspects of the radiology images, including plane, quality,
position, organ, abnormality, size, color, shape, knowledge graph, etc. The
training/validation/testing splits contain 9849, 2109, and 2070 QA pairs,
respectively.

  * •

ROCO (Radiology Objects in Context) dataset is a large-scale medical and
multimodal imaging dataset (Pelka et al., 2018). The ROCO images are from
publications available on the PubMed Central Open Access FTP mirror, which are
automatically labeled as either radiology or non-radiology. Each image has its
caption, keywords, the corresponding UMLS Semantic Types (SemTypes), and UMLS
Concept Unique Identifiers (CUIs). We use the official training set across
radiology and non-radiology, which contain 29907 image-caption pairs, and set
up a captioning task for fine-tuning. We only include the images under CC BY,
CC BY ND, CC BY SA and CC0 licenses in ROCO.

  * •

ECG-QA is a sensor-text multimodal benchmark for assessing cardiac health (Oh
et al., 2023). It is the first QA dataset specifically designed for
electrocardiogram analysis based on PTB-XL (Wagner et al., 2020), containing
diverse question templates, each validated by an ECG expert to ensure clinical
utility. Strong performance on ECG-QA indicates the ability to grasp complex
medical concepts and their connections to raw waveform signals. ECG-QA
contains two types of questions involving (1) single ECG and (2) comparing two
ECGs; each question type consists of (1) yes/no questions, (2) multiple-choice
questions, and (3) open-ended questions to provide ECG-related attributes. We
focus on single ECG questions in this work, which contain 159306, 31137, and
41093 samples for train, validation and test sets, respectively.

###  D.2 Multimodal evaluation datasets

In addition to the in-distribution datasets (details in the above section), we
include three out-of-distribution datasets to evaluate the mulitmodal
capability of Gemini:

Task Type | Modality | Dataset | Test sample size | Description | Reference
---|---|---|---|---|---
VQA | Radiology | Slake-VQA | 2070 | English-Chinese bilingual VQA on radiology images | Liu et al. (2021)
Pathology | Path-VQA | 6761 | Close/open-ended VQA on pathology images | He et al. (2020)
Dermatology | PAD-UFES-20 | 460 | 6-class skin condition multiple-choice | Pacheco et al. (2020)
Cross-specialty | NEJM Image Challenge | 934 | OOD, Close-ended VQA on open domain medical images | The New England Journal of Medicine (2024)
Cross-specialty | USMLE-MM | 46 | OOD, Close-ended VQA on open domain medical images | Novel
Cross-domain | MMMU-HM | 150 | OOD, Close/open-ended VQA on health and medical images | Yue et al. (2023)
Signal QA | Cardiology | ECG-QA | 41093 | Close-ended signal QA of electrocardiograms | Oh et al. (2023)

Table D2: Overview of the datasets used for multimodal understanding
evaluation. OOD: out-of-distribution dataset.

  * •

New England Journal of Medicine (NEJM) Image Challenge is a renowned clinical
case challenge series that tests the diagnostic acumen and visual observation
skills of medical professionals worldwide (The New England Journal of
Medicine, 2024). Every week, the NEJM presents a clinical image accompanied by
a brief case description. The images include radiographic images, natural and
dermatoscopic skin images, electrocardiograms, histopathology images,
endoscopy images, and ophthalmoscopy images. Readers are invited to carefully
analyze the photograph, consider the patient’s history, and select the final
diagnosis from five possible diagnosis candidates. We collect 942 NEJM Image
Challenge cases from 2005 to 2023. Each case consists of a medical image and
an associated question (e.g., “What is the most likely diagnosis?”), five
multiple-choice options, and a correct answer. Some cases additionally provide
text captions with relevant clinical context or other background information
in the question. We have collected 942 cases in total, yet 934 cases are
evaluated in the end for the fair comparison (until October 12, and two cases,
20160519 and 20111103 were not evaluated due to GPT-4V filters preventing
images that are assumed to be sexually explicit (Buckley et al., 2023)).

  * •

USMLE-MM (Multimodal) is a multimodal multiple-choice question dataset with 46
questions identified in the sample exams provided by www.usmle.org, which
includes images in the question. The sample exams are used for USMLE
preparation.

  * •

MMMU-HM (health and medicine) is a subset of the publicly available benchmark,
MMMU (Massive Multi-discipline Multimodal Understanding) validation set (Yue
et al., 2023). MMMU-HM includes 150 questions related to basic medical
science, clinical medicine, diagnostics and laboratory medicine, pharmacy, and
public health domains.

Figure D1: Representative examples and prompts of multimodal understanding
tasks. Med-Gemini is evaluated on various tasks, including image
classification and visual question answering (VQA), demonstrating its ability
to analyze and interpret diverse biomedical data formats. Note that the input
for ECG-QA is the raw ECG sensor sequence, visualized here as a 12-lead ECG
image from PTB-XL (Wagner et al., 2020). Also note that three MIMIC-CXR tasks
are only used for instruction fine-tuning.

###  D.3 Additional results for multimodal tasks

##### ECG-QA

To expand Med-Gemini’s capability to process raw biomedical signals for ECG-QA
tasks, we augment Gemini 1.0 Nano with an ECG-specific encoder and fine-tune
using two approaches: keeping Gemini model unchanged (frozen) and fine-tuning
Gemini model (unfrozen). We compare our Med-Gemini-S 1.0 to their baseline
counterparts: our model with frozen Gemini model to GPT-4 with SE-WRN ECG
features in input prompts (Oh et al., 2023) and our model with unfrozen Gemini
model to an ECG foundation model based on M3AE (Oh et al., 2023). Med-Gemini-S
1.0 with frozen and unfrozen Gemini yield accuracies of 57.7% and 58.4% on
single ECG questions, respectively, outperforming GPT-4 (51.6%) by 6.1% and
M3AE (57.6%) by 0.8%.

##  Appendix E Additional details on long-context understanding tasks

###  E.1 Long-context evaluation datasets

  * •

MIMIC-III Needle-in-a-Haystack is a specially curated dataset from MIMIC-III
(Johnson et al., 2016) for subtle medical condition search-retrieval task over
long EHRs. It is designed to mimic a clinically-relevant “needle-in-a-
haystack” challenging problem (Gemini Team, Google, 2023). MIMIC-III is a
large publicly-available medical database that contains medical records of
patients admitted to intensive care units. We randomly select unstructured
medical notes from 44 unique patients with more than 100 “high-
value”222Consult notes, Progress notes, History and Physical notes and
Discharge Summary notes authored by physicians/PAs/NPs/APRNs and Operative
notes by physicians/PAs. clinical notes. To construct “needle-in-a-haystack”
examples, we use our prior work (Feder et al., 2022), which aims at
identifying the problem list (conditions/symptoms/procedures) from patients’
collection of EHR documents through (1) labeling all mentions (text spans) of
problems on the medical records with machine learning based annotators; (2)
rule-based selection and aggregation of mentions to decide whether a problem
is actually existent or not. We select the examples where there is only 1
evidence snippet identified in the aggregation step, and then randomly sample
100 negative and 100 positive examples determined by the rule-based method.
200 selected examples are then sent to 3 human medical raters to decide
whether the problem actually exists or not. Specifically, the raters are
presented with the condition name and retrieved supporting evidence snippets.
The raters are then asked to answer the question: “Select ALL problems the
patient HAS HAD based on the evidence in the provided note excerpts.” As a
result, we have 121 positive examples and 79 negative examples based on the
majority voting333Example of a negative example: a patient’s records with one
mention of Sepsis in a text segment “Received IV Ceftriaxone for concern of
UTI/sepsis.”. Here the patient should not be labeled as having history of
sepsis as there is no definitive diagnosis of the condition without other
context. (Krippendorff’s alpha at 0.77, see Table E3). The majority-vote
labels are then used as ground truth labels for the subsequent evaluation. For
each example, it consists of a set of medical records, a test question
regarding to whether or not a condition of interest exists, and a binary
ground truth label. The length of the medical records varies from 200K to 700K
words.

  * •

Medical Instructional Video QA (MedVidQA) is a video-language cross-modal
dataset for the Medical Visual Answer Localization (MVAL) task (Gupta et al.,
2023). Three medical informatics experts created 3010 health-related
instructional questions for 899 videos extracted from YouTube, and localized
the visual answer to those questions by annotating their timestamps in the
video, i.e., identifying the timestamp span given text question query. The
mean duration time of these videos is 383.29 seconds. We follow the official
data split, where 2710, 145, and 155 questions and visual answers are used for
training, validation, and testing respectively. However, 7 questions are
excluded due to the YouTube video access restriction (private videos, removed
videos).

  * •

Cholec80 and Cholec80-CVS. Cholec80 is a dataset containing 80 high-quality
videos of laparoscopic cholecystectomy performed by 13 surgeons (Twinanda et
al., 2016). Cholec80 is one of the most popular benchmarks for research in
laparoscopic cholecystectomy video analysis with deep learning, and it has
been widely used in recent research, on different video understanding tasks,
including temporal segmentation of surgical phases (Golany et al., 2022; Chen
et al., 2018), and surgical tool detection (Nwoye et al., 2019; Leifman et
al., 2022). Cholec80-CVS (Ríos et al., 2023) contains Critical View of Safety
(CVS) criteria annotations, provided by skilled surgeons, for each video in
the Cholec80 dataset. The CVS (Strasberg and Brunt, 2010) is a mandatory
method, defined by three visual criteria, used for secure identification of
the cystic duct and cystic artery to minimize the risk of Bile Duct Injury
(BDI). For each video in Cholec80, skilled surgeons selected different video
segments where at least one CVS criteria was satisfied, and then for each
selected video segment, the surgeons assigned a score of 0, 1, or 2 for each
of the three CVS criteria, following an extension of the original scoring
system proposed by (Sanford and Strasberg, 2014) and (Mascagni et al., 2021).
In total, Cholec80-CVS provides CVS criteria annotations for 572 video
segments within Cholec80 videos. We assess the performance of Med-Gemini-M 1.5
in comparison to GPT-4V and Resnet3D. It is important to note that GPT-4V does
not officially support video data as input. Therefore, we sample frames from
each video clip at a rate of 1 frame per second and combine a sequence of
frames as input to the model. During our experimentation, we observe that
GPT-4V’s vision context length is limited, and we are able to insert up to 300
low-resolution images. Consequently, we filter out all video clips longer than
5 minutes. For fair comparison, we evaluate Med-Gemini-M 1.5 on the same
filtered subset of video clips. To conduct the evaluation on Resnet3D, we
randomly split the dataset into 5 consecutive folds and assess the performance
on each validation fold separately. The average accuracy across all five folds
is reported.

Task Type | Modality | Dataset | Test sample size | Description | Reference
---|---|---|---|---|---
EHR | Text | MIMIC-III-Needle-in-a-Haystack | 200 | Problem identification from EHR records |  Curated based on Feder et al. (2022)
MVAL |  Video $\pm$ Text | MedVidQA | 148 | Video span prediction | Gupta et al. (2023)
Classification | Video | Cholec80/Cholec80-CVS | 572 | Critical view of safety assessment | Ríos et al. (2023)

Table E1: Overview of the datasets used for the long-context capability
evaluation. MVAL: medical visual answer localization.

Methods | Precision | Recall | F1
---|---|---|---
Heuristic-based baseline | 0.85 (0.78, 0.92) | 0.73 (0.64, 0.80) | 0.78 (0.72, 0.84)
Med-Gemini-M 1.5 (one-shot) | 0.77 (0.66, 0.86) | 0.76 (0.67, 0.86) | 0.77 (0.68, 0.84)

Table E2: Performance comparison of Med-Gemini-M 1.5 versus the heuristic-
based annotation-aggregation baseline.

###  E.2 Rater agreement metrics for the long EHR understanding task

To ensure the reliability of the EHR benchmark, we collect ratings from three
independent raters for each of the 200 example questions. The following
metrics demonstrate strong consistency among raters:

  * •

Jaccard Similarity Index: Measures the overlap between sets of rater
selections. Let $A$, $B$, and $C$ represent the sets of selections made by
each rater. The Jaccard similarity index for unanimous selections is defined
as $J_{=3}=\frac{|A\cap B\cap C|}{|A\cup B\cup C|}$. The Jaccard similarity
index for at least 2 raters being in agreement is defined as $J_{\geq
2}=\frac{|(A\cap B)\cup(A\cap C)\cup(B\cap C)|}{|A\cup B\cup C|}$.

  * •

Krippendorff’s Alpha: A reliability coefficient designed for multiple raters.

| Num. tasks | $J_{=3}$ | $J_{\geq 2}$ | Krippendorff’s alpha
---|---|---|---|---
Existence of condition | 200 | 0.83 | 0.915 | 0.77

Table E3: Rater agreement metrics on long EHR understanding task.

A Jaccard similarity index of $0.83$ for unanimous selections indicates
substantial agreement when all three raters select identical choices. An even
higher Jaccard index of $0.915$ reflects strong consistency when at least two
out of three raters make the same selections. A Krippendorff’s alpha of $0.77$
indicates good agreement on the existence of medical conditions within the EHR
data.