# Specialist vision-language models
for clinical ophthalmology

Robbie Holland  Biomedical Image Analysis, Department of Computing, Imperial
College London, United Kingdom  Thomas R. P. Taylor  Clinical and Experimental
Sciences, Faculty of Medicine, University of Southampton, United Kingdom
Christopher Holmes  Moorfields Eye Hospital NHS Foundation Trust, London,
United Kingdom  Sophie Riedl  Laboratory for Ophthalmic Image Analysis,
Medical University of Vienna, Austria  Julia Mai  Laboratory for Ophthalmic
Image Analysis, Medical University of Vienna, Austria  Maria Patsiamanidi
Clinical and Experimental Sciences, Faculty of Medicine, University of
Southampton, United Kingdom  Dimitra Mitsopoulou  Clinical and Experimental
Sciences, Faculty of Medicine, University of Southampton, United Kingdom  Paul
Hager  Institute for AI in Healthcare and Medicine, Klinikum rechts der Isar,
Technical University of Munich, Germany  Philip Müller  Institute for AI in
Healthcare and Medicine, Klinikum rechts der Isar, Technical University of
Munich, Germany  Hendrik P. N. Scholl  Institute of Molecular and Clinical
Ophthalmology Basel, Switzerland  Department of Ophthalmology, University of
Basel, Switzerland  Department of Clinical Pharmacology, Medical University of
Vienna, Austria  Hrvoje Bogunović  Laboratory for Ophthalmic Image Analysis,
Medical University of Vienna, Austria  Ursula Schmidt-Erfurth  Laboratory for
Ophthalmic Image Analysis, Medical University of Vienna, Austria  Daniel
Rueckert  Biomedical Image Analysis, Department of Computing, Imperial College
London, United Kingdom  Institute for AI in Healthcare and Medicine, Klinikum
rechts der Isar, Technical University of Munich, Germany  Sobha Sivaprasad
Moorfields Eye Hospital NHS Foundation Trust, London, United Kingdom  Andrew
J. Lotery  Clinical and Experimental Sciences, Faculty of Medicine, University
of Southampton, United Kingdom  and Martin J. Menten  Biomedical Image
Analysis, Department of Computing, Imperial College London, United Kingdom
Institute for AI in Healthcare and Medicine, Klinikum rechts der Isar,
Technical University of Munich, Germany  on behalf of the PINNACLE consortium

###### Abstract

Clinicians spend a significant amount of time reviewing medical images and
transcribing their findings regarding patient diagnosis, referral and
treatment in text form. Vision-language models (VLMs), which automatically
interpret images and summarize their findings as text, have enormous potential
to alleviate clinical workloads and increase patient access to high-quality
medical care. While foundational models have stirred considerable interest in
the medical community, it is unclear whether their general capabilities
translate to real-world clinical utility. In this work, we show that
foundation VLMs markedly underperform compared to practicing ophthalmologists
on specialist tasks crucial to the care of patients with age-related macular
degeneration (AMD). To address this, we initially identified the essential
capabilities required for image-based clinical decision-making, and then
developed a curriculum to selectively train VLMs in these skills. The
resulting model, RetinaVLM, can be instructed to write reports that
significantly outperform those written by leading foundation medical VLMs in
disease staging (F1 score of 0.63 vs. 0.11) and patient referral (0.67 vs.
0.39), and approaches the diagnostic performance of junior ophthalmologists
(who achieve 0.77 and 0.78 on the respective tasks). Furthermore, in a reader
study involving two senior ophthalmologists with up to 32 years of experience,
RetinaVLM’s reports were found to be similarly correct (78.6% vs. 82.1%) and
complete (both 78.6%) as reports written by junior ophthalmologists with up to
10 years of experience. These results demonstrate that our curriculum-based
approach provides a blueprint for specializing generalist foundation medical
VLMs to handle real-world clinical tasks.

††footnotetext: † denotes shared senior authorship

##  1 Introduction

Figure 1: We introduce RetinaVLM, the first specialist medical generative
vision-language model (VLM). (a) Using a needs-driven training curriculum, we
trained RetinaVLM in specialist medical skills that medical foundation VLMs
are currently lacking (b) RetinaVLM is able to process retinal optical retinal
optical coherence tomography (OCT) images and flexibly respond to text-based
queries. (c) Its abilities entail the analysis of imaging biomarkers of age-
related macular degeneration (AMD), disease staging, and the referral for
treatment.

Medical images are central to many clinical decisions regarding patient
diagnosis, referral, and treatment. Clinicians spend a significant amount of
time transcribing image-based decisions into text in order to store and
communicate their findings [1, 2]. Visual-language models (VLM), which
automatically interpret medical images and generate detailed textual
descriptions, have enormous potential to alleviate clinical workloads and
increase patient access to high-quality medical care [3, 4]. To date, the
majority of medical VLMs have been trained to output a finite set of pre-
determined textual responses [5, 6, 7, 8]. Only recently, the combination of
large language models (LLM) with medical vision encoders has led to the
development of more powerful and versatile generative VLMs that are able to
write comprehensive text reports or answer complex questions [9, 10, 11].

This current generation of medical language models is fueled by vast amounts
of unstructured training data that is extracted from medical textbooks,
scientific publications or social media posts of healthcare professionals [9,
6, 10]. These foundation language models have stirred considerable interest
among the medical community for their expert-level performance on standardized
medical question-answering tasks, such as licensing exams and case studies
[12, 13]. However, it is unclear whether this general performance translates
to clinical utility in specialist medical domains [14]. Despite its impressive
scale, the training data of foundation language models has been collected
agnostically towards their downstream application and inherently lacks
specialist information related to the challenges faced by clinicians in their
daily practice.

In this study, we identify this missing piece in foundation models with the
aim of developing generative medical VLMs with real-world clinical utility. We
propose to deconstruct clinical problems into sets of mandatory capabilities
required for their resolution and selectively train VLMs in these skills. We
demonstrate the feasibility of this approach in ophthalmology. To this end we
introduce RetinaVLM, the first specialist medical generative VLM (see Figure
1a). RetinaVLM is trained using a dedicated curriculum that is specific to the
clinical management of age-related macular degeneration (AMD), the leading
cause of blindness in the elderly [15, 16]. It is able to process optical
coherence tomography (OCT) images of the retina and flexibly respond to
instructions and questions (see Figure 1b). In extensive experiments, we
evaluate RetinaVLM’s utility and versatility regarding disease staging,
patient referral and biomarker analysis in AMD (see Figure 1c).

##  2 Results

###  2.1 RetinaVLM, a specialist vision-language model for retinal image
analysis

RetinaVLM combines two main components: an ophthalmic vision encoder that
processes input OCT images, and a generative LLM that handles textual
instructions and outputs the corresponding responses (see Figure 1a). The
vision encoder is based on our previous work, in which we found it to perform
on par with RETFound [17], a large foundation model for retinal image analysis
[18]. We use Meta’s Llama 3 as generative LLM, the best performing model that
is openly available at the time of this study [19]. However, without
additional training it lacks specialist knowledge related to the analysis of
OCT images and the clinical management of AMD. Both these deep neural networks
have already been pre-trained on large OCT and natural language datasets,
respectively, and we only finetune them in the scope of this study. Additional
details about the network architecture and training are included in Section
5.2.

###  2.2 A curriculum to encode specialist ophthalmological knowledge in
vision-language models

An intuitive strategy to specialize VLMs while preserving their ability to
flexibly interact with text queries is to provide them with a set of medical
images and corresponding question-answer pairs. They are then optimized based
on the similarity of their predicted answers to the ground truth. However,
visual question-answering (VQA) datasets do not exist for most medical
specializations, including ophthalmology.

Together with a large team of ophthalmologists, which are involved with the
patient care and academic research of AMD, we defined a set of required
capabilities for accurate image-based clinical management of AMD. They include
the identification of AMD biomarkers in OCT images, the linking of these to
the AMD disease stage, and ultimately deciding on the required referral and
treatment of the patient. We then curated a training curriculum, which
consists of 41,926 OCT images, and 479,710 visual questions and answers to
progressively specialize VLMs in these capabilities.

Figure 2: We curated a two-part curriculum to specialize medical VLMs for
clinical use. (a and b) Based on a retrospectively collected OCT imaging
dataset, we created a large number of tabular reports as well as a small
number of comprehensive textual reports. (c and d) We then used an independent
LLM to automatically generate visual question-answers (VQA) based on these
reports. (e and f) This yielded two VQA datasets, the first on basic imaging
biomarkers of AMD and the second covering more advanced clinical skills. (g
and h) Finally, we trained two specialist medical generative VLMs, RetinaVLM-
Base and RetinaVLM-Specialist, using either the first or both VQA datasets.

##### Curriculum part 1: Introduction to retina

The first part of the curriculum, named Introduction to retina, primarily
covers the appearance of the retina and AMD biomarkers in OCT images. Using
automated data collection, we obtained tabular reports for 41,926
retrospectively collected OCT images of AMD patients (see Figure 2a). Each
report describes the visible biomarkers, patient’s diagnosis, visual acuity
and demographic information in 34 data fields. A full description of the OCT
dataset can be found in Section 5.1.1, the list of all tabular data fields and
example tabular reports in Figure 7, and the methodology for their automated
procurement in Section 5.4.1.

Next, we tasked an independent LLM to generate question-answer pairs based on
these reports (see Figure 2c). The model processed the content of the tabular
reports – but not the OCT images – to output a numbered list of question-
answer pairs. We generated an average of ten question-answer pairs per report
that are mostly related to the presence or absence of specific biomarkers (see
Figure 2e). The LLM was instructed to create both closed-ended ’yes or no’
style questions, and simple open-ended questions. Detailed information on the
LLM setup can be found in Section 5.4.1.

This automated approach allowed us to generate a large dataset of 408,545
question-answer pairs. However, the questions were limited in scope to the set
of biomarkers documented by the tabular reports. Training on these yielded the
first of two specialist VLMs, RetinaVLM-Base (see Figure 2g).

##### Curriculum part 2: Advanced retinal specialism

The second part of the curriculum, named Advanced retinal specialism, builds
on top of the first part to link imaging biomarkers to AMD stage and the
recommended course of treatment. As this reasoning cannot be fully conveyed
via tabular information, we tasked two ophthalmologists with 3 and 10 years of
experience, respectively, to create comprehensive textual reports for a subset
of 330 OCT images (see Figure 2b). The ophthalmologists were asked to
primarily describe the main pathological biomarkers related to AMD while also
noting any other observations regarding the retinal anatomy. This task yielded
high-quality reports that go beyond the short notes that are typically written
by ophthalmologists in their clinical routine. Instructions given to the
ophthalmologists as well as a set of sample reports are provided in Section
5.4.2 and Figure 8, respectively.

Similar to before, an independent LLM was then employed to automatically
generate question-answer pairs based on the reports (see Figure 2d). Due to
the substantially increased depth and scope of the full-text reports compared
to the tabular ones, we used several advanced LLM instructions to create 216
diverse question-answer pairs per image on average (see Figure 2f). These
cover additional biomarkers and sub-categorize them based on their size, type,
and location. Other question-answer pairs are related to the causal
relationship between biomarkers and six AMD disease stages as well as three
levels of patient referral urgency. Moreover, the question-answer pairs were
more varied in their structure in order to preserve interactive capabilities
of the foundation LLM. For example, some queries asked to summarize the
existing reports or provide several answers in succession. An example
interaction with the LLM to generate question-answer pairs with the LLM is
shown in Figure 9. Furthermore, a list of all the LLM instructions is provided
in Section A.2 and example question-answers yielded by this approach are shown
in the ‘part 2’ section of Figure 10.

This resulted in a dataset of 71,165 advanced question-answer pairs. By
further training RetinaVLM-Base on the second part of the curriculum, we
obtained our most performant VLM for the clinical management of AMD,
RetinaVLM-Specialist (see Figure 2h).

###  2.3 RetinaVLM-Specialist outperforms foundation models and approaches
junior ophthalmologists in AMD disease staging and report writing

Figure 3: (a) Comparison of the ability of four VLMs to write reports on
retinal OCT images and derive the AMD stage. (b) Overall staging accuracy for
each model was calculated using micro F1 scores with 95% CI, with tests of
statistical significance calculated using McNemar’s test. (c) Confusion
matrices between the senior ophthalmologists’ assessments (rows) against the
image-based clinical decision maker’s prediction (columns). (d) Qualitative
comparison of reports written by human ophthalmologists and RetinaVLM-
Specialist with text markings highlighting findings regarding biomarker
observations and disease stage.

Estimating the disease stage is crucial to patient management as it allows
clinicians to monitor and treat patients using standardized protocols. We
assessed the ability of four different generative VLMs to determine the AMD
disease stage from retinal OCT images. Specifically, we compared two medical
foundation VLMs, Med-Flamingo [10] and LLaVA-Med [9], to our two specialist
VLMs, RetinaVLM-Base and RetinaVLM-Specialist. Using a testing dataset of
previously unseen 276 OCT images, VLMs were tasked to write reports that
describe the OCT image before classifying the patient into one of six disease
stages (see Figure 3a). The model predictions were compared to ground truth
labels obtained from ophthalmologists. Each image was initially graded by two
out of six junior ophthalmologists, whose experience in the field ranges from
2 to 15 years. Inter-rater disagreements were resolved by a panel of two
senior ophthalmologists with 25 and 32 years of experience, respectively. For
additional methodological details, including the instruction given to the VLMs
to generate these reports, see Section 5.8.1.

We found that our intermediate RetinaVLM-Base model already performs
significantly better than both foundation VLMs, which lack the
ophthalmological specialism to stage disease (see Figure 3b). The most
performant foundation VLM, Med-Flamingo, achieved a F1 score of 0.11. This was
markedly outperformed by our advanced RetinaVLM-Specialist model, scoring at
0.63. This approached, but did not match, the accuracy of the junior
ophthalmologists who achieved an F1 score of 0.78. We analyze this discrepancy
in detail in Section 3. Both foundation VLMs and RetinaVLM-Base returned a
substantial number of invalid reports that did not conclude with one of the
six disease stages (see Figure 3c). Conversely, all generated reports by
RetinaVLM-Specialist were valid. Similar to human experts, RetinaVLM-
Specialist struggled the most when diagnosing wet inactive AMD. We attribute
this to the high number of shared imaging biomarkers that indicate either
intermediate and late-wet forms of AMD, which sometimes leads to misdiagnosis
by both ophthalmologists and RetinaVLM-Specialist (see Figure 3d). This was
despite the identification of emerging features related to inactive late wet
AMD (a small amount of hyperreflective material, or scar tissue) by RetinaVLM.
Four more examples of success and failure cases of RetinaVLM-Specialist are
shown in Figure 11a. Moreover, full numerical results as well as the confusion
matrix for Med-Flamingo are shown in Figure 12 and 13, respectively.

Figure 4: (a) Correctness, completeness and conciseness of the reports written
by LLaVA-Med, RetinaVLM-Specialist and junior ophthalmologists. Reports were
scored for on each of the three criteria by senior ophthalmologists using a
five-point Likert scale. (b) Representative reports with ratings by one of the
senior ophthalmologists.

Eighty-four of the generated reports were scored by the two senior
ophthalmologists for their correctness, completeness, and conciseness. They
were shown 28 reports written by LLaVA-Med, 28 by RetinaVLM-Specialist, and 28
by the two annotating junior ophthalmologists in random order without
knowledge of the author. For each report, the senior ophthalmologist first
reviewed the corresponding OCT image before rating the generated report in the
three criteria on a five point Likert scale [20].

Echoing our previous findings, the senior ophthalmologists observed that
LLaVA-Med failed to comprise factually correct image reports (see Figure 4a),
even though it uses specialist terminology that may give the reports the
initial appearance of being composed by an ophthalmologist (see Figure 4b).
LLaVA-Med also failed to keep reports brief, despite being explicitly
instructed to do so.

Conversely, the reports generated by RetinaVLM-Specialist were mostly graded
as correct and complete and of similar quality as those written by junior
ophthalmologists. Specifically, out of the 28 reports, the senior
ophthalmologists either agreed or strongly agreed that 22 (78.6%) written by
RetinaVLM-Specialist, and 23 (82.1%) written by junior ophthalmologists, were
correct in their observations and conclusions. Similarly, they found that for
20 (71.4%) of the images, neither of these report writers missed any features
that were relevant to AMD.

Reports by RetinaVLM-Specialist were deemed to be slightly less concise, where
18 out of 28 (64.3%) were rated as complete compared to 23 (82.1%) by the
junior ophthalmologists. This discrepancy can be seen in the second sample in
4b, where RetinaVLM-Specialist correctly identified that the image showed a
healthy retina, but also detects a small cyst that was not found in the image,
resulting in a lower conciseness rating. Junior ophthalmologists wrote a
concise report, but incorrectly associated subretinal drusenoid deposits with
intermediate AMD. These shortcomings of both the junior ophthalmologists and
RetinaVLM-Specialist are discussed in more detail in Section 3.

###  2.4 RetinaVLM-Specialist surpasses opticians and approaches junior
ophthalmologists in AMD patient screening and referral

Figure 5: (a) Evaluation of the ability of four VLMs to assess the need for
patient referral for treatment of wet AMD. (b) Overall referral accuracy was
calculated using F1 score for urgent referral with a 95% CI. Tests of
statistical significance were carried out using McNemar’s test. The
performance of individual ophthalmologists is shown by two white points. (c)
Confusion matrices between the senior ophthalmologists assessment (rows)
against the image-based clinical decision maker’s referral assessment
(columns). (d) Image reports written by the non-specialist optician who
originally referred the patient, compared with reports of the same patient
written by RetinaVLM-Specialist.

As the prevalence of AMD is expected to further increase in the upcoming
decades [16], ocular screening programs are being introduced around the world.
In the United Kingdom, some projects involve opticians and pharmacies that
acquire and interpret OCT images. They may refer a patient to a specialist
clinic, summarizing their findings and the estimated level of the patient’s
risk in a letter. In the United Kingdom, treatment guidelines for AMD mandate
that patients with signs of neovascularization are referred for immediate
treatment within two weeks. However, non-specialists exhibit a tendency to
over-diagnose these cases. An internal audit at Southampton Eye Unit found
that 74.2% of the referrals made to the clinic do not have any form of
treatable AMD. The processing and assessment of these false positives affects
the clinic’s ability to care for the remaining patients with treatable forms
of AMD.

We evaluated the ability of VLMs to assess the level of referral urgency from
OCT image (see 5a). For each case, the VLMs were provided explicit referral
guidelines, and asked to recommend which of three levels of referral urgency
was most appropriate for the patient: no referral for healthy patients, to be
seen within 18 weeks (routine referral) for patients that are at risk of
progressing to active late wet AMD but do not require treatment yet, and
referral within two weeks for patients with any signs of neovascularization
that should be urgently referred for antiangiogenic treatment. Two junior
ophthalmologists independently reviewed images of 95 patients that have
previously been referred to the hospital for treatment of wet AMD. For each
patient, they independently decided the most appropriate of the three levels
of referral urgency, and disagreements were arbitrated by the two senior
ophthalmologists. In line with previous audits, they found the false discovery
rate for urgent referrals was 69.5%. We then calculated F1 scores for the
highest risk patients in need of urgent referral between the VLM’s predictions
and the ground truth. The full referral protocol and report generation
instructions given to the VLMs are provided in Section 5.8.3.

We found that both medical foundation VLMs and Retina-Base perform worse than
opticians regarding their ability to refer patients in need of urgent
treatment (see Figure 5b). While Med-Flamingo failed to refer any of the 29
high-risk patients cases, LLaVA-Med and RetinaVLM-Base were ineffective for
differentiating high-risk patients from low- to moderate-risk patients (see
Figure 5c). RetinaVLM-Specialist was able to detect 23 out of the 29 high-risk
cases that require immediate treatment. At the same time, RetinaVLM’s false
discovery rate, defined as the ratio of the number of false positives over the
number of predicted positives, of 42.5% is substantially lower than that of
opticians at 69.5%. Owing to their ability to better differentiate moderate
from high-risk cases, the human ophthalmologists had the lowest false
discovery rate of 9.1%, although they simultaneously missed three more cases
in urgent need for treatment.

In practice, referral letters should communicate the reason for referral by
citing suspected abnormalities in the OCT image that can inform the
ophthalmologist’s initial diagnostic plan. As in the conciseness study in
Figure 4, RetinaVLM-Specialist sometimes documents the presence of small
biomarkers that cannot be found in the image. More often, RetinaVLM-Specialist
wrote an accurate imaging report but did not accurately follow the complex set
of referral guidelines provided in the instruction. This led RetinaVLM-
Specialist to incorrectly recommend that 17 of the moderate-risk patients
potentially require treatment. However, this occurred less for the 25 low-risk
patients, where RetinaVLM-Specialist correctly identified patients with little
or no abnormalities, which are often referred to the treatment clinic for a
second opinion by non-specialists (samples 1 and 3 in Figure 5d). Crucially,
we find that RetinaVLM-Specialist is effective in the detection of
intraretinal cysts and fluid that differentiate high-risk from moderate-risk
patients (samples 4 and 5). Four more examples of success and failure cases of
RetinaVLM-Specialist are shown in Figure 11b.

###  2.5 RetinaVLM accurately detects imaging biomarkers to make
recommendations

Figure 6: (a) We test four VLMs on their ability to describe the presence of
10 important imaging biomarkers of AMD. (b) Overall detection accuracy was
computed using F1 scores. (c) Detection sensitivity for each level of
biomarker severity for the most important biomarkers. (d) Regions highlighted
by RetinaVLM-Specialist that correspond with different biomarkers and disease
stage assessments in its written report. Regions with greater than 25% and 50%
importance are highlighted by yellow and red contours, respectively. Four
additional examples are shown in Figure 14.

It is important that clinical decision makers can provide evidence for their
recommendations. Disease staging reports and written referral recommendations
commonly contain descriptions of the most salient biomarkers that were
detected in the scan. We tested the ability of four VLMs to correctly identify
the presence or absence of 10 different biomarkers related to AMD. To this
end, all VLMs were tasked with writing reports for 396 OCT images that
conclude by stating the presence or absence of the biomarker in question (see
Figure 6a). The VLMs predictions were compared against the ground truth labels
obtained from junior ophthalmologists. The instruction used to generate these
biomarker focused reports is provided in Section 5.8.4.

We find that RetinaVLM-Specialist outperforms both LLaVA-Med and Med-Flamingo
in the detection of seven out of the ten of main biomarkers related to AMD
(see Figure 6b). Biomarkers that were more severe, larger, and more numerous
were detected with higher accuracy by RetinaVLM-Specialist than less advanced
presentations (see Figure 6c). Most of the smaller biomarkers, such as small
amounts of intraretinal fluid, drusen and hyperreflective foci, which can be
as small as 30 $\mu m$ in size [21], were detected with lower sensitivity.
Overall, clinically important hallmarks of late AMD were detected with a very
high sensitivity. Large volumes of subretinal and intraretinal fluid were
detected in 80% and 78% of cases, respectively, and severe levels of
hypertransmission in 84% of cases.

Finally, to visualize the functioning of RetinaVLM-Specialist, we calculated
saliency maps based on Grad-CAM [22]. These saliency maps highlight the image
regions deemed most important by the model when writing specific passages of
the report (see Figure 6d). We refer to Figure 14 for four additional image
reports with corresponding saliency maps. We qualitatively found that
RetinaVLM-Specialist is influenced by different imaging biomarkers when
writing different passages of the report, and in making its final
recommendation. We observed the saliency maps were especially effective for
highlighting hyperreflective material, RPE irregularities and
hypertransmission.

##  3 Discussion

#### Main findings of the study

In this study, we have presented RetinaVLM, the first specialist generative
visual language model in medicine. Given a retinal OCT image, RetinaVLM
provides accurate, detailed textual responses related to disease staging,
referral or biomarker identification of AMD. While large foundation deep
learning models have been employed for retinal image analysis before [23, 18],
our generative VLM is the first model that can flexibly process varied textual
queries related to complex ophthalmological decisions and return detailed
written responses. Through the use of language as primary communication
medium, artificial intelligence systems are able to dynamically perform new
tasks and meet the evolving requirements of image-based clinical decision
makers.

In extensive experiments, RetinaVLM significantly outperformed state-of-the-
art generative VLMs designed for medical use. We found that these are unable
to interpret OCT images, derive the AMD disease stage and follow standard
referral guidelines for AMD. Specifically, we have shown that in disease
staging, RetinaVLM surpasses LLaVA-Med, the most performant open-source
medical VLM, and is approaching the accuracy of junior ophthalmologists. When
testing the ability of VLMs to screen for high-risk patients, LLaVA-Med
substantially underperforms compared to junior ophthalmologists and even non-
specialist opticians. In comparison, RetinaVLM-Specialist’s reports reduced
the number of incorrect urgent referrals by almost four times compared to
opticians and had higher recall for urgent referrals than junior
ophthalmologists. Finally, RetinaVLM is able to reinforce its decisions by
citing observable biomarkers within the written report, and highlighting their
corresponding regions within the image.

We postulate that the poor performance of existing medical foundation models
stems from their lack of detailed knowledge related retinal OCT and AMD.
Current VLMs are trained on broad, unstructured datasets that are extracted
from medical textbooks, scientific publications or social media posts of
healthcare professional [9]. In the United Kingdom and the United States,
clinical trainees aspiring to become specialists must undergo up to ten years
of post-graduate training to obtain the grade of a board-certified consultant.
We argue that training data of current foundation medical VLMs lacks this
specialist knowledge and experience, hindering their effective application to
real-world clinical tasks.

A core innovation of our work was the creation of a dedicated training
curriculum that specializes VLMs in image-based clinical decision making.
Analogously to current medical education, this curriculum deconstructs
clinical problems into sets of mandatory capabilities required for their
resolution and selectively trains VLMs in these skills. To this end, we
obtained a large number of tabular reports by processing of retrospectively
collected clinical data using advanced algorithms. Additionally, we tasked
ophthalmologists to produce a limited number of highly specific textual
reports. In total, our curriculum comprises 41,926 OCT images with 479,710
corresponding visual questions and answers. While still modest in size
compared to substantially larger foundation datasets, we believe such curated
needs-driven approaches are required to deploy language models specialist
healthcare. In a similar vein, leading technology companies in artificial
intelligence have also started to look beyond the internet’s image and text
data to source specialized training data to train LLMs and VLMs in disciplines
such as computer programming, journalism, mathematics [24, 25].

#### Limitations and future research directions

Naturally, the quality of our curriculum depends on the underlying reports, in
particular that of the 330 collected detailed textual reports. The majority of
the reports used to create RetinaVLM-Specialist were written by a junior
ophthalmologist with three years experience. The remaining reports were
written by an ophthalmologist with ten years of experience, and their reports
were more comprehensive. To match or surpass the performance of the average
junior ophthalmologist, we seek to finetune this model further on a small
number of reports written by intermediate and senior ophthalmologists.

Additionally, the data may reflect local clinical definitions and workflows.
While RetinaVLM-Specialist was trained on reports written by the two UK-based
ophthalmologists, it was tested on labels derived from four separate
ophthalmologists from different countries and hospitals. In our testing we
identified differences in the staging definitions used by the UK-based and
Austria-based ophthalmologists, especially with regard to identifying
fibrovascular features that differentiate intermediate from inactive late wet
AMD. This discrepancy resulted in a number of the misclassifications by
RetinaVLM.

Similarly, we observed a discrepancy in image interpretation between junior
and senior ophthalmologists. Junior ophthalmologists did not recommend
patients for referral if it was likely that the retinal fluid observed was
caused by traction rather than neovascularization, as it is not treatable with
antiangiogenic drugs. Conversely, the senior ophthalmologist preferred that
these patients be still referred for immediate assessment to rule out
neovascularization. RetinaVLM was explicitly instructed to refer patients with
any sign of fluid of any cause (see Section 5.8.3), and correctly referred
more patients as a result.

We found that the LLM generating the question-answer pairs from the reports
was sensitive to the specifics of the generation instruction. Extensive trial
and error were required to arrive at several instructions, listed in Section
A.2, that resulted in diverse sets of high quality question-answer pairs. We
discern that all aspects of dataset creation - deciding on the required
capabilities, collecting specialized annotations and converting these to
question-answer pairs - should be formalized to systematically compare
different approaches and ultimately scale dataset curation in the future.

Beyond the formalization and extension of the creation of the curriculum,
there are other potential technical improvements to RetinaVLM. Currently,
RetinaVLM processes a single two-dimensional OCT image from one type of OCT
scanner. In ophthalmological practice, decisions are made based on three-
dimensional images from multiple time points, although many recent studies on
the use of foundation models in ophthalmology also analyze two-dimensional
images [18]. We mitigated the impact of this discrepancy on our study by
tasking ophthalmologists to select the most relevant two-dimensional slice of
the imaged volume before proceeding with the referral decision. In the future,
a more sophisticated vision encoder, which is able to handle three-dimensional
data from diverse OCT imaging devices, could be integrated with RetinaVLM.
Similarly, one may opt to incorporate multimodal information, such as health
questionnaires, clinical tests or the patient’s medical history, into the
decision making process [26]. The fundamental model architecture and training
would remain similar, but the level of reasoning required for differential
diagnosis across multiple scans would potentially increase.

RetinaVLM also inherits some of the fundamental limitations of language
models. LLMs are prone to confidently present false or fabricated information,
termed hallucinations, which has been identified as problematic in medical
contexts [13, 14]. RetinaVLM occasionally hallucinates observations of retinal
fluid and consequently diagnoses more advanced AMD stages than necessary.
RetinaVLM’s output was also sensitive to the wording of questions and
instructions. While this had little impact on our qualitative analysis,
extensive trial and error was necessary to ensure that RetinaVLM responded
with one of the provided options in the quantitative analyses.

In this study, we exclusively trained RetinaVLM for the management of AMD from
OCT images, ignoring other retinal pathologies, such as diabetic retinopathy
or glaucoma, or imaging modalities, such as color fundus photography [27, 28].
While this enabled us to explore the potential to encode advanced clinical
levels of specialism into VLMs at depth, it would limit the applicability of
the current version of our models for ocular screening. We hypothesize that
VLMs could be specialized on a wider range of ophthalmological tasks by
incorporating additional VQA datasets and training VLMs on them. This requires
costly and time-consuming curation of specialized training datasets by medical
experts. However, we believe that the involvement of medical experts is
necessary as routine clinical skills and patient management protocols are
rarely documented in existing datasets used to train AI models.

#### Conclusion

Foundation vision-language models (VLMs) have the potential to revolutionize
healthcare by automatically interpreting medical images and communicating
their findings in detailed written reports. Trained on large datasets
containing millions of medical images and textual annotations, foundation
language models have stirred considerable interest for their expert-level
performance on medical licensing exams and case studies. However, in this work
we have shown that foundation medical VLMs substantially underperform human
experts on routine clinical tasks.

We hypothesize that the training data of foundation medical VLMs currently
lacks specialist clinical knowledge and experience. To address this, we
developed a curriculum-based approach that integrates the expertise of domain
specialists into the training of medical VLMs. The resulting model, RetinaVLM,
can produce detailed imaging reports that make accurate recommendations for
the clinical management of AMD. It approaches and often matches the
performance of junior ophthalmologists in disease staging, and outperforms
non-specialist opticians in patient referral.

These results indicate that merely increasing the scale of training datasets
is insufficient for the development of VLMs with real-world clinical utility.
Instead, medical VLMs require high-quality data directly related to the
challenges faced by clinicians in their daily practice. We believe our
proposed curriculum-based approach provides a blueprint for specializing VLMs
that generate true value in healthcare.

## References

  * [1] Moy, A. J. _et al._ Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review.  _Journal of the American Medical Informatics Association_ 28, 998–1008 (2021).
  * [2] Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J.  Multimodal biomedical AI.  _Nature Medicine_ 28, 1773–1784 (2022).
  * [3] Moor, M. _et al._ Foundation models for generalist medical artificial intelligence.  _Nature_ 616, 259–265 (2023).
  * [4] Rajpurkar, P. & Lungren, M. P.  The current and future state of ai interpretation of medical images.  _New England Journal of Medicine_ 388, 1981–1990 (2023).
  * [5] Zhang, Y., Jiang, H., Miura, Y., Manning, C. D. & Langlotz, C. P.  Contrastive learning of medical visual representations from paired images and text.  In _Machine Learning for Healthcare Conference_ , 2–25 (PMLR, 2022).
  * [6] Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J. & Zou, J.  A visual–language foundation model for pathology image analysis using medical twitter.  _Nature medicine_ 29, 2307–2316 (2023).
  * [7] Lu, M. Y. _et al._ A visual-language foundation model for computational pathology.  _Nature Medicine_ 30, 863–874 (2024).
  * [8] Christensen, M., Vukadinovic, M., Yuan, N. & Ouyang, D.  Vision–language foundation model for echocardiogram interpretation.  _Nature Medicine_ 1–8 (2024).
  * [9] Li, C. _et al._ Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * [10] Moor, M. _et al._ Med-flamingo: a multimodal medical few-shot learner.  In _Machine Learning for Health_ , 353–367 (PMLR, 2023).
  * [11] Tu, T. _et al._ Towards generalist biomedical AI.  _NEJM AI_ 1, AIoa2300138 (2024).
  * [12] Kung, T. H. _et al._ Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models.  _PLoS digital health_ 2, e0000198 (2023).
  * [13] Singhal, K. _et al._ Large language models encode clinical knowledge.  _Nature_ 620, 172–180 (2023).
  * [14] Hager, P. _et al._ Evaluation and mitigation of the limitations of large language models in clinical decision-making.  _Nature Medicine_ (2024).  URL https://www.nature.com/articles/s41591-024-03097-1.
  * [15] Mitchell, P., Liew, G., Gopinath, B. & Wong, T. Y.  Age-related macular degeneration.  _The Lancet_ 392, 1147–1159 (2018).
  * [16] Wong, W. L. _et al._ Global prevalence of age-related macular degeneration and disease burden projection for 2020 and 2040: a systematic review and meta-analysis.  _The Lancet Global Health_ 2, e106–e116 (2014).
  * [17] Holland, R. _et al._ Metadata-enhanced contrastive learning from retinal optical coherence tomography images.  _arXiv preprint arXiv:2208.02529_ (2022).
  * [18] Zhou, Y. _et al._ A foundation model for generalizable disease detection from retinal images.  _Nature_ 622, 156–163 (2023).
  * [19] Meta AI.  Introducing Meta Llama 3: The most capable openly available LLM to date (2024).  URL https://ai.meta.com/blog/meta-llama-3/.  Accessed: 2024-06-25.
  * [20] Van Veen, D. _et al._ Adapted large language models can outperform medical experts in clinical text summarization.  _Nature Medicine_ 1–9 (2024).
  * [21] Fragiotta, S. _et al._ Significance of hyperreflective foci as an optical coherence tomography biomarker in retinal diseases: characterization and clinical implications.  _Journal of ophthalmology_ 2021, 6096017 (2021).
  * [22] Selvaraju, R. R. _et al._ Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626 (2017).
  * [23] De Fauw, J. _et al._ Clinically applicable deep learning for diagnosis and referral in retinal disease.  _Nature medicine_ 24, 1342–1350 (2018).
  * [24] Sambasivan, N. _et al._ “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI.  _SIGCHI, ACM_ (2021).  URL https://research.google/pubs/everyone-wants-to-do-the-model-work-not-the-data-work-data-cascades-in-high-stakes-ai/.
  * [25] Heikkilä, M.  OpenAI’s hunger for data is coming back to bite it.  _MIT Technology Review_ (2023).  URL https://www.technologyreview.com/2023/04/19/1071789/openais-hunger-for-data-is-coming-back-to-bite-it/.
  * [26] Baltrušaitis, T., Ahuja, C. & Morency, L.-P.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 423–443 (2018).
  * [27] Flaxman, S. R. _et al._ Global causes of blindness and distance vision impairment 1990–2020: a systematic review and meta-analysis.  _The Lancet Global Health_ 5, e1221–e1234 (2017).
  * [28] Abràmoff, M. D., Garvin, M. K. & Sonka, M.  Retinal imaging and image analysis.  _IEEE reviews in biomedical engineering_ 3, 169–208 (2010).
  * [29] Grill, J.-B. _et al._ Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ 33, 21271–21284 (2020).
  * [30] AI@Meta.  Llama 3 model card (2024).  URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
  * [31] Zhu, D., Chen, J., Shen, X., Li, X. & Elhoseiny, M.  MiniGPT-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ (2023).
  * [32] Chen, Z. _et al._ Chexagent: Towards a foundation model for chest x-ray interpretation.  _arXiv preprint arXiv:2401.12208_ (2024).
  * [33] Alayrac, J.-B. _et al._ Flamingo: a visual language model for few-shot learning.  _Advances in neural information processing systems_ 35, 23716–23736 (2022).
  * [34] Lin, W. _et al._ PMC-CLIP: Contrastive language-image pre-training using biomedical documents.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 525–536 (Springer, 2023).
  * [35] Liu, H., Li, C., Wu, Q. & Lee, Y. J.  Visual instruction tuning.  _Advances in neural information processing systems_ 36 (2024).
  * [36] Zhang, S. _et al._ Large-scale domain-specific pretraining for biomedical vision-language processing.  _arXiv preprint arXiv:2303.00915_ (2023).
  * [37] Holland, R. _et al._ Deep-learning-based clustering of OCT images for biomarker discovery in age-related macular degeneration (PINNACLE study report 4).  _Ophthalmology Science_ 100543 (2024).
  * [38] Kirkpatrick, J. _et al._ Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ 114, 3521–3526 (2017).
  * [39] McNemar, Q.  Note on the sampling error of the difference between correlated proportions or percentages.  _Psychometrika_ 12, 153–157 (1947).
  * [40] Paszke, A. _et al._ Automatic differentiation in pytorch (2017).
  * [41] Sutton, J. _et al._ Developing and validating a multivariable prediction model which predicts progression of intermediate to late age-related macular degeneration—the PINNACLE trial protocol.  _Eye_ 37, 1275–1283 (2023).

##  4 Extended figures

Figure 7: (a) The prevalence of each automatically collected tabular variable
among the images in the dataset. (b) Four randomly selected images and their
corresponding tabular reports. Figure 8: (a) Three example reports randomly
selected from the 244 written by the first junior ophthalmologist with 3 years
experience in ophthalmology. (b) Three example reports randomly selected from
the 86 written by the more experienced junior ophthalmologist with 10 years
experience. Figure 9: Example interaction with the LLM used to generate the
question-answer pairs. Given the report text, the LLM is instructed to
generate a set number of question-answer pairs in a specific format, according
to pre-determined rules, and the guidelines derived by ophthalmologists.
References to guidelines are replaced with the corresponding guideline text,
documented in Figure 16. Figure 10: (a) Example question-answer pairs
generated for curriculum part 1 (based on the tabular report), and for the six
modules that constitute part 2 (based on the specialist report). (b) Example
question-answer pairs for a second example image, derived from different
tabular and specialist reports. Figure 11: (a) Four example reports by
RetinaVLM-Specialist that result in two correct and two incorrect disease
stage conclusions. Images showing inactive late wet AMD sometimes had fewer
abnormalities and were misclassified as intermediate AMD (third sample).
Another mode of failure was the hallucination of retinal fluid, which
unnecessarily upgraded the disease classification to active late wet AMD
(fourth sample). (b) Four example reports that result in two correct and two
incorrect patient referral recommendations. In each failure case RetinaVLM-
Specialist correctly identifies the lack of fluid, but fails to follow the
referral protocol and makes the wrong referral recommendation, which was the
most common mode of failure for this task. Figure 12: (a) Full results tables
for the disease staging and patient referral tasks. (b) Full results table for
the detection of ten different biomarkers indicative of AMD. Figure 13:
Confusion matrix for Med-Flamingo, included here for space reasons. Med-
Flamingo always predicted that the patient was healthy irrespective of the
true disease state of the image or returned an invalid report. Figure 14:
Additional examples of saliency maps computed with RetinaVLM-Specialist. Each
saliency map is generated for specific phrases of the report, where color
indicates correspondence. Saliency maps related to fluid often contain the
fluid boundary. Saliency maps sometimes include unrelated regions, such as for
subretinal fluid in the second sample, and the PED in the fourth sample.
Moreover, in the last sample the model misidentifies the presence of
subretinal fluid and the saliency map is centered on hypertransmission.
Overall, the saliency maps assist in highlighting more severe biomarkers, and
in interpreting the working of the model. Figure 15: RetinaVLM consists of an
ophthalmic vision encoder, a LLM and an image-to-language adapter. The vision
encoder and LLM are not updated during training. Textual instructions are
projected to the embedding space of the LLM using the pre-existing tokenizer
and embedding layer, while the vision embeddings are projected via the
learnable adapter. During training, the adapter is optimized such the the LLM
gives the corresponding response to the given instruction. Figure 16: Three
guidelines defined by junior and senior ophthalmologists for the assessment
and referral of AMD from retinal OCT. These guidelines are used to guide the
LLM generating question-answer pairs that train VLMs in the mandatory
capabilities of image-based clinical decision makers for AMD. (a) Biomarkers
observable in retinal OCT that are relevant to AMD. (b) Linking observable
biomarkers to deduce progressively severe disease stages. (c) Protocol for
patient referral for treatment with antiangiogenic drugs and routine referral,
based on the observations and disease stage.

##  5 Methods

###  5.1 Retinal image dataset curation

We use two retinal OCT dataset in this study. The first, described in Section
5.1.1, contains a cohort of patients with AMD collected retrospectively at the
Southampton Eye Unit. The second dataset, described in Section 5.1.2, contains
scans of the initial visits of patients referred, primarily by opticians, to
the Southampton Eye Unit.

All data was collected in the scope of the PINNACLE study (ClinicalTrials.gov
NCT04269304), which received approval from the East Midlands–Leicester Central
Research Ethics Committee in the United Kingdom (ref. 19/EM/0163) and the
institutional review boards of all participating institutions. It complies
with the principles of Good Clinical Practice and the Declaration of Helsinki.
All images were captured using Topcon 3D OCT scanners (Topcon Corporation,
Tokyo, Japan). Both datasets contain images of size $416\times 512$ with a
pixel size of 3.5$\times$11.7 $\mu m^{2}$.

####  5.1.1 Retrospective cohort for training RetinaVLM and testing disease
staging, report writing and biomarker analysis

The retrospective dataset contains 45,379 OCT images from 6,152 eyes belonging
to 3,468 patients, collected over eight years, between 2012 and 2020, at the
Southampton Eye Unit and aggregated by the PINNACLE consortium. For each OCT
scan we use the mediolateral 2D slice centered at the fovea.

We designated 41,926 of the 45,379 OCT images from 5,547 eyes of 3,057
patients patients for training purposes. Additionally, we reserved 2,311
images from 326 eyes of 187 patients for validation, and 396 images from 279
eyes of 224 patients for testing. We ensured that images from each patient do
not appear in more than one of the training, validation or test sets.

The training set was used to create both curriculum parts 1 and 2, detailed in
Sections 5.4.1 and 5.4.2. The test set was used to evaluate the resulting
model in Sections 2.3 and 2.5. For each patient in the test set, two junior
ophthalmologists independently decided the disease stage, and disagreements
were arbitrated by the two senior ophthalmologists. Inactive late wet AMD was
defined by the presence of any subretinal hyperreflective material or
fibrosis. Active late wet AMD was defined by presence of any fluid within the
image and took precedence over the inactive classification.

####  5.1.2 External cohort of patients referred to Southampton AMD treatment
clinic

We also collected an external dataset of 95 patients that were referred
primarily by opticians to the Southampton Eye Unit between 02/2023 and
12/2023. None had yet received treatment for AMD, and mostly had no AMD,
intermediate AMD or small features related to active wet AMD. This represents
a distribution shift from the retrospective cohort, where many patients had
already received treatment for AMD and were in the inactive late wet stage of
AMD. As such, it enabled us to estimate the robustness of both variants of
RetinaVLM to shifts in patient population. This dataset was not used for model
training and was reserved for testing VLMs on patient referral, detailed in
Section 2.4.

For each patient we sourced scans of both their left and right eye that were
acquired on their first visit to the clinic. We also collected the originally
issued letter of referral, as depicted in Figure 5d. Then, two junior
ophthalmologists analyzed the 3D OCT volumes of each eye to assess the
patient’s risk and recommend a level of referral urgency. They then selected
the image slice that most supported their assessment of the patient’s risk. In
healthy patients where both volumes contained no pathological signs in any of
the image slices, they were instructed to select the mediolateral fovea-
centered 2D slice from one of the two volumes.

###  5.2 Vision-language model architecture

RetinaVLM consists of two main components: an ophthalmological vision encoder
and a generative LLM (see Figure 15). For the ophthalmological vision encoder,
we adopt a Resnet50 convolutional neural network with over 23 million
parameters which was previously pre-trained with self-supervised contrastive
learning on the 41,926 OCT images from the train set of the retrospective
cohort. Specifically, it was trained with Bootstrap Your Own Latent (BYOL)
[29] using the same implementation details as the standard contrastive
approaches used in [17], which consistently performed on par with RETFound
[18] specifically on data from the Southampton Eye Unit. This vision encoder
projects each $192\times 192$ input image to a set of spatially arranged
6$\times$6 visual embeddings, which are extracted from the last layer before
global average pooling. Each embedding has a dimension of $h_{img}=2048$. They
also have a receptive field of size 336, so each embedding contains global
knowledge of the image that is contextualized at its local position. For the
LLM, we employ the 8 billion parameter instruction-tuned Llama3 model by Meta
[19, 30] as the generative LLM, which was was the most performant openly
available model at the time of our study. LLama3 uses an embedding dimension
of $h_{lang}=4096$.

The ophthalmological vision encoder provides visual information regarding the
OCT image to the LLM via an adapter. The adapter is a linear layer of size
$h_{img}\times h_{lang}$ that processes visual information for use by the LLM.
This follows the design used in MiniGPT4 [31], and results in an adapter with
over 8 million parameters.

###  5.3 Foundation medical vision-language models

We used the two most widely adopted foundation vision-language models for
medical applications at the time of this study [10, 9]. They were both trained
on large biomedical datasets sourced from the Internet, and have been applied
in chest x-ray [32]. The first, Med-Flamingo [10], which was built on Flamingo
[33] and finetuned on image and text data from medical textbooks and the
PubMed Central Open Access (PMC-OA) dataset [34]. The second, LLaVA-Med [9],
developed by Microsoft, is a VLM built on LLaVA [35] and finetuned to follow
textual instructions regarding a broad range of biomedical images contained in
PubMed Central 15M (PMC-15M) [36]. As they were trained as generalist models
on various imaging modalities, they were both purportedly capable of
interpreting retinal OCT images. Both correctly identified that the provided
image was a retinal OCT scan when instructed to report the modality of the
given image.

For Med-Flamingo, we then provide instructions using the following template
provided in their code, replacing {question} with the instruction text:

> You are a helpful medical assistant. You are being provided with images, a
> question about the image and an answer. Follow the examples and answer the
> last question. <image>Question: {question} Answer:

Similarly, for LLaVA-Med we use their following system prompt:

> You are a helpful medical assistant. You are being provided with images, a
> question about the image and an answer. Follow the examples and answer the
> last question. <image>Question: {question} Answer:

###  5.4 Report curation and question-answer pair generation

####  5.4.1 Curriculum part 1: Introduction to retina

To create the tabular reports for the first part of the curriculum we used a
cluster-based approach to efficiently label the 41,926 training images with
biomarker annotations [37]. Contrastive learning is used to extract self-
supervised features from the dataset. The dataset is then partitioned into 40
clusters of images that share common features. Labels are then assigned to
these clusters by senior ophthalmologists. To this end, 20 images from each
cluster were reviewed by senior ophthalmologists. If the majority of the
images exhibited common features, such as ’large drusen’ or ’subretinal
fluid’, these labels were assigned to the entire cluster. These labels were
used in in combination with the patient’s age, sex and their functional visual
acuity score (measured on a LogMAR chart and converted to Letter score) to
create the tabular reports. Additionally, the reports list three biomarkers
that are stated as not being present. These are drawn from a distribution of
all biomarkers, weighted by their prevalence in the dataset, that were not
among the cluster labels for that image. Counts of the prevalence of each
tabular variable among the images are shown in Figure 7a, and a sample of four
tabular reports they result in are shown in Figure 7b.

To generate question-answer pairs from the large volume of tabular reports we
used WizardLLM-70B, which was the most capable freely-available LLM at the
time of the creation of the first part of the curriculum. This resulted in a
total of 408,505 question-answer pairs. Examples of the question-answers pairs
generated by this approach are shown in the ‘curriculum part 1’ section of
Figure 10a and Figure 10b.

####  5.4.2 Curriculum part 2: Advanced retinal specialism

The second part of the curriculum used manually curated reports written by
retinal specialists. Two junior ophthalmologists were tasked with describing
the main pathological biomarkers and diagnoses related to AMD, while also
noting any other observation regarding the retinal anatomy. This yielded high-
quality textual reports that go beyond the short notes that ophthalmologists
typically write in clinical routine. The first junior ophthalmologist, with
three years of experience specializing in ophthalmology, wrote the majority of
244 reports (see Figure 8a). While these were highly accurate, they were less
comprehensive in their analysis than the remaining 86 reports written by the
junior ophthalmologist with 10 years of experience (see Figure 8b). In total,
this process yielded the 330 specialist reports.

In total, we used 10 different instructions for generating up to 230 questions
per image. The exact instructions used are documented in Section A.2. We take
two preliminary steps before providing the instruction to the LLM. We firstly
replace the <ReportText> identifier with the raw text of the image report.
Additionally, many of the QA generation instructions make references to the
guidelines that describe the mandatory capabilities of image-based clinical
decision makers with regard to disease staging and patient referral for
patients with AMD. The second step involves replacing any reference to the
<ObservationGuidelines>, <DiseaseStagingGuidelines> or
<PatientReferralGuidelines> by the text of the corresponding guidelines
(documented in Figure 16). These guidelines were verified by senior
ophthalmologists, and were instrumental for the generation of questions with
improved diversity and coverage, and also for creating questions about
biomarkers that were absent in the image (and typically not mentioned in the
report).

The smaller number of reports in the advanced curriculum permitted the use of
the more performant proprietary models for generating question-answer pairs.
We used the ‘gpt-4o’ API endpoint from OpenAI. An example interaction with
’gpt-4o’ for generating these question-answer pairs is shown in Figure 9. A
sample of question-answers yielded by this approach are shown in the ‘part 2’
section of both Figure 10a and Figure 10b.

###  5.5 Vision-language model training and generation

####  5.5.1 Combined vision language training

When training on both curriculum part 1 and part 2, both the vision encoder
and LLM used in RetinaVLM are kept frozen, that is, they are not updated
during the entire training process. This mitigates issues of catastrophic
forgetting [38], a phenomenon in which foundation models lose their general
capabilities during the finetuning process. This is key to ensuring RetinaVLM
can respond to versatile questions and instructions that it was not exposed to
during training. While the vision encoder and LLM are frozen, RetinaVLM gains
all its specialized capabilities solely through the training of the adapter.

Before training on the retrospective cohort we downsample each image by a
factor of 2 from $416\times 512$ to $208\times 256$ pixels. We then augment
each image using the protocol outlined in [17], which results in a randomly
cropped OCT image of size $192\times 192$. During training, we set the Llama
3’s system prompt to:

You are a helpful ophthalmological specialist chatbot capable of interpreting
retinal OCT images.

We begin the instruction that RetinaVLM will be trained on with the following
line:

Here is an encoding of a retinal OCT image <Img><ImageHere></Img>\n

Then, for each image in the batch, we randomly select a corresponding
question-answer pair from the current curriculum dataset. We then add the text
of the question to the instruction. We then populate LLama3’s conversation
template with the full instruction and corresponding answer to form the full
textual input. We project this textual input to the embedding space of the LLM
using the pre-existing tokenizer and embedding layer of the LLM (see Figure
15).

Simultaneously, we use the ophthalmic vision encoder $E$ to extract the
$6\times 6$ vision embeddings from each image. After flattening these, we
apply the adapter to each embedding separately to project them to the
embedding space of the language model. To create the final set of embeddings
that are provided to the LLM, we replace the embeddings corresponding to the
<ImageHere> phrase in the input with the 36 adapted vision embeddings.

Finally, the resulting sequence of adapted visual embeddings and language
embeddings are passed together through the frozen LLM. This yields a list of
predicted token logits with the same length as the input sequence. We then
compute the causal language modeling loss between these predicted answer
logits and the ground truth answer tokens. We then optimize the adapter to
minimize this loss.

Beginning by randomly initializing the adapter, we train in this fashion for
100,000 steps on questions and answers regarding the 41,926 images in
curriculum part 1 (introduction to retina) to obtain RetinaVLM-Base. The
trained adapter is then further finetuned on the 71,165 questions and answers
regarding the 330 images in in curriculum part 2 (advanced retinal specialism)
for 100,000 steps, resulting in the final RetinaVLM-Specialist model. For both
curriculum parts, we use a batch size of 12 and the AdamW optimizer with
$\beta_{1}=0.9$ and $\beta_{2}=0.999$.

###  5.6 Testing for catastrophic forgetting

As a quick test to indicate whether RetinaVLM-Base and RetinaVLM-Specialist
incurred catastrophic forgetting during training, we provided them a retinal
OCT image and tested them on the following instruction which does not feature
in curriculum parts 1 or 2:

Here is an encoding of a retinal OCT image <Img><ImageHere></Img> What is the
capital of England?

Previous iterations of RetinaVLM that used a more complex adapter design were
susceptible to responding with answers regarding the retinal OCT image, rather
than answering the provided question. The simplified architectural approach
used in this study results in models that pass this test.

###  5.7 Using VLMs for inference and generating text

After VLMs have been trained on image and text datasets, they can be used to
generate responses to new questions and images. To use the baseline foundation
VLMs for testing we take the central crop in each $416\times 512$ OCT image,
resulting in an image of size $384\times 384$. We then downsample the image to
$224\times 224$ and repeat it along the color dimension, resulting in the
$3\times 224\times 224$ images that are required by both foundation models. To
provide the image to the RetinaVLM variants during testing, we take a central
crop of the $208\times 256$ downsampled images, resulting in an image of size
$192\times 192$ required of both RetinaVLM variants.

We then employ the same method with all VLMs for generating responses to
instructions. Provided the image and textual instruction, we build the output
sequence of tokens by repeatedly appending the token to the output assigned
the highest probability by the VLM. This is equivalent to using a temperature
parameter set to 0, and is the standard approach for generating the most
accurate and least creative output from LLM-based models. This process is
repeated until a stop token is generated, signaling the end of the VLM’s
response. The model’s tokenizer is then used to convert the numeric output
tokens to the final free text output.

###  5.8 Experimental setup

Our entire evaluation was conducted in zero-shot, that is, after training on
curriculum part 1 and part 2 RetinaVLM requires no further finetuning in order
to perform tasks related to disease staging, patient referral and biomarker
analysis. Instead, for each test we designed a specific instruction that was
provided to all VLMs to generate the application-specific reports that were
used in our analyses. These instructions were derived through experimentation
with all VLMs on the validation set, so while they are not designed for any
one VLM in particular, they do contain information related to the task at
hand.

####  5.8.1 Tests of disease staging

The following instruction was given to all VLMs to and obtain the reports of
the 276 test images that were analyzed in Section 2.3. The instruction
requests VLMs to begin by describing the image, and then deduce the most
advanced disease stage:

Describe the OCT image in detail and list any biomarkers or abnormalities,
including the most likely AMD stage of the patient. Then, based on those
observations, state if the patient’s most advanced AMD stage is ’healthy’,
’early’, ’intermediate’, ’late dry’, ’late wet (inactive)’ or ’late wet
(active)’?

After the VLM generated its report (using a maximum of 500 tokens), we
appended the following text to its response.

Based off the image and those findings, the patient’s most advanced AMD stage
is

before continuing generation for another 300 tokens. From these tokens, we
extracted the final disease staging prediction by searching for the first
instance of any of the listed disease stages. This post-processing step is
only necessary for quantitative tests of accuracy, as it enables the reliable
extraction of the disease stage from the free text report. In cases where the
VLM discusses multiple disease stages, such as in ‘more advanced than early
AMD, and is intermediate AMD as there is no evidence of late wet AMD’, the
disease stage was manually extracted. In cases where no disease stage was
provided or could be extracted manually, this counted as an ‘Invalid
response’.

####  5.8.2 Evaluations of correctness, completeness and conciseness by senior
ophthalmologists

For the direct evaluation by the senior ophthalmologists, we randomly selected
28 of the test images from the retrospective dataset, and tasked the two
junior ophthalmologists with annotating the images. We provided the following
instruction to RetinaVLM-Specialist and LLaVA-Med to generate their image
reports:

Write an extensive report describing the OCT image, noting any biomarkers or
abnormalities related to AMD, and their qualities. Also comment on which
biomarkers are absent. Finally, based on the image and these findings, your
report should estimate the AMD disease stage of the patient. You should not
include any patient referral recommendations in your report, but you can
comment if they need treatment with anti-vegf.

We found that LLaVA-Med tended to write excessively long reports. To improve
the performance of LLaVA-Med, we adjusted the instruction to request a ‘brief
report’ instead of an ‘extensive report’. Even after this change, LLaVA-Med
still produced very long and inaccurate reports.

We then assigned 14 of the images and the corresponding 42 reports by LLaVA-
Med, RetinaVLM-Specialist and the junior ophthalmologists, to each of the two
senior ophthalmologists, who evaluated them independently according to the
correctness, completeness and conciseness. These criteria, also documented in
Figure 4, were:

  * –

Correctness - The report is accurate in its main observations and conclusions
regarding the image

  * –

Completeness - The report contains all relevant observations and conclusions
that can be inferred from the image

  * –

Conciseness - The report does not make observations and conclusions that are
not supported by, or not seen in, the image

####  5.8.3 Tests of patient referral

The following instruction was given to all VLMs to generate reports that focus
on patient referral recommendations, which are analyzed in Section 2.3. This
instruction was run for the 95 referral images, introduced in Section 5.1.2.
In order to accurately convey the specific requirements of the wet AMD
treatment clinic, we provided the comprehensive referral protocol used by the
senior ophthalmologists in the instruction:

Write an extensive report describing the OCT image and listing any present
biomarkers or other observations. Do not provide a disease stage, or referral
recommendation yet. Being seen by a specialist at the Southampton clinic: A.
The Southampton clinic requires that patients with any sign of intraretinal
fluid, any sign of subretinal fluid, or any sign of cyst(s), MUST be seen by a
specialist at the Southampton clinic within the next two weeks. B. The
Southampton clinic requires that patients who do not have any sign of
intraretinal fluid, any sign of subretinal fluid, or any sign of cyst(s), but
do have some biomarkers of early or intermediate AMD, should be seen by a
specialist at the Southampton clinic for routine referral. C. The Southampton
clinic requires that patients who do not have any sign of intraretinal fluid,
any sign of subretinal fluid, or any sign of cyst(s), but do have medium to
large drusen, drusenoid PED, hypertransmission or atrophy, should be seen by a
specialist at the Southampton clinic for routine referral. D. The Southampton
clinic does not need to see patients who have no biomarkers and healthy
retinas at all. Southampton specialist visit: Next, tell me if your initial
report of the OCT image indicates that the patient should be seen by a
specialist at the Southampton clinic "within the next two weeks", to be seen
"within 18 weeks (routine referral)", or "not be seen" at all?

As before, after the VLM generated its report (using a maximum of 500 tokens),
we added the following text to its output:

My report indicates that the patient

before continuing generation for another 300 tokens. We then searched these
tokens for the first instance of one of the three levels of referral urgency,
‘within the next two weeks’, ‘within 18 weeks (routine referral)’ or ‘not be
seen’, in the VLM’s output report.

####  5.8.4 Tests of biomarker analysis

The following instruction was given to all VLMs to generate reports that
conclude the presence of absence of the 10 different biomarkers, evaluated on
the 396 test images in Section 2.5:

Describe the OCT image in detail and list all biomarkers or abnormalities.
Detail if there are any signs indicating that {biomarker} might be present,
even if there is only a small amount. Finally, conclude your findings by
telling me if {biomarker} {article} "not present", or if potentially any
amount of {biomarker} {article} "present" in the OCT image.

For each of the 10 biomarkers, the phrase {biomarker} was replaced by the
actual biomarker name (such as ‘subretinal fluid’), and the {article} replaced
by is for singular biomarkers or are for plural biomarkers (such as drusen).
After the VLM generated its report (using a maximum of 500 tokens), we added
the following text to its output.

To conclude these findings, in the OCT image {biomarker} {article}

before continuing generation for another 300 tokens. We then searched for the
first instance of not present or present to extract the model’s prediction of
the absence or presence of the biomarker, respectively.

####  5.8.5 Computing language-based image saliency maps

We provide methodogolical details for the computation of the language-based
saliency maps discussed in Section 2.5, and shown in Figure 6d and Figure 14.
With saliency maps we aim to identify which regions of the image were most
relevant to certain passages, such as large subretinal fluid, of RetinaVLM-
Specialist’s responses. The most direct way to generate these visualizations
to use attention maps, but we found Llama3’s pretrained attention maps did not
result in any meaningful saliency maps. To address this, we used Grad-CAM
[22], a technique for highlighting the most relevant image regions to the
prediction of an image classifier. By defining the predicted class as the sum
over the tokens in the output passage, which formulates the LLM as an image
classifier, we were able to generate the saliency maps shown in Figure 6. A
code implementation can be found at the repository referenced in Section 7.

###  5.9 Measurements of performance and statistical analysis

To calculate the performance of each VLM in multiple-choice question
answering, we used the micro F1 score. This aggregates the total number of
false positives (FP), false negatives (FN), true negatives (TN) and true
positives (TP) over all classes before computing the F1 score using equation 1

$F_{1}=\frac{2\cdot TP}{2\cdot TP+FP+FN}$ (1)

In cases where the VLM returned an ’Invalid response’ this was counted as a
false negative for the ground truth class. After calculating the F1 score, we
determined the 95% confidence interval through bootstrapping $N=1000$ times
with replacement.

Tests of significance (aggregated in Figure 12) were calculated using a two-
sided McNemar’s test [39]. This test assesses the difference in the number of
correctly versus incorrectly predicted samples, focusing on cases where the
models agree or disagree on the labels. A significant p-value from the McNemar
test allows us to reject the null hypothesis that both models have identical
classification performance. We then used the following notation to indicate
levels of statistical significance: *** for $p\leq 0.001$, ** for $p\leq
0.01$, and * for $p\leq 0.05$ and ‘ns’ (not significant) for $p>0.05$.

###  5.10 Computing hardware and software

We use Python 3.12.2 to conduct all model question-answer generation, VLM
training, and VLM evaluation. To generate the question-answer pairs for
curriculum part 1 we used 3 40GB NVIDIA A40 GPUs. For both training RetinaVLM
and for evaluating all VLMs we use a single 80GB NVIDIA A100 GPU and PyTorch
[40] version 2.1.2. Training RetinaVLM on takes 1 day on curriculum part 1,
and another day on curriculum part 2. Llama3 was downloaded via Huggingface
with model ID ‘meta-llama/Meta-Llama-3-8B-Instruct’. The baseline VLM Med-
Flamingo’s code and model weights were installed following the instructions at
https://github.com/fastscience-ai/MedFlamingo, and LLaVA-Med’s from
https://github.com/microsoft/LLaVA-Med. Confusion matrices and results
calculations were computed with scikit-learn version 1.4.1 and numpy version
1.26.4. Figures and tables were created in draw.io v24.4.0 using plots
generated by matplotlib version 3.8.4 and seaborn version 0.13.1. Grad-CAM was
computed using grad-cam version 1.5.0. McNemar’s tests of significance were
calculated using statsmodels version 0.14.1.

##  6 Data availability

Both imaging datasets are currently being curated and maintained by the Vienna
Reading Center on behalf of the PINNACLE consortium. The data will be made
available to once the PINNACLE study concludes in 2026 [41].

##  7 Code availability

The code used to create the question-answer pairs, train, and evaluate the
models will be uploaded to https://github.com/RobbieHolland/SpecialistVLMs.
The code may be used to develop the models can be repurposed for other medical
specialties. The model weights for RetinaVLM-Base and RetinaVLM-Specialist are
both available at https://huggingface.co/RobbieHolland/RetinaVLM. These are
only intended for research purposes related to retinal OCT images.

##  A Supplementary material

###  A.1 Question-answer generation prompts (introduction to retina)

The limited scope of the first part of the tabular reports led us to use only
one prompt (see Section A.1.1) to create the 408,545 question-answers in
curriculum part 1, ‘introduction to retina’.

####  A.1.1 General question-answering module

I am constructing a dataset to train a model to answer questions based solely
on OCT images.

The model will access ONLY the image to deduce attributes. For context, the
image in question has attributes as follows:

<ReportText>

Based on these attributes, generate a numbered list of diverse questions and
answers.

Ensure the format is:

1\. Q: [Question about an image attribute]

A: [Specific answer deduced from the image]

Rules:

\- Questions should be crafted in a way that they don’t explicitly state the
attribute values, but the answers should be based on them. All attributes can
be determined from the image.

\- Incorporate both yes/no and open-ended styled questions, but always provide
a definitive answer in the answer section.

\- Occasionally touch on patient outcome/treatment.

###  A.2 Question-answer generation prompts (advanced retinal specialism)

Creating a large quantity and variety of question-answer pairs was important
for preventing overfit when finetuning on the 330 images that constitute
curriculum part 2. The prompts each focus on a different aspect of question
answering and report writing, but are expected to have some overlap in the
question-answer pairs they create. The six modules used to generate up to 230
question-answers per image report, as shown in Figure 2, are covered by the
following 10 prompts:

  1. 1.

Advanced biomarkers (Up to 30 total question-answers)

     * –

Up to 30 question-answer pairs specific to biomarkers (see Section A.2.1)

  2. 2.

Disease staging definitions (Up to 50 total question-answers)

     * –

Up to 20 question-answers that use the observation and staging guidelines (see
Section A.2.2)

     * –

Up to 30 question-answers that train the model to replicate any reasoning
linking observations to the disease stage that is present in the report (see
Section A.2.3)

  3. 3.

Staging reasoning (Up to 60 total question-answers)

     * –

Up to 20 question-answers that use the observation and staging guidelines to
train the model to explain the diagnosed disease stage in terms of the visible
biomarkers (see Section A.2.4)

     * –

Up to 40 question-answers that focus on training the model in differentiating
active from inactive late wet AMD (see Section A.2.5)

  4. 4.

Referral reasoning (Up to 25 total question-answers)

     * –

Up to 25 question-answer pairs that focus on training the model to reason
about different levels of urgency in patient referral (see Section A.2.6)

  5. 5.

General visual question-answering (Up to 30 total question-answers)

     * –

Up to 15 question-answers created directly from the report, but specific to
one attribute (see Section A.2.7)

     * –

Up to 15 question-answers created directly from the report, but including
longer and more open-ended questions A.2.8)

  6. 6.

Report writing (Up to 35 total question-answers)

     * –

Up to 15 question-answers that focus on replicating the specialist’s report
(see Section A.2.9)

     * –

Up to 20 question-answers that all three observational, disease staging and
patient referral guidelines to train the model to create more sophisticated
reports A.2.10)

####  A.2.1 Advanced biomarkers module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. Below are guidelines outlining what can appear in an OCT image:
<ObservationGuidelines> However, the image the model is being asked about is
characterised by the following description: DESCRIPTION OF IMAGE:
"<ReportText>" Task: Write 30 varied questions and answers that ask the model
about the image. Ensure the format is: 1\. Q: [Question or statement to
describe the image] A: [Augmented version of actual image description] Rules:
\- Ask separately about the presence, amount, location and type of some of the
biomarkers in the guidelines \- Ask about the presence or absence of the
biomarkers in the guidelines \- Rather than saying the image/description does
not specify/mention a biomarker, instead say ’the image does not
show/display/exhibit evidence of’ the biomarker UNLESS its presence is already
implied by another present biomarker. The model does not see the above
description of the image, it’s only given the original image when answering
questions. \- Try not to give away too much information included in the image
description in the question text. The model must learn to use the image to
determine the answer, and not make educated guesses based on the question
alone. \- The answer must be accurate and reflect the same information in the
image description. \- Write nothing except the questions and answers. Tips: \-
Example questions: "Does this image show any subretinal fluid?" "Do you see
any intraretinal fluid?" "Is there any hypertransmission? "Does the image show
a PED?" \- Answer style variation: Finally, do not start too many questions
with "No, ..." or "Yes, ...". Vary the answer style (the biomarker is ’not
present’, ’is shown’, ’exhibits no’, ’does contain’ etc...) \- Positive and
negative balance: Try to include an even balance of questions with positive
responses (i.e. ask the model about each of the biomarkers that were reported
in the description) and negative responses (i.e. that biomarker is not
present) To do this, if you create a question about a biomarker that isn’t in
the description, try to create a second, similar question about a biomarker
that is observable in the image. In order to make the question set not give
too much away about the image, you can make paired questions which have
positive and negative responses. For example, for an image with a PED but no
subretinal fluid, if you ask f.e. "Q: Is there a PED? If so, where? A: There
is a PED present, it’s in the center..." you should also create a question
with a negated answer f.e. "Q: Is there any subretinal fluid? And in what
quantity? A: There is no sign of subretinal fluid in the image...". This will
help you keep an even balance of positive and negative responses, so that the
model cannot guess the answer to the question without considering the image.

####  A.2.2 Disease staging definitions modules

I am constructing a dataset to train a model to answer questions based solely
on OCT images. Below are guidelines outlining what can appear in an OCT image:
<ObservationGuidelines> <DiseaseStagingGuidelines> However, the image the
model is being asked about is characterised by the following description:
DESCRIPTION OF IMAGE: "<ReportText>" Task: Write 20 varied questions and
answers that ask the model about the image. Ensure the format is: 1\. Q:
[Question or statement to describe the image] A: [Augmented version of actual
image description] Rules: \- Information about the image should not be in the
question. \- The answer must be accurate and reflect the same information in
the image description. \- Write nothing except the questions and answers.
Tips: \- Questions should be specific and ask about certain attributes, or
sets of attributes. For example "Q: Is there any subretinal fluid in this
image?" or "Q: Is the AMD stage intermediate, or is it more advanced?". \- Ask
separately about the presence, amount, location and type of some of the
biomarkers. Try to create an even balance of ’yes and ’no’ answers. \- Rather
than saying the image does not directly/explicitly specify/mention the
presence of an attribute, instead say ’the image does not
exhibit/show/display/evidence’ the attribute, UNLESS its presence is already
directly implied by the presence of another attribute. The model does not see
the above description of the image, it’s only given the original image when
answering questions. \- Sometimes the desired output format should be specific
in the question (f.e. Answer with ’yes’ or ’no’, or answer by stating if the
image ’does’ or ’does not’ contain the biomarker in question.)

####  A.2.3 Disease staging from the report module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. Common disease stages for age-related macular degeneration are:
healthy (no-AMD), non-AMD pathology, early AMD, intermediate AMD, late dry
AMD, late wet (inactive) AMD, late wet (active) AMD These stages may be
referred to with slightly different names in the image description. If the
description doesn’t specify whether the late wet AMD is inactive or active,
then you shouldn’t either (just see how to description refers to it, but don’t
feel the need to copy it verbatim) However, the image the model is being asked
about is characterised by the following description: DESCRIPTION OF IMAGE:
"<ReportText>" Task: Write 30 varied questions and answers that require the
model to estimate the patient’s disease stage from the image. Ensure the
format is enumerated: 1\. Q: [Question or statement to describe the image] A:
[Augmented version of actual image description] Rules: \- If the estimated
disease stage is not provided in the report, do not write ANY questions and
answers. Simply write "No disease stage in report". \- Try not to give away
too much information included in the image description in the question text.
The model must learn to use the image to determine the answer, and not make
educated guesses based on the question alone. \- The answer must be accurate
and reflect the same information in the image description. \- The questions
and answers must vary in their style, formulation and vocabulary. \- Write
nothing except the questions and answers. Tips: \- Most questions should ask
the model to first estimate the disease stage For example, "Decide the most
advanced AMD stage supported by the image, and explain your reasoning by
noting any biomarkers most relevant to that stage." \- If the explanation or
reasoning for the disease stage is given in the report, make sure to include
that in the model’s answers \- Questions 1 to 20 should first ask the model to
decide/estimate/determine/identify/... the (most advanced) disease stage, and
then explain their answer. \- Questions 21 to 30 questions should ask for the
biomarkers and then the disease stage, such as "Describe any
relevant/notable/significant biomarkers, and link them to the most likely
disease stage" (which will be the one in the report)

####  A.2.4 Disease staging reasoning (with guidelines)

I am constructing a dataset to train a model to answer questions based solely
on OCT images. Below is a schema outlining what can appear in an OCT image:
<ObservationGuidelines> <DiseaseStagingGuidelines> However, the image the
model is being asked about is characterised by the following description:
DESCRIPTION OF IMAGE: "<ReportText>" Task: Write 20 varied questions and
answers that require the model to perform chain-of-thought reasoning. Ensure
the format is: 1\. Q: [Question or statement to describe the image] A:
[Augmented version of actual image description] Rules: \- Information about
the image listed in the description should not be used in the question, only
in the answer. \- Never use the word ’description’ or ’mention’ in the answer,
the model does not see the description of the image, it only sees the original
image itself. \- The answer must be accurate and reflect the same information
in the image description. \- Write nothing except the questions and answers.
Tips: \- Some questions should ask the model to provide a long and detailed
answer to questions like ’Describe all the observable biomarkers in the image
and then link these to the most likely disease stage’. \- One or two questions
should ask the model to explain/deduce the highest precedent AMD stage (i.e.
the overall AMD stage) by describing the biomarker(s) in the image which
belong to the most advanced disease stage, and linking them to the relevant
disease stage. \- Some questions should ask the model to summarise any
relevant biomarkers and, explaining its reasoning using the guidelines,
conclude with the AMD stage. For example, a drusenoid PED suggests
intermediate AMD, but if coupled with subretinal fluid the overall AMD stage
becomes active late wet due to the fluid. \- Some questions should ask the
model to fully describe and list all its image observations, and then conclude
the presence, absence, location or quantity of a specific biomarker (f.e.
’Describe the OCT image in detail and note any abnormalities, and then tell me
if the image contains subretinal fluid.’)

####  A.2.5 Disease staging reasoning second module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. Below are guidelines outlining what can appear in an OCT image:
<DiseaseStagingGuidelines> However, the image the model is being asked about
is characterised by the following description: DESCRIPTION OF IMAGE:
"<ReportText>" Task: Write 40 varied questions and answers that require the
model to estimate the patient’s disease stage from the image. Ensure the
format is enumerated: 1\. Q: [Question or statement to describe the image] A:
[Augmented version of actual image description] Rules that always apply: \- If
the estimated disease stage is not provided in the report, do not write ANY
questions and answers. Simply write "No disease stage in report". \- If
reasoning for the disease stage is provided in the report, you MUST include
this logic/nuance in the model’s answers. \- You MUST not give away
information about the image description in the question text. The model must
learn to use the image to determine the answer, and not make educated guesses
based on the question alone. \- The answer must be accurate and reflect the
same information in the image description. \- The questions and answers must
vary in their style, formulation and vocabulary. \- Write nothing except the
questions and answers. Rules that are specific to differentiating cases of
active, vs inactive, late wet AMD: \- In cases with an active late wet
diagnosis, you MUST make it clear that the PRESENCE OF FLUID is what
differentiates active from inactive late wet AMD. Make this clear when
explaining the reasoning behind active late wet diagnoses. So do not imply
that f.e. "The disease stage is late wet AMD (active), as indicated by the
subretinal/intraretinal fluid, subretinal hyperreflective material, ...."
Instead, you MUST explain that f.e. "The overall disease stage is late wet
AMD, which is active due to the detection/presence of subretinal/intraretinal
fluid. Inactive late wet features include fibrovascular PED, ..." Or "Late wet
AMD best describes the AMD stage. The detection/presence of
subretinal/intraretinal fluid means this is active late wet AMD. Other late
wet features include fibrovascular PED, ..." \- Similarly, in cases with an
inactive late wet diagnosis, you MUST make it clear in the model’s reasoning
that the lack of fluid, in combination with the other biomarkers, is what
resulted in the inactive late wet diagnosis. For example, "The stage is late
wet AMD according to the evidence/presence of subretinal hyperreflective
material, but it is inactive as there is no detectable fluid of any kind in
the image" \- The exact formulation of this answer must vary according to the
question. Do not copy the examples too many times. Add a lot of diversity in
the model’s responses. Tips: \- Most questions should ask the model to first
estimate the disease stage \- Some questions should first ask the model for
its disease stage prediction, and then ask it to explain its reasoning by
noting visible biomarkers that relate to that stage For example, "Decide the
most advanced AMD stage supported by the image, and explain your reasoning by
noting any biomarkers most relevant to that stage." \- The explanation for the
disease stage may already be given in the report, but you can also use the
guidelines provided to work out which biomarkers resulted in that disease
stage. \- Questions 31 to 40 questions should ask for the biomarkers and then
the disease stage, such as "Describe any relevant/notable/significant
biomarkers, and link them to the most likely disease stage" (which will be the
one in the report)

####  A.2.6 Patient referral reasoning module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. Below are guidelines outlining what can appear in an OCT image:
<DiseaseStagingGuidelines> <PatientReferralGuidelines> However, the image the
model is being asked about is characterised by the following description:
DESCRIPTION OF IMAGE: "<ReportText>" Task: Write 25 varied questions and
answers that require the model to perform explain its reasoning before making
conclusions and recommendations. Ensure the format is: 1\. Q: [Question or
statement to describe the image] A: [Augmented version of actual image
description] Rules: \- Information about the image should not be in the
question. \- Never use the word ’description’ or ’mention’ in the answer, the
model does not see the description of the image, it only sees the original
image itself. \- The answer must be accurate and reflect the same information
in the image description. \- Write nothing except the questions and answers.
\- The model sees each questions separately so they will not be seen together.
Tips: \- Some questions should ask the model to list any relevant biomarkers
and, based on these, recommend if the patient should be referred or not. \-
Many questions should make a series of requests, by asking the model to write
a long and detailed answer reporting all the observable biomarkers, linking
those to the most likely disease stage and then summarising the report with a
patient referral recommendation. For example, ’Describe the all biomarkers in
the image, and based off of your observations which AMD best describe the
patient. Summarise your report with a referral recommendation that follows the
treamtent guidelines.’ \- Some questions should ask the model to recommend if
the patient does: not need referral, if they need general attention by a
specialist, or if they likely need treatment with anti-vegf based off the
models observations \- Some questions should ask the model to
explain/deduce/estimate the patient’s risk (i.e. the recommended referral
action) by first describing the most advanced or concerning biomarker(s) (i.e.
the observable biomarker(s) which belong to the most severe disease stage)

####  A.2.7 Specific report-based questions module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. The image in question is characterised by the following
description: DESCRIPTION OF IMAGE: "<ReportText>" Task: Write 15 varied
questions and answers that ask the model about the image. They should Ensure
the format is: 1\. Q: [Question or statement to describe the image] A:
[Augmented version of actual image description] Rules: \- Try not to give away
too much information included in the image description in the question text.
The model must learn to use the image to determine the answer, and not make
educated guesses based on the question alone. \- The answer must be accurate
and reflect the same information in the image description. \- Write nothing
except the questions and answers. Tips: \- Questions should be specific and
ask about certain attributes, or sets of attributes. For example "Q: Is there
any subretinal fluid in this image?".

####  A.2.8 General report-based questions module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. The image in question is characterised by the following
description: DESCRIPTION OF IMAGE: "<ReportText>" Task: Write 15 varied
questions and answers that ask the model about the image. Ensure the format
is: 1\. Q: [Question or statement to describe the image] A: [Augmented version
of actual image description] Rules: \- The answer should be a modified and
augmented version of the actual description. \- Try not to give away too much
information included in the image description in the question text. The model
must learn to use the image to determine the answer, and not make educated
guesses based on the question alone. \- The answer must be accurate and
contain the same information as the actual description. However, the order of
the sentences as they are written must change and randomly vary. \- Write
nothing except the questions and answers. Tips: \- Some question should be
general and ask to describe the image. \- Other questions should be more
specific such as "Q: Is there any subretinal fluid in this image?" that have
shorter answers. \- Example questions/statements might be "Describe the OCT
image in detail." or "Can you give me a summary of the image?".

####  A.2.9 Basic report writing module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. The image in question is characterised by the following
description. DESCRIPTION OF IMAGE: "<ReportText>" Task: Write 15 questions and
answers that ask the model to describe the image in full. The first ten
questions should ask to describe the entire image (with
jumbled/permuted/randomised answers), while the final five should be more
specific or use segments of the description. Ensure the format is: 1\. Q:
[Question or statement to describe the image] A: [Augmented version of actual
image description] Rules: \- The answer should be a modified and augmented
version of the actual description. \- The answer should contain the same
information as the actual description but the order of the sentences as they
are written must change and randomly vary \- The question must not contain any
information about the image. \- Write nothing except the questions and
answers. Tips: \- Example questions/statements might be "Describe the OCT
image in detail." or "Can you give me a summary of the image?" or "Write a
report on this image to be given to an optometrist.".

####  A.2.10 Advanced report writing module

I am constructing a dataset to train a model to answer questions based solely
on OCT images. Below are guidelines outlining what can appear in an OCT image:
<ObservationGuidelines> <DiseaseStagingGuidelines> <PatientReferralGuidelines>
However, the image the model is being asked about is characterised by the
following description: DESCRIPTION OF IMAGE: "<ReportText>" Task: Write 20
requests for reports that ask the model reason all the way from the observable
biomarkers, linking these to the most advanced disease stage, and finally to a
referral recommendation for the patient. Ensure the format is: 1\. Q: [Request
to the model] A: [Answer or report deduced using actual image description]
Rules: \- Never use the word ’description’ or ’mention’ in the answer, the
model does not see the description of the image, it only sees the original
image itself. \- Try not to give away too much information included in the
image description in the question text. The model must learn to use the image
to determine the answer, and not make educated guesses based on the question
alone. \- Write nothing except the questions and answers. \- The model sees
each questions separately so they will not be seen together. Tips: \- Example
questions/statements might be "Describe the OCT image in detail." or "Can you
give me a summary of the image?" or "Write a report on this image to be given
to a retinal specialist.". \- Ask the model to explain why the AMD stage is
not more advanced than it is (i.e. the absence of certain late stage
biomarkers) \- Ask the model to explain why the AMD stage is more advanced
than an earlier stage (i.e. the presence of certain late stage biomarkers) \-
A few times, request verbose reports like "Write a report that starts by
highlighting the most significant and salient biomarkers, and link these to
the most probable disease stage. Conclude with a referral recommendation."