#  Dr-LLaVA: Visual Instruction Tuning with
Symbolic Clinical Grounding

Shenghuan Sun∗
UCSF
&Gregory M. Goldgof∗
MSK Cancer Center
&Alexander Schubert
UC Berkeley & UCSF
&Zhiqing Sun
CMU
&Thomas Hartvigsen
University of Virginia
&Atul J. Butte
UCSF
&Ahmed Alaa
UC Berkeley & UCSF

###### Abstract

Vision-Language Models (VLM) can support clinicians by analyzing medical
images and engaging in natural language interactions to assist in diagnostic
and treatment tasks. However, VLMs often exhibit “hallucinogenic” behavior,
generating textual outputs not grounded in contextual multimodal information.
This challenge is particularly pronounced in the medical domain, where we do
not only require VLM outputs to be accurate in single interactions but also to
be consistent with clinical reasoning and diagnostic pathways throughout
multi-turn conversations. For this purpose, we propose a new alignment
algorithm that uses symbolic representations of clinical reasoning to ground
VLMs in medical knowledge. These representations are utilized to (i) generate
GPT-4-guided visual instruction tuning data at scale, simulating clinician-VLM
conversations with demonstrations of clinical reasoning, and (ii) create an
automatic reward function that evaluates the clinical validity of VLM
generations throughout clinician-VLM interactions. Our algorithm eliminates
the need for human involvement in training data generation or reward model
construction, reducing costs compared to standard reinforcement learning with
human feedback (RLHF). We apply our alignment algorithm to develop Dr-LLaVA, a
conversational VLM finetuned for analyzing bone marrow pathology slides,
demonstrating strong performance in multi-turn medical conversations.

Code: https://github.com/AlaaLab/Dr-LLaVA

##  1 Introduction

Large language models (LLMs) exhibit impressive capabilities such as
generating human-like dialogue and answering complex questions [1, 2, 3, 4,
5]. When combined with vision encoders and fine-tuned using text-image pairs
or visual instruction tuning datasets, these models evolve into (large)
vision-language models (VLMs) [6, 7, 8], which can conduct tasks involving
visual reasoning and grounding [9]. In the medical domain, VLMs hold great
promise—they could serve as helpful assistants for clinicians, researchers,
and trainees, providing an interactive natural language interface for the
analysis of medical images within clinical workflows [10, 11, 12, 13, 14, 15].
Despite their promise, VLMs are limited by their tendency to “hallucinate”
[16, 17, 18]. As noted in [18, 7], the significantly smaller amount of
multimodal training data compared to text-based data can cause a misalignment
between vision and language inputs, leading VLMs to generate answers that are
ungrounded in the visual context. This issue is even more pronounced in the
medical domain, where data is naturally scarce and clinician-VLM image-based
dialogues are harder to synthesize. The problem of finetuning VLMs to mitigate
hallucinatory responses in the multimodal medical context remains largely
unaddressed.

Reinforcement Learning from Human Feedback (RLHF) has been shown to be a
promising method to reduce hallucinations and align LLM outputs with human
preferences [19, 20, 21]. In RLHF, human annotators evaluate the quality of
various LLM outputs, and this feedback is used to train a reward model, which
is then employed to finetune the LLM. In [18], this approach was adapted for
finetuning VLMs by collecting human feedback on their responses in synthesized
single-turn visual QA tasks.

However, finetuning VLMs for visually-grounded diagnostic assistance involves
more than just handling single-turn QA tasks and addressing the standard
notion of hallucination. Instead, we seek models capable of engaging in multi-
turn interactions with clinicians, consistently demonstrating valid and
coherent clinical reasoning throughout the conversation. In this context,
hallucination occurs not only when a model provides responses not grounded in
the visual input but also when its responses during the interaction are
incoherent, contradictory, or fail to logically align with diagnostic pathways
and domain knowledge. For instance, in the exemplary conversation in Fig. 1,
hallucination resulting from poor visual grounding will reflect in the model
failing to identify the proliferating cell types in the provided pathology
slide. However, even if the model correctly identifies the cell types, it must
also answer follow-up questions from clinicians in a manner consistent with
domain knowledge and its own previous responses.

Using RLHF to align VLMs with visually-grounded clinical reasoning requires
multimodal training data showcasing the reasoning process within multi-turn QA
dialogues. These datasets are not readily available in health systems.

Figure 1: Symbolic clinical grounding. We utilize symbolic representations of
diagnostic workflows to align multi-turn clinician-VLM conversations with
valid clinical reasoning.

Synthesizing these datasets and collecting clinician feedback on VLM responses
is bottle-necked by the expertise of medical professionals. Unlike the LLaVA-
RLHF model in [18], which gathered human feedback from non-expert crowdworkers
for simple, common-sense visual QA tasks, this process cannot be scaled
without the involvement of clinicians. Due to these limitations, specialized
medical VLMs like LLaVA-Med [10] and PathChat [7] have been confined to
supervised finetuning, relying on automatically generated QA tasks using image
captions. Moreover, both existing general-purpose and medical VLMs have only
been finetuned for single-turn QA, rather than for multi-turn conversations
that convey complex and interactive clinical reasoning.

Summary of contributions. In this paper, we introduce a novel finetuning
algorithm for adapting VLMs to imaging-based conversational diagnostic tasks,
which eliminates the need for human involvement in training data generation or
feedback collection. The core idea of our algorithm is to represent the
clinical reasoning process for specific diagnostic tasks using a predefined
set of symbolic rules. For instance, in the example shown in Fig. 1, these
symbolic rules can enumerate all possible cell types and corresponding
diagnoses for blood malignancies. Our algorithm leverages these symbolic rules
to automatically generate realistic VLM-clinician conversations and to
evaluate a reward function that assesses the alignment of VLM responses with
these rules. Our algorithm comprises three steps:

Step 1 (Synthesizing clinician-VLM conversations): Starting with labeled
medical images, we use symbolic representations of clinical reasoning and
GPT-4 to generate realistic conversations between a VLM and a clinician about
the visual content of each image. These multi-turn conversations are designed
to reflect various styles of clinician-VLM interactions, with each
conversation comprising a sequence of related questions that demonstrate
accurate clinical reasoning.

Step 2 (Designing clinically-informed symbolic rewards111In our framework, the
term “symbolic reward” refers to the alignment between a sequence of model
responses and symbolic rules that represent valid clinical reasoning. This
usage differs from that in [18], where the term “symbolic reward” was used to
describe a correctness penalty and length penalty in a different context.): We
use the training data generated in Step 1 to finetune VLMs. Instead of relying
on human feedback to ensure the model responses align with correct clinical
reasoning, we use the same symbolic representations of clinical reasoning used
in generating the training data to evaluate a symbolic reward. This approach
allows us to automatically evaluate a reward function by parsing the VLM
textual responses in a multi-turn conversation and verifying their consistency
with the symbolic rules defining correct clinical reasoning.

Step 3 (Finetuning the VLM for clinical correctness & consistency): We propose
a new finetuning loss function that assesses both the correctness and
consistency of VLM responses. Correctness is evaluated with a standard
supervised loss, while consistency is evaluated using the symbolic reward
introduced in Step 2. This finetuning objective rewards the model not only for
producing correct answers to individual clinician queries but also for
demonstrating logically consistent clinical reasoning that aligns with medical
domain knowledge throughout the entire conversation.

We demonstrate the utility of our proposed algorithm in image-based diagnostic
tasks by developing Dr-LLaVA, a VLM for diagnosing blood cancer through
analysis of bone marrow images, by fine-tuning the LLaVA model [7]. To achieve
this, we compiled a dataset of 16,340 bone marrow image patches along with
corresponding multi-turn clinician-VLM conversations. Our results show that
Dr-LLaVA outperforms state-of-the-art VLMs in both single- and multi-turn
conversational settings. Moreover, incorporating the symbolic rewards led to a
10% increase in diagnostic accuracy and reduced the model hallucination rate
by a factor of four. Additionally, fine-tuning with the symbolic rewards
enabled Dr-LLaVA to outperform other baselines in identifying and correcting
erroneous information in clinician prompts. These findings underscore the
value of integrating clinical domain knowledge into fine-tuning approaches
using a hybrid symbolic and data-driven method, thereby developing trustworthy
and accurate conversational assistants in medicine.

Figure 2: Depiction of our symbolic representation of clinical reasoning in
blood cancer diagnosis.

##  2 Visual Instruction Tuning with Symbolic Clinical Grounding

Multimodal conversational datasets do not naturally occur in current clinical
practice. Constructing such datasets requires conducting real experiments with
deployed VLMs, and finetuning models using RLHF requires collecting expert
feedback on the quality of VLM responses, a process that is prohibitively
expensive and impractical. We address this challenge by leveraging the fact
that many medical diagnostic processes can be described using a relatively
small number of logical rules applied sequentially. Based on this insight, we
build a symbolic representation of clinical reasoning within the diagnostic
process. This representation serves as the core engine for synthesizing our
instruction-tuning dataset and generating automatic feedback on VLM responses.

Symbolic representation of clinical reasoning. Fig. 2(a) presents a symbolic
representation of a decision tree, constructed and adjudicated by an expert
pathologist, that outlines each step in the process for diagnosing blood
cancer based on bone marrow pathology slides. Typically, a medical pathologist
starts by analyzing the image quality, then checks if the image patch contains
enough nucleated cells for diagnosis. Following this, they assess the image
for abnormalities and cell proliferation to arrive at a final diagnosis. Any
valid analysis must conform to this process. For example, it would be
contradictory if a bone marrow pathology slide were deemed too low in quality
for further assessment, yet the model proceeded to use it for diagnosis, or if
the identified proliferating cells were incompatible with the final diagnosis.
The decision tree defines a set of symbolic rules that VLM responses must
follow to constitute a valid reasoning process (Fig. 2(b)). Formally, we
define

$\displaystyle\mathcal{S}=\\{$ (Low image quality) $\to$ Inconclusive, (High
image quality $\wedge$ No abnormality) $\to$ Healthy,
$\displaystyle\mbox{\footnotesize(High image quality $\wedge$ Abnormality
$\wedge$ Plasma cell proliferation) $\to$ Multiple Myeloma, \ldots}\\},$ (1)

as the set of all symbolic rules that correspond to valid reasoning,
determined by the representation in Fig. 2(a). We define $\bar{\mathcal{S}}$
as the set of all possible (valid and invalid) rules, i.e.,
$\mathcal{S}\subset\bar{\mathcal{S}}$.

Our instruction tuning framework uses the symbolic representation in Fig. 2
to: (a) synthesize a dataset of visually-grounded clinician-VLM conversations,
(b) evaluate symbolic rewards for VLM responses in synthesized conversations,
and (c) finetune the VLM for clinical correctness and consistency with the
symbolic rules. (See Fig. 3 for a pictorial depiction.) Details of each step
are provided below.

###  2.1 Step 1: Synthesizing clinician-VLM conversations

We synthesize clinician-VLM conversations using a dataset derived from bone
marrow aspirate (BMA) whole slide images, annotated by hematopathologists and
sourced from the clinical archives of an academic medical center. The dataset
includes images indicative of various conditions: blood contamination,
particle-enriched contamination, acute myeloid leukemia, multiple myeloma, and
healthy states. For each image, we use the hematopathologists annotations to
select the symbolic rule from $\mathcal{S}$ that describe the corresponding
diagnostic analysis. We use the selected rule to construct a multimodal
instruction tuning dataset
$\mathcal{D}=\\{(\mathcal{I}_{i},\\{X^{t}_{i},Y^{t}_{i}\\}_{t})\\}_{i}$
consisting of multi-turn clinical conversations
$\\{X^{t}_{i},Y^{t}_{i}\\}_{t}$ associated with each image $\mathcal{I}_{i}$,
where $X_{i}^{t}$ is the $t$-th clinician prompt and $Y_{i}^{t}$ is the
corresponding response of the VLM. These conversations comprise five question-
answer pairs ${(X^{1}_{i},Y^{1}_{i}),\ldots,(X^{5}_{i},Y^{5}_{i})}$ that
follow all components of the corresponding symbolic rule. Each question-answer
pair is generated using textual templates with contents derived from
annotations of image quality, cell types, and disease diagnoses provided by
hematopathologists. To ensure diversity in clinician prompts, we used GPT-4 to
generate a wide range of paraphrased question-answer pairs. Illustration of
the dataset synthesis process is depicted in Fig. 3.

Figure 3: Pictorial depiction of the Dr-LLaVA training pipeline (a) For each
medical image, we generate multi-turn conversations that are consistent with a
symbolic model of clinical reasoning. This process uses GPT-4 to introduce
diversity in the phrasing of the synthesized conversations. (b) We design a
symbolic reward function that translates the responses provided by a VLM into
symbolic rules and checking the correctness of the individual responses as
well as the clinical validity of these rules. (c) Using the dataset in (a) and
the reward model defined in (b), we use an RL approach in order to finetune a
pretrained VLM.

Multimodal Supervised Finetuning. Using the dataset
$\mathcal{D}=\\{(\mathcal{I}_{i},\\{X^{t}_{i},Y^{t}_{i}\\}_{t})\\}_{i}$
generated in step 1, a straightforward approach to adapt VLMs for the
diagnostic task at hand is by applying supervised finetuning. To construct
this baseline, we use the LLaVA architecture [22, 18] and jointly instruction-
tune a vision encoder and a pre-trained LLM using token-level supervision to
derive a supervised fine-tuned (SFT) model $\pi_{\text{SFT}}^{\phi}$.
Following prior work [7, 22], the model is trained based on the LLMs original
autoregressive training objective, where, for an answer sequence of length
$T$, we compute the probability of the target answer as

$p(Y^{t}_{i}|X^{t}_{i},{I}_{i})=\Pi^{T}_{t=1}\pi_{\text{SFT}}^{\phi}(y_{j}|\mathcal{I}_{i},\\{X^{t^{\prime}}_{i},Y^{t^{\prime}}_{i}\\}_{t^{\prime}<t})$
(2)

where $y_{j}$ refers to the current prediction token in the answer sequence
and $\\{X^{t^{\prime}}_{i},Y^{t^{\prime}}_{i}\\}_{t^{\prime}<t}$ refers to the
tokens in the previous parts of the answer sequence.

###  2.2 Step 2: Designing clinically-informed symbolic rewards

In standard RLHF approaches, human feedback is used to evaluate ambiguous
qualities of model outputs, such as helpfulness [18], general human preference
[21], or realism [23]. This feedback helps to align model outputs with user-
preferred qualities that are difficult to define mathematically. For a
conversational diagnostic system, we aim for models that not only provide
accurate diagnoses but also engage in clinical reasoning through conversations
with clinicians. The symbolic representation in Fig. 2 allows us to
automatically evaluate the quality of such conversations. This is achieved by
projecting the model’s arbitrary textual generations onto a low-dimensional
discrete space of a finite number of symbolic rules, and checking if the model
generations are consistent with these rules.

To automate the feedback collection process, we construct a symbolic reward
function $R_{\mathcal{S}}(.)$ that maps the VLM textual outputs
$\\{(X^{t}_{i},\widehat{Y}^{t}_{i})\\}_{t}$ to a symbolic rule
$\hat{s}_{i}\in\bar{\mathcal{S}}$, and then assigns a reward if the rule is
valid, i.e., $\hat{s}_{i}\in\mathcal{S}\subset\bar{\mathcal{S}}$. The function
$R_{\mathcal{S}}(.)$ is implemented using an efficient keyword matching
algorithm that evaluates whether the VLM responses include specific keywords
associated with certain paths in the decision tree in Fig. 2(a), ensuring
these keywords are not used in a negated context. The reward score
$R_{\mathcal{S}}(.)$ assesses the alignment between the VLM sequence of
responses and the valid symbolic rules in $\mathcal{S}$ based on the depth of
the decision tree in Fig. 2(a) traversed by the VLM responses
$\\{\widehat{Y}^{t}_{i}\\}_{t}$. Given the relatively small set of synonyms
for each classification, this method is practical. The detailed list of
keywords used is available in Appendix Table B.2.

We adapt pretrained VLMs to medical diagnostic tasks by using a loss function
that evaluates both the correctness of the model responses and their alignment
with valid clinical reasoning. Given an input image $\mathcal{I}_{i}$ and a
sequence of prompted VLM outputs ${(X^{t}_{i},\widehat{Y}^{t}_{i})}_{t}$, we
compute the reward function as:

$R((\widehat{Y}^{t}_{i},Y^{t}_{i}),\ldots,(\widehat{Y}^{T}_{i},Y^{T}_{i}))=\frac{1}{T}\sum_{t=1}^{T}\underbrace{R_{C}(Y^{t}_{i},\widehat{Y}^{t}_{i})}_{\mbox{\scriptsize{\bf
Correctness} of responses}}+\underbrace{\lambda\cdot
R_{\mathcal{S}}(\\{\widehat{Y}^{t}_{i}\\}_{t})}_{\mbox{\scriptsize{\bf
Cosistency} with valid reasoning}}+R_{l}-R_{m}.$ (3)

Here, $R_{C}$ is a “correctness reward” that evaluates the accuracy of each
individual model response compared to the ground-truth, whereas
$R_{\mathcal{S}}(.)$ is the symbolic reward evaluated for the entire
conversation. The hyperparameter $\lambda$ balances both the correctness and
consistency losses. Small values of $\lambda$ will encourage the model to
optimize its responses to individual questions, potentially disregarding the
coherence of reasoning across responses. Conversely, high values of $\lambda$
will encourage the model to produce a sequence of clinically consistent
responses, even if these responses are not entirely grounded in the actual
visual content of $\mathcal{I}_{i}$. By combining both losses, we integrate an
inductive bias through prior knowledge of valid diagnostic pathways in
$\mathcal{S}$, guiding the model to provide correct and coherent responses
within a sequence of interactions with the clinician.

In addition to the correctness and consistency rewards, we use two additional
penalties to counteract reward-hacking, similar to [18]. First, the penalty
term $R_{m}$ reduces scores for outputs that remain ambiguous and cannot be
clearly categorized for a given analysis step. This penalty is quantified by
the proportion of ambiguous answers within a conversation. Second, to address
the potential for the symbolic rewards to encourage either overly verbose
responses that incidentally include relevant keywords, or overly concise,
keyword-dense answers, we implement a length-based penalty $R_{l}$.
Specifically, $R_{l}$ is set to 1 for responses whose token length deviates by
more than 10 from the token length of the ground-truth answer, to achieve a
balance between brevity and descriptiveness. This approach is designed to
promote a conversational style that is both natural and engaging.

###  2.3 Step 3: Finetuning the VLM for clinical correctness and consistency

Starting with the SFT model $\pi_{\text{SFT}}^{\phi}$, we further finetune the
model using Reinforcement Learning based on the automatically evaluated
symbolic rewards. We consider $\pi_{\text{SFT}}^{\phi}$ as our policy model,
training it to generate accurate responses to clinical queries to optimize the
output of the reward model. Following the approaches in [21, 18], we use
Proximal Policy Optimization (PPO) [24] and integrate a per-token Kullback-
Leibler (KL) penalty to mitigate reward hacking. This penalty restricts the
divergence of the RL-tuned model from the base SFT model. Given a dataset of
medical images, clinical analysis prompts, and their respective answers
$\mathcal{D}_{RL}=\\{(\mathcal{I}_{i},\\{X^{t}_{i},Y^{t}_{i}\\}_{t})\\}_{i}$,
the full finetuning loss is:

$\mathcal{L}(\pi_{\text{RL}}^{\phi})=-\mathbb{E}_{(\mathcal{I},X,Y)\in\mathcal{D}_{RL},\widehat{Y}\sim\pi_{\text{RL}}(\hat{Y}|\mathcal{I},X)}\left[R(\\{\widehat{Y}^{t},Y^{t}\\}_{t})-\beta\cdot
D_{\text{KL}}(\pi_{\text{RL}}^{\phi}(\widehat{Y}|\mathcal{I},X)\|\pi_{\text{SFT}}^{\phi}(\widehat{Y}|\mathcal{I},X))\right].$

Unlike previous RLHF methods [18], our loss function
$\mathcal{L}(\pi_{\text{RL}}^{\phi})$ is computed based on the entire
conversation, since the consistency reward in (3) is evaluated using the
sequence of all model responses.

##  3 Related Work

Vision-Language Models for medicine. Large Language Models (LLMs) [1, 5, 25,
26, 27, 28, 2, 3, 29, 30] have excelled in generating high-quality textual
responses across diverse tasks, fueling advancements in chat-based AI
assistants [4, 31]. Recent work has extended these models to handle multimodal
image-text data [32, 33, 34], which has led to the emergence of powerful
vision-language models (VLM) including OpenFlamingo [35], MiniGPT-4 [36] and
LLaVA [10]. In the medical domain, the integration of images and texts has
been explored in areas such as ultrasound [37, 38], pathology [39, 13], and
radiology [40, 41], typically utilizing modality-specific vision encoders.
Additionally, recent studies have proposed models that directly finetune
state-of-the-art VLMs for medical applications including Med-Alpaca [42], Med-
Flamingo [43] and LLaVAMed [10]. However, these models solely leverage
instruction-tuning with token-level supervision, but do not consider
regularizing the model outputs on a conversation-level by incorporating domain
knowledge on diagnostic pathways.

Hallucination in generative models. In the Natural Language Processing (NLP)
literature, “hallucination” was defined as the phenomenon where a model
generates content diverging from the original source material [44]. With the
advent of advanced LLMs, this definition has expanded. As noted in [45],
hallucination can manifest in three distinct ways: 1) Input-conflicting
hallucination, observed in scenarios like machine translation and
summarization, where the model’s response alters or misinterprets the static
context of the user’s prompt [46, 47, 48, 49]; 2) Context-conflicting
hallucination, where the model’s output contradicts its previous responses
[50, 51]; and 3) Fact-conflicting hallucination, in which the generated
content conflicts with established factual knowledge [52, 16].

To the best of our knowledge, our finetuning framework is the first to address
context-conflicting hallucinations, which are particularly important in
clinical applications [51, 53, 54]. This is because medical practitioners
adhere to stringent logical processes in diagnosis and avoid conclusions that
contradict previous observations [55]. Therefore, a VLM that accurately
identifies the final diagnosis but fails to correctly respond to preceding
observation-related questions would be deemed unreliable [56]. Similar to
prior work, our reward model in (3) addresses input- and fact-conflicting
hallucination, but is distinguished by inclusion of the symbolic reward
$R_{\mathcal{S}}$ to address context-conflicting hallucination.

Addressing misalignment in Vision-Language Models. The two predominant methods
for aligning VLM outputs with specific domain requirements or general human
preferences are supervised finetuning and Reinforcement Learning from Human
Feedback (RLHF). Similar to LLMs, supervised finetuning typically involves
training a pre-trained model on a dataset tailored to the task at hand [57,
58, 22, 8, 59]. However, this approach can lead to misalignment between image
and text modalities in VLMs, resulting in outputs insufficiently grounded in
the visual context [18]. Conversely, RLHF has proven effective in
recalibrating models to match human preferences. This method relies on
preference data from human labelers to train a reward model, which then
finetunes the VLM using reinforcement learning techniques such as Proximal
Policy Optimization (PPO) [24]. In our work, which, to the best of our
knowledge, is the first to apply RL-based finetuning to VLMs for the medical
domain, we introduce a new RL framework tailored to the medical decision-
making contexts by using an automatic reward function to reduce the reliance
on expensive specialist annotators.

##  4 Experiments

We use our finetuning algorithm to develop Dr-LLaVA, a conversational VLM
specialized in analyzing bone marrow pathology slides. In this Section, we
describe our training process and evaluation setup, and compare the
performance of Dr-LLaVA with state-of-the-art VLMs in diagnosing blood cancer.

###  4.1 Experimental set-up

Training details. As our study concentrates on the performance of the
finetuning algorithm, we base Dr-LLaVA on the same model architecture as LLaVA
[7]. Our LLM utilizes Vicuna-V1.5-7b [2, 3, 30], paired with the pre-trained
CLIP visual encoder ViT-L/14 at an image resolution of $256\times 256$ [60].
Grid features are employed both before and after the final transformer layer
to enhance the model’s integration of visual data. We use a linear layer to
map image features into the word embedding space, drawing on the pre-trained
linear projection matrix checkpoints from LLaVA. We then conducted supervised
fine-tuning for four epochs.

During the RL phase, following [61] and [18], we initialized the value model
based on the LLavA-13B-based reward model. We used LoRA-based finetuning with
a rank of 64 for both the attention and feed-forward network modules.
Consistent with [61], we used a batch size of 512 and normalized the advantage
across the batch for each PPO step. The peak learning rate was set at $3\times
10^{-5}$, applying cosine decay, and gradients were clipped by their Euclidean
norm with a threshold of 1. Training was conducted through four complete
rounds using our held-out RL data. For generalized advantage estimation, we
set both $\lambda$ and $\gamma$ to 1, and adopted a constant KL regularizer
coefficient of 0.1. The Dr-LLaVA model was trained using four A100 80 GB GPUs.

We leverage 80% of our synthesized clinical multi-turn conversation dataset
for supervised finetuning and RL and use the remaining 20% for evaluation. We
split the data at the conversation level such that all question-answer pairs
pertaining to a particular image belong to the same sample. We use different
prompt templates and rephrasing for the question-answer pairs in the training
and testing sets to ensure that the models do not over-fit to specific
phrasing of the clinician-VLM conversations

Baselines. We evaluated Dr-LLaVA against state-of-the-art VLMs including the
LLaVA [7], OpenFlamingo [35], MiniGPT-4 [36], and LLaMA-Adapter models [58].
Given the poor zero-shot performance of these models in this specialized
domain, all models were supervised finetuned on our synthesized conversational
dataset over four epochs before evaluation on the test set. To mitigate the
risk of overfitting to specific conversational patterns, we also introduced a
baseline, LLaVA-SFT(p), which is trained using conversations with randomly
permuted ordering of question-answer pairs.

Evaluation metrics. We assess model performance using three metrics: Question-
level Accuracy ($A_{Q}$), Conversation-level Accuracy ($A_{C}$), and
Diagnostic Accuracy ($A_{D}$). $A_{Q}$ is the accuracy of each question-answer
pair independently and calculate the overall proportion of questions answered
correctly across all conversations. $A_{C}$ is the proportion of conversations
where all questions were answered correctly by the model. Finally, $A_{D}$
evaluates the model ability to make a correct diagnosis, regardless of the
quality of its responses in the preceding steps of the image analysis process.

| Metrics
---|---
Baseline | $A_{Q}$ | $A_{C}$ | $A_{D}$
LLaVA-0-shot [7] | 16.5 | 0.0 | 12.6
OpenFlamingo-SFT [35] | 60.5 | 31.3 | 55.8
LLaMA-Adapter-SFT [58] | 68.6 | 37.8 | 70.2
MiniGPT-4-SFT [36] | 64.1 | 32.9 | 50.0
LLaVA-Med-SFT [10] | 78.2 | 55.6 | 76.5
LLaVA-SFT [7] | 77.4 | 47.6 | 77.3
LLaVA-SFT(p) [7] | 80.8 | 57.0 | 73.3
Dr-LLaVA | 89.6 | 70.0 | 84.7

Table 1: Performance in single-turn conversations.

Model | Metric | Experiments
---|---|---
CQ-R |  CQ-W |  RQ-R |  RQ-W
LLaVA-SFT | AQ |  99.2 |  13.8 |  91.5 |  33.3
AD |  99.3 |  13.6 |  99.8 |  31.3
LLaVA-SFT(p) | AQ |  99.6 |  19.6 |  93.8 |  39.7
AD |  99.4 |  20.5 |  98.8 |  41.5
Dr-LLaVA | AQ |  99.0 |  22.7 |  93.0 |  39.0
AD |  97.9 |  33.7 |  98.6 |  48.6

Table 3: Performance under misleading prompts by clinician.

Scenarios | Single-turn VQA | Multi-turn VQA
---|---|---
| $A_{Q}$ | $H_{cc}$ | $A_{Q}$ | $H_{cc}$
Dr-LLaVA | 89.6 | 22.5 | 92.5 | 5.4
Dr-LLaVA w/o $R_{c}$ | 32.4 | 1.5 | 52.1 | 0.0
Dr-LLaVA w/o $R_{S}$ | 78.4 | 47.5 | 83.0 | 20.2
Dr-LLaVA w/o $R_{m}$ | 85.2 | 30.6 | 87.6 | 8.8
Dr-LLaVA w/o $R_{l}$ | 87.9 | 25.8 | 89.1 | 7.0
Dr-LLaVA w/o $R_{m}$/$R_{l}$ | 84.2 | 33.1 | 87.2 | 10.1

Table 4: Ablation study for the reward model.

Model | Metric | Experiments
---|---|---
SI |  DF |  II
MiniGPT-4-SFT | $A_{Q}$ |  75.8 |  66.2 |  72.2
$A_{C}$ |  44.2 |  40.8 |  41.6
$A_{D}$ |  75.4 |  50.0 |  71.4
LLaMA-Adapter-SFT | $A_{Q}$ |  70.4 |  65.2 |  66.4
$A_{C}$ |  42.5 |  40.2 |  43.5
$A_{D}$ |  75.4 |  74.6 |  70.0
OpenFlamingo-SFT | $A_{Q}$ |  81.4 |  65.2 |  70.0
$A_{C}$ |  46.4 |  40.3 |  41.2
$A_{D}$ |  69.9 |  55.2 |  72.0
LLaVA-Med-SFT | $A_{Q}$ |  91.2 |  86.2 |  85.4
$A_{C}$ |  85.6 |  70.8 |  71.6
$A_{D}$ |  90.3 |  82.2 |  81.3
LLaVA-SFT | $A_{Q}$ |  92.4 |  83.1 |  82.0
$A_{C}$ |  90.1 |  67.5 |  74.6
$A_{D}$ |  91.8 |  76.9 |  76.9
LLaVA-SFT(p) | $A_{Q}$ |  93.3 |  80.0 |  87.9
$A_{C}$ |  91.8 |  72.8 |  82.1
$A_{D}$ |  97.8 |  73.4 |  83.5
Dr-LLaVA | $A_{Q}$ |  93.6 |  88.9 |  92.0
$A_{C}$ |  90.8 |  84.4 |  87.4
$A_{D}$ |  92.0 |  85.9 |  89.0

Table 2: Performance in multi-turn conversations.

###  4.2 Results

Dr-LLaVA outperforms VLM baselines. We start by evaluating Dr-LLaVA with
single-question scenarios, focusing on instances where a clinician seeks
clarification on a specific step in the image analysis process, without the
model having access to prior conversational context. The results, detailed in
Table 1, reveal that our finetuning algorithm significantly boosts Dr-LLaVA’s
performance across all metrics, outperforming state-of-the-art VLMs.
Specifically, Dr-LLaVA achieved a Question-level Accuracy of 89.6%, notably
higher than the top baseline model, LLaVA-SFT(p). Most impressively, Dr-LLaVA
exhibited a 13 percentage point increase in Conversation-level Accuracy over
the best baseline, even in the absence of conversational context. This
improvement underscores the effectiveness of our finetuning algorithm in
ensuring that answers are consistent with clinical reasoning. We also note
that zero-shot generalization to this specialized domain is notably
challenging, as evidenced by the general LLaVA model’s performance falling
below 20% across all metrics.

Dr-LLaVA can handle diverse styles of interactions with clinicians. We also
evaluated Dr-LLaVA in a conversational context. To capture the diverse forms
of possible interactions between clinicians and VLMs, we assessed all VLMs
using 3 conversational scenarios: (1) Standard Interaction (SI): adheres to
the logical dialogue sequence in Fig. 2, starting with image quality
assessment and advancing through morphological analysis to reach a final
diagnosis; (2) Diagnosis First (DF): inverts the sequence in Fig. 2, where the
clinician starts by asking about the patient diagnosis and then interacts with
the model to understand the reasoning behind it; (3) Improvised Interaction
(II): mimics the unpredictability of real-world interactions by randomizing
the question sequence, presenting questions in a non-linear and potentially
repetitive sequence. This is implemented by randomly sampling questions
pertaining to specific conversation with replacement.

Table 2 presents the comparative results. While LLaVA-SFT, LLaVA-SFT(p), and
Dr-LLaVA all perform well SI conversations, Dr-LLaVA significantly outperforms
the baseline models in non-traditional sequences, with performance gains
ranging from 4.1 to 12.5 percentage points. This superior performance
underlines Dr-LLaVA’s advanced adaptive reasoning capabilities, allowing it to
extract critical information from conversational contexts effectively,
regardless of the question sequencing. The ability of our model to handle
these varied conversational dynamics demonstrates its potential in realistic
clinical settings where dialogues may not follow a predefined order.

Dr-LLaVA is better at correcting wrong hypotheses by clinicians. We also
evaluate the model’s performance in scenarios where physicians incorporate
hypotheses into their prompts. Specifically, we examine two types of queries:
Confirmation Queries (CQ), where the clinician seeks model validation of their
(potentially wrong) hypothesis, and Rationalization Queries (RQ), where the
clinician presents (potentially wrong) explanation for their hypothesis and
ask the model about next diagnostic steps. The precise formulations for each
query type are detailed in Appendix C.

Table 3 shows the accuracy of various VLMs in distinguishing between accurate
and misleading information. (“R” corresponds to clinician prompts with right
information and “W” refers to wrong ones.) The performance of all models was
robust when physician prompts included accurate hypotheses, but accuracy
notably declined for all models when the prompts contained misleading
information. In these scenarios, Dr-LLaVA consistently exhibited a higher rate
of disagreement with the misleading content, suggesting that our alignment
algorithm more effectively anchors the symbolic reasoning process (Fig. 2) in
the visual data, thus enabling the model to detect erroneous textual inputs.

Balancing the correctness and consistency trade-off. The hyperparameter
$\lambda$ in (3) balances between the model correctness on responses to
individual question and the overall alignment of these responses with a valid
reasoning process across a conversation. Setting $\lambda$ to a large value
heavy regularizes the conversational output and may encourage the model to
follow valid reasoning processes that are not grounded in the input image,
e.g., the model could always follow a (Low image quality $\to$ Inconclusive)
symbolic rule regardless of the input image. On the other hand, setting
$\lambda=0$ reduces to the standard supervised finetuning setup where the
model optimizes for question-level accuracy but is likely to exhibit context-
conflicting hallucinations within conversations.

Figure 4: Impact of the hyperparameter $\lambda$.

Fig. 4 demonstrates the impact of the choice of $\lambda$ on the model
performance in terms of $A_{Q}$ and the corresponding rate of context-
conflicting hallucinations $H_{cc}$. Here, we define $H_{cc}$ as the fraction
of conversations that map to invalid symbolic rules, i.e.,
$H_{cc}=E[\boldsymbol{1}\\{\hat{s}\notin\mathcal{S}\\}]$. The plot shows that
increasing $\lambda$ initially improves accuracy and consistency (quantified
through $H_{cc}$) reaching an optimal point beyond which further increases in
$\lambda$ lead to diminished accuracy. These findings show that our alignment
with valid clinical reasoning not only improves the model’s coherence and
trustworthiness, but can also improve the model accuracy on individual
questions by regularizing the entire conversational output using prior
knowledge on diagnostic scenarios.

Ablation study for different components of the reward model. We examined the
impact of excluding various components of the reward model in (3). Our
findings, shown in Table 4, indicate that omitting either the correctness or
consistency rewards significantly reduces predictive accuracy. As expected,
removing the correctness reward ($R_{c}$) improves answer consistency. This
occurs because the model is then primarily driven to align with abstract
reasoning, disregarding the actual correctness of the responses in the context
of the visual input. Eliminating the length penalty ($R_{l}$) and no-match
penalty ($R_{m}$) resulted in moderate yet noticeable declines in both
accuracy and consistency. Qualitatively, the absence of these penalties
demonstrate their vital role in preventing reward hacking and maintaining the
integrity of medical dialogue. For instance, the removal of the no-match
penalty caused a marked deterioration in content relevance and accuracy, with
the model occasionally generating blatantly unrelated medical suggestions. An
example of this is the inappropriate reference to renal conditions when
analyzing bone marrow images (Fig. 5(b)). Additionally, without the length
penalty, the model tended towards producing brief, often incomplete responses
as observed in Fig. 5(c).

Figure 5: Exemplary generations from the Dr-LLaVA model and baselines.

##  5 Conclusion and Limitations

Vision-language models (VLMs) hold the potential of becoming valuable tools
for clinicians, researchers, and trainees, offering an interactive natural
language interface for medical image analysis within clinical workflows. Yet,
their utility is often compromised by the generation of “hallucinated” outputs
that deviate from sound medical reasoning, leading to a lack of trust in their
responses. This paper presents a novel alignment algorithm designed to
finetune VLMs, grounding them in the symbolic representations of medical image
analysis processes. This approach ensures the production of clinically valid
and consistent responses throughout multi-turn interactions. Applying this
algorithm, we developed Dr-LLaVA, a VLM specifically tailored for analyzing
bone marrow image patches. Our findings indicate that Dr-LLaVA not only
performs well in straightforward question-answer scenarios but also exhibits
superior adaptability and accuracy in intricate, multi-turn clinical
dialogues, surpassing other advanced VLMs. Additionally, the alignment
algorithm enhances Dr-LLaVA’s ability to identify and rectify misleading
information in clinician prompts. These outcomes highlight the critical role
of precise model alignment with medical knowledge, in order to make VLMs more
reliable and effective in supporting decision-making processes in specialist
domains.

Limitations This work is not without limitations. Firstly, it is confined to
scenarios where the clinician inquires information from the model, excluding
cases where the model prompts the clinician for additional information, such
as lab tests, which would enhance its applicability in comprehensive medical
decision-making. Secondly, our research focuses on a single disease area;
expanding the model to include diagnoses across multiple areas could provide
insights into its robustness and versatility in more complex contexts. Lastly,
future work should further validate the model’s real-world utility by
assessing its performance when prompted by clinicians in practical
environments.

## References

  * [1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models, 2023.  URL https://arxiv. org/abs/2307.09288, 2023.
  * [4] OpenAI.  Chatgpt: Optimizing language models for dialogue.  https://openai.com/blog/chatgpt, 2022.  Accessed: [Your Access Date].
  * [5] OpenAI.  Gpt-4 technical report.  arXiv, 2023.  Accessed: [Your Access Date].
  * [6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [7] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2024.
  * [8] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun.  Llavar: Enhanced visual instruction tuning for text-rich image understanding.  arXiv preprint arXiv:2306.17107, 2023.
  * [9] Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu.  Improving visual grounding with visual-linguistic verification and iterative reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9499–9508, 2022.
  * [10] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  arXiv preprint arXiv:2306.00890, 2023.
  * [11] Masoud Monajatipoor, Mozhdeh Rouhsedaghat, Liunian Harold Li, C-C Jay Kuo, Aichi Chien, and Kai-Wei Chang.  Berthop: An effective vision-and-language model for chest x-ray disease diagnosis.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 725–734. Springer, 2022.
  * [12] Yinda Chen, Che Liu, Wei Huang, Sibo Cheng, Rossella Arcucci, and Zhiwei Xiong.  Generative text-guided 3d vision-language pretraining for unified medical image segmentation.  arXiv preprint arXiv:2306.04811, 2023.
  * [13] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Andrew Zhang, Long Phi Le, et al.  Towards a visual-language foundation model for computational pathology.  arXiv preprint arXiv:2307.12914, 2023.
  * [14] Usman Naseem, Matloob Khushi, and Jinman Kim.  Vision-language transformer for interpretable pathology visual question answering.  IEEE Journal of Biomedical and Health Informatics, 27(4):1681–1690, 2022.
  * [15] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Kenji Ikamura, Georg Gerber, Ivy Liang, Long Phi Le, Tong Ding, Anil V Parwani, et al.  A foundational multimodal vision language ai assistant for human pathology.  arXiv preprint arXiv:2312.07814, 2023.
  * [16] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  arXiv preprint arXiv:2305.10355, 2023.
  * [17] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Aligning large multi-modal model with robust instruction tuning.  arXiv preprint arXiv:2306.14565, 2023.
  * [18] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.  Aligning large multimodal models with factually augmented rlhf.  arXiv preprint arXiv:2309.14525, 2023.
  * [19] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al.  Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.  arXiv preprint arXiv:2312.00849, 2023.
  * [20] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  * [21] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [22] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv preprint arXiv:2310.03744, 2023.
  * [23] Shenghuan Sun, Greg Goldgof, Atul Butte, and Ahmed M Alaa.  Aligning synthetic medical images with clinical knowledge using human feedback.  Advances in Neural Information Processing Systems, 36, 2024.
  * [24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [25] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  Journal of Machine Learning Research, 24(240):1–113, 2023.
  * [26] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  arXiv preprint arXiv:2305.10403, 2023.
  * [27] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * [28] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al.  Crosslingual generalization through multitask finetuning.  arXiv preprint arXiv:2211.01786, 2022.
  * [29] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: An instruction-following llama model, 2023.
  * [30] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
  * [31] Introducing Claude, 3 2023.
  * [32] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv preprint arXiv:1908.08530, 2019.
  * [33] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International conference on machine learning, pages 8821–8831. Pmlr, 2021.
  * [34] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  * [35] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  arXiv preprint arXiv:2308.01390, 2023.
  * [36] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.
  * [37] Michal Byra, Muhammad Febrian Rachmadi, and Henrik Skibbe.  Few-shot medical image classification with simple shape and texture text descriptors using vision-language models.  arXiv preprint arXiv:2308.04005, 2023.
  * [38] Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua.  Qilin-med-vl: Towards chinese large vision-language model for general healthcare.  arXiv preprint arXiv:2310.17956, 2023.
  * [39] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou.  A visual–language foundation model for pathology image analysis using medical twitter.  Nature medicine, 29(9):2307–2316, 2023.
  * [40] Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung.  Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2021.
  * [41] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Towards generalist foundation model for radiology.  arXiv preprint arXiv:2308.02463, 2023.
  * [42] Chang Shu, Fu Liu, and Collier Shareghi.  Visual med-alpaca: A parameter-efficient biomedical llm with visual capabilities, 2023.
  * [43] Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar.  Med-flamingo: a multimodal medical few-shot learner.  In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023.
  * [44] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.  Survey of hallucination in natural language generation.  ACM Computing Surveys, 55(12):1–38, 2023.
  * [45] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  arXiv preprint arXiv:2309.01219, 2023.
  * [46] Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo.  Hallucinations in neural machine translation.  2018\.
  * [47] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald.  On faithfulness and factuality in abstractive summarization.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online, July 2020. Association for Computational Linguistics.
  * [48] Xiao Pu, Mingqi Gao, and Xiaojun Wan.  Summarization is (almost) dead.  arXiv preprint arXiv:2309.09558, 2023.
  * [49] Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad.  Detecting hallucinated content in conditional neural sequence generation.  arXiv preprint arXiv:2011.02593, 2020.
  * [50] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.  Object hallucination in image captioning.  arXiv preprint arXiv:1809.02156, 2018.
  * [51] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih.  Replug: Retrieval-augmented black-box language models.  arXiv preprint arXiv:2301.12652, 2023.
  * [52] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  arXiv preprint arXiv:2109.07958, 2021.
  * [53] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al.  Towards generalist biomedical ai.  NEJM AI, 1(3):AIoa2300138, 2024.
  * [54] Rami Hatem, Brianna Simmons, and Joseph E Thornton.  A call to address ai “hallucinations” and how healthcare professionals can mitigate their risks.  Cureus, 15(9), 2023.
  * [55] Donald E Stanley and Daniel G Campos.  The logic of medical diagnosis.  Perspectives in Biology and Medicine, 56(2):300–315, 2013.
  * [56] Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Zhuo Ran Cai, Roxana Daneshjou, and Pranav Rajpurkar.  Guidelines for rigorous evaluation of clinical llms for conversational reasoning.  medRxiv, pages 2023–09, 2023.
  * [57] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al.  Llama-adapter v2: Parameter-efficient visual instruction model.  arXiv preprint arXiv:2304.15010, 2023.
  * [58] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  arXiv preprint arXiv:2303.16199, 2023.
  * [59] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee.  Making large multimodal models understand arbitrary visual prompts.  arXiv preprint arXiv:2312.00784, 2023.
  * [60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [61] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback.  arXiv preprint arXiv:2305.14387, 2023.

##  Appendix A Appendix

###  A.1 Additional Medical Context

In this paper, we focus on the analysis of bone marrow pathology slides for
the diagnosis of blood cancer disorders. Specifically, our dataset contains
512x512 pixel images of 16,340 pathology patches corresponding to healthy,
inconclusive, acute myeloid leukemia, and multiple myeloma cases. The process
for the analysis of bone marrow pathology slides, involves multiple steps. A
pathologists has to first identify image regions that are deemed adequate for
evaluation, excluding cases with either too low image quality or where the
presence of other cells (e.g. red blood cells) prevents accurate medical
diagnosis. Subsequently, the remaining adequate regions are examined to
determine if they exhibit characteristics indicative of cancerous tissue.
Specifically, in bone marrow aspirates, the assessment focuses on whether
there is abnormal proliferation of cells in the regions of interest. Depending
on the type of cells proliferating, the patient may be diagnosed with a
corresponding hematological disorder. For instance, in our dataset, the
uncontrolled proliferation of blast cells is indicative of acute myeloid
leukemia, while similar proliferation of plasma cells suggests multiple
myeloma.

###  A.2 Dataset generation

The below section provides further details on the steps we took to derive a
multimodal multi-turn conversation dataset in this specialist problem domain.

Image data: We obtained whole pathology slide images sourced from the clinical
archives of an academic medical center. These were then segmented into 512x512
pixel patches222During training, we resize the image to a resolution of
256x256 pixels before feeding it into the image encoder. and labelled as
either "adequate for analysis", "particle-enriched contamination" or "blood
contamination" after pathologist review. Subsequently we leveraged specialist
software in order to obtain cell-counts, based on which a pathologist labelled
cases with a high increase in blast or plasma cells as acute myeloid leukemia
or multiple myeloma, respectively. Table A.1 details the distribution of the
final diagnosis corresponding to the image data.

Question Answer generation: Next, we utilize the symbolic representation of
the bone marrow pathology slide analysis process to create clinically
meaningful multi-turn conversations. This is accomplished by filling in
question and answer templates based on the respective label for each analysis
step. To prevent our model from overfitting to specific expressions in these
templates, we increased the diversity of questions and answers by obtaining
multiple question templates from our clinical collaborators and using GPT-4 to
paraphrase these templates. The respective prompts are provided below:

Prompt for question paraphrasing: "Perform $X$ times augmentation of the
following sentence, it is for medical questions so make sure you preserve the
meaning concisely."

$\textbf{Prompt for answer paraphrasing}:$ "Perform $X$ times augmentation of
the following sentence, it is for medical diagnosis so make sure you preserve
the meaning concisely: ’sentence’. Also note that the question is ’question’,
also don’t repeat anything related to in response to the question, just make
sure the single sentence is grammatically correct and makes sense."

Diagnosis | Number
---|---
Blood contamination | 10083
Particle enriched contamination | 3510
Acute myeloid leukemia | 1531
Multiple myeloma | 932
Healthy | 284
Table A.1: Distribution of Final Diagnoses in the Pathology Slide Image
Dataset

##  Appendix B Evaluation

###  B.1 Evaluation metrics

To effectively assess the performance of our proposed model, we measure the
accuracy of our model at the question, conversation and diagnosis level.

  1. 1.

Question-level Accuracy ($A_{Q}$): This metric evaluates the model’s
performance at the single question level. It is obtained by dividing the
number of questions answered correctly by the total number of questions:

$A_{Q}=\frac{\text{Number of correct answers}}{\text{Total number of
questions}}$ (4)

  2. 2.

Conversation-level Accuracy ($A_{C}$): This metric assesses the model’s
accuracy at the conversation level. Here we only consider a VLM’s response as
correct if it is able to correctly answer all questions pertaining to a multi-
turn conversation about a specific case.

$A_{C}=\frac{\text{Number of conversations with all questions answered
correctly}}{\text{Total number of cases}}$ (5)

This metric asses the model’s capability to consistently provide accurate
answers across all questions within a multi-turn conversation, enabling the
model to be a trustworthy companion throughout the full image analysis
process.

  3. 3.

Diagnostic Accuracy ($A_{D}$): This metric focuses solely on the VLMs’
responses to questions about the final diagnosis, as this is often the primary
concern for medical decision-makers:

$A_{D}=\frac{\text{Number of correctly answered diagnosis
questions}}{\text{Total number of cases}}$ (6)

In conclusion, these three distinct levels of accuracy—$A_{Q}$, $A_{C}$, and
$A_{D}$—provide a comprehensive evaluation of the proposed model’s
effectiveness in handling different aspects of medical inquiries. By breaking
down the analysis to question, conversation, and diagnosis levels, we can
better understand the model’s strengths and pinpoint areas for improvement in
handling complex medical scenarios.

###  B.2 Rule-based reward model implementation

In this work we leverage a simple rule-based reward model that evaluates the
correctness of LLM responses based on the presence of relevant keywords in
their answer. The respective keywords are depicted in Table B.2. For a certain
keyword to be valid we require it to appear without negation. An answer is
classified as ’no match’ in case it does not contain any of the considered
keywords for the respective analysis step.

Analysis Steps |  Classification |  Keywords
---|---|---
Image Quality Assessment |  High quality |  effective, appropriate, suitable, sufficient, optimal
|  Low quality |  not, no, inadequate, unsuitable
|  No Match |  -
Cell Quality Assessment |  Adequate |  optimal, advantageous, suitable, adequate, well, prime
|  Blood |  blood, RBC
|  Clot |  particles
|  No Match |  -
Cell Abnormality Analysis |  Normal |  normal, healthy, no abnormality
|  Abnormal |  cancer, disorder, malignancy
|  Inadequate |  low, subpar, inadequate
|  No Match |  -
Detailed Cell Proliferation Reasoning |  Blast Cell Proliferation |  myeloblast
|  Plasma Cell Proliferation |  plasma cells
|  Normal |  no abnormal, no proliferation, normal
|  Inadequate |  low, subpar, inadequate
|  No Match |  -
Final Diagnosis |  Healthy |  no malignancy phenotype, healthy
|  Acute Myeloid Leukemia |  acute myeloid leukemia, AML
|  Multiple Myeloma |  multiple myeloma, MM
|  Inconclusive |  low quality, inadequate
|  No Match |  -
Table B.2: Keywords considered in rule-based reward model

###  B.3 Assessing hallucination rates

In this work we focus on context-conflicting hallucinations, which we define
as an answer that deviates from the expected pathways outlined in the symbolic
representation of the pathology slide analysis process, as illustrated in 2.
We use a rule-based reward model to classify the VLM responses according to
the possible choices within the symbolic representation of medical reasoning.
This allows us to quantify the proportion of answers that do not follow any
logical trajectory within this framework.

##  Appendix C Example Prompts for Evaluating Model Performance with Clinician
Hypotheses

This section presents example prompts crafted to evaluate our model’s ability
to respond to scenarios where physicians incorporate hypotheses into their
prompts. The prompts are divided into two categories: Confirmation Queries
(CQ) and Rationalization Queries (RQ).

### Confirmation Query (CQ) Prompts

Confirmation Queries aim to assess the model’s ability to validate clinician
opinions. These queries challenge the model to either concur with or contest a
clinician’s judgment, which may be accurate (CQ-R) or erroneous (CQ-W).

Example Prompt 1 (CQ-R): "After reviewing the image, the clinician believes
that [correct statement]. Do you agree with this assessment?"

Example Prompt 2 (CQ-W): "After examining the image, the clinician suggests
that [misleading statement]. Do you concur with this opinion?"

### Rationalization Query (RQ) Prompts

Rationalization Queries present the model with a previous conclusion, which
may be correct (RQ-R) or incorrect (RQ-W), and ask about the next diagnostic
steps. These queries assess the model’s ability to correct incorrect
hypotheses even when not explicitly prompted to do so.

Example Prompt 3 (RQ-R): "A previous clinician reviewed the image and
concluded that [accurate rationale]. Considering this, what would be your next
step in the diagnostic process? [Question]"

Example Prompt 4 (RQ-W): "A previous clinician interpreted the image and
believed [erroneous rationale]. With this in mind, how would you proceed with
the diagnosis? [Question]"