Advance Access Publication Date: Day Month Year Manuscript Category

∗To whom correspondence should be addressed.

Received on XXXXX; revised on XXXXX; accepted on XXXXX

Associate Editor: XXXXXXX

# Improving Medical Reasoning through Retrieval and Self-Reflection with
Retrieval-Augmented Large Language Models

Data and text mining

Minbyul Jeong${}^{\text{\sfb 1}}$ Jiwoong Sohn${}^{\text{\sfb 1}}$ Mujeen
Sung${}^{\text{\sfb 1}}$ and Jaewoo Kang ${}^{\text{\sfb 1,2,}*}$
${}^{\text{\sf 1}}$Department of Computer Science, Korea University, Seoul
02841, Republic of Korea
${}^{\text{\sf 2}}$AIGEN Sciences, Seoul 04778, Republic of Korea

(2015; 2015)

###### Abstract

Recent proprietary large language models (LLMs), such as GPT-4, have achieved
a milestone in tackling diverse challenges in the biomedical domain, ranging
from multiple-choice questions to long-form generations. To address challenges
that still cannot be handled with the encoded knowledge of LLMs, various
retrieval-augmented generation (RAG) methods have been developed by searching
documents from the knowledge corpus and appending them unconditionally or
selectively to the input of LLMs for generation. However, when applying
existing methods to different domain-specific problems, poor generalization
becomes apparent, leading to fetching incorrect documents or making inaccurate
judgments. In this paper, we introduce Self-BioRAG, a framework reliable for
biomedical text that specializes in generating explanations, retrieving
domain-specific documents, and self-reflecting generated responses. We utilize
84k filtered biomedical instruction sets to train Self-BioRAG that can assess
its generated explanations with customized reflective tokens. Our work proves
that domain-specific components, such as a retriever, domain-related document
corpus, and instruction sets are necessary for adhering to domain-related
instructions. Using three major medical question-answering benchmark datasets,
experimental results of Self-BioRAG demonstrate significant performance gains
by achieving a 7.2% absolute improvement on average over the state-of-the-art
open-foundation model with a parameter size of 7B or less. Overall, we analyze
that Self-BioRAG finds the clues in the question, retrieves relevant documents
if needed, and understands how to answer with information from retrieved
documents and encoded knowledge as a medical expert does. We release our data
and code for training our framework components and model weights (7B and 13B)
to enhance capabilities in biomedical and clinical domains.
Availability: Self-BioRAG is available at https://github.com/dmis-lab/self-
biorag
Contact<EMAIL_ADDRESS>
Supplementary information: Supplementary data are available at Bioinformatics
online.

##  1 Introduction

The recent proprietary large language models (LLMs) such as ChatGPT (OpenAI,
2023a), GPT-4 (OpenAI, 2023b), and BARD (Google, 2023) have succeeded in
reaching near or comparable levels to human experts in solving many
challenging problems, ranging from multi-choice question answering to long-
form text generations. While these models exhibit high efficiency and
demonstrate their versatility in various domains, they fall short in
comprehensively covering user-dependent information such as patient reports
with encoded knowledge. These limitations can result in a groundless statement
and inadvertent generation of false information, commonly known as the
hallucination issue (Wei et al., 2022; Singhal et al., 2022; Cao et al., 2022;
Ji et al., 2023). To address this challenge, retrieval-augmented generation
(RAG) enhances explainability for readers by supplying supporting facts that
underpin the responses generated by LLMs (Lewis et al., 2020; Guu et al.,
2020). As illustrated in Figure 1, various RAG frameworks search documents
from the knowledge corpus such as Wikipedia and appending them unconditionally
or selectively to the input of LLMs for generation. In alignment with this
approach, the authors of Asai et al. (2023) introduce Self-RAG, which
possesses the following capabilities using reflective tokens: deciding when to
use on-demand retrieval, assessing whether retrieved evidence provides useful
information to solve the question, criticizing whether the evidence supports
the answer, and judging whether the answer is a useful response to the
question. However, using Self-RAG is unsuitable for domain-specific questions
like biomedical or clinical domains which shows poor generalization, leading
to fetching incorrect documents or making inaccurate judgments.

Figure 1:  Comparison between three frameworks: generation using language
model (LM), retrieval-augmented generation (RAG) using LM, and our Self-
BioRAG. (A) depicts the process of sequence-to-sequence generation of LM. (B)
The RAG framework first finds relevant documents from large-scale corpus such
as PubMed Central and then provides the answer based on this factual content
to address the shortage of scarce knowledge. (C) Initially, our domain-
specific instruction-tuned model predicts whether retrieval is necessary. If a
query doesn’t require any retrieval of knowledge (factual content), it
directly predicts the answer. However, if the query necessitates retrieval
knowledge, Self-BioRAG utilizes the domain-specific retriever (MedCPT, in our
case) to retrieve relevant documents. After retrieving the top-$k$ evidence,
the model selects the most pertinent evidence for the query. Ultimately, our
language model is employed to select the best evidence and generate the answer
based on the selected evidence and encoded knowledge.

In this paper, we introduce Self-BioRAG, trained with a focus on biomedical
and clinical text instructions, enabling it to address corresponding
instructions adeptly. It preserves generation quality and reasoning ability
while incorporating on-demand retrieval and self-reflection capabilities. Note
that we use the term reasoning to indicate that Self-BioRAG can provide
explanations on answers. To build a Self-BioRAG framework, four essential
components are required: (1) biomedical instruction sets, (2) biomedical
retriever, (3) self-reflection language model, and (4) domain-specific
instruction-tuned language model. We initially construct instruction sets
focused on biomedical and clinical text. In addition to the distributed MoL-
instructions (Fang et al., 2023) and MedInstruct (Zhang et al., 2023), we
synthetically generate an additional 18k biomedical and clinical instructions
following the Self-Instruct (Wang et al., 2022). By combining three datasets,
we could construct 120k instruction sets addressing various biomedical
instructions, including information extraction, question answering,
summarization, text classification, relation extraction, and multi-choice
questions (Section 3.1).

Furthermore, we use the off-the-shelf MedCPT (Jin et al., 2023) retriever and
construct biomedical corpora as follows: PubMed Abstract, PMC Full Text,
Clinical Guideline, and Medical Textbook, all tailored to biomedical and
clinical text (Section 3.2). The training process for the self-reflection
language model and the domain-specific instruction-tuned language model is
similar to Self-RAG, except that, instead of directly training instructions
into the LLaMA2 (Touvron et al., 2023) model, we achieve better performance by
training the model weights provided by Self-RAG (Section 3.3, 3.4). The goal
of our work is to construct a language model encoded with domain-specific
knowledge, enabling it to autonomously assess explanations and answers it
generates.

Self-BioRAG demonstrates its effectiveness using three open-domain question-
answering benchmark datasets: MedQA (Jin et al., 2021), MedMCQA (Pal et al.,
2022), and MMLU (Hendrycks et al., 2020). Experimental results on the three
benchmark datasets demonstrate that Self-BioRAG significantly outperforms
open-foundation LLMs and RAG approaches with a parameter size of 7B or less.
Self-BioRAG achieves a 7.2% absolute improvement compared to the state-of-the-
art model. We demonstrate that domain-specific components contribute to the
performance gains, with training on domain-specific instructions showing the
highest improvement. Our biomedical corpora supplement scarce knowledge, and
particularly, Self-BioRAG uses appropriate documents if needed corresponding
to the benchmark datasets. We further analyze that using reflective tokens to
adaptively retrieve factual content is effective in solving open-domain
question-answering datasets. Overall, Self-BioRAG finds the clues in the
question, retrieves relevant evidence, and understands how to answer with
information using encoded knowledge.

Our contributions are as follows: (1) We introduce a Self-BioRAG framework
which is extensively trained on biomedical and clinical instructions. (2) We
prove that domain-specific components such as retriever, documents, and
instruction sets are necessary to address its domain-related instructions. (3)
Self-BioRAG demonstrates its effectiveness in three open-domain biomedical
question-answering benchmark datasets by achieving an average absolute
improvement of 7.2% compared to the state-of-the-art open-foundation model
with a parameter size of 7B or less. (4) We release our biomedical instruction
sets, code for training our components used in Self-BioRAG, and model weights
(7B & 13B) to be more capable in biomedical and clinical domains.

Figure 2:  Overview of our Self-BioRAG process: data construction, training,
and inference of Self-Reflection Language Model (critic LM $C$) and Domain-
specific Instruction-tuned Language Model (generator LM $M$). We construct
120k biomedical instruction sets using two off-the-shelf instruction sets
(Mol-Instructions (Fang et al., 2023) and MedInstruct (Zhang et al., 2023))
and one self-generated biomedical instruction set. We first sample 5k
instructions to generate reflective tokens via GPT-4 API calls and then train
the critic LM $C$ with these instructions. Using trained critic LM $C$, we
filter out mispredicted reflective tokens, such as [Continue Generation]. We
preserve 84k instruction sets annotated with pre-defined reflective tokens to
train the generator LM $M$. Note that critic LM $C$ is only used for
annotating reflective tokens used to filter instruction sets to train
generator LM $M$. After training, the model $M$ can predict whether or not to
use the retrieval method and combine the results of evidence and encoded
knowledge to answer the question. We use the MedQA (Jin et al., 2021) test
sample to gain a proper understanding of how our Self-BioRAG works.

##  2 Background

###  2.1 Proprietary & Open Language Models

Instructions serve as guidelines for how language models should perform a
particular task. In the commercial field, proprietary language models such as
InstructGPT (Ouyang et al., 2022) and ChatGPT (OpenAI, 2023a) have gained
significant advantages in tuning through instructions. However, researchers
not involved in commercial fields may face challenges in using these models
due to a lack of resources. Hence, research-friendly open foundation models
like the LLaMA family (Touvron et al., 2023), Self-instruct (Wang et al.,
2022), and Alpaca (Taori et al., 2023) are released. In this regard, domain-
specific language models tailored for areas such as biomedical and clinical
domains, like Galactica (Taylor et al., 2022) and Meditron (Chen et al.,
2023), have also been released. Our research also aims to provide labor-
inexpensive methods that are easy to use in various vertical domains,
including biomedical and clinical domains. Specifically, Self-BioRAG strives
to develop a model capable of solving challenging tasks, ranging from multi-
choice questions to long-form generations.

###  2.2 Learning with Reward Strategy

The proprietary language models trained with reinforcement learning from human
feedback (RLHF), such as ChatGPT (OpenAI, 2023a) and GPT-4 (OpenAI, 2023b),
excel at executing straightforward instructions (e.g., translation, code
generation, and question answering) in alignment with human intent (Schulman
et al., 2017; Christiano et al., 2017; OpenAI, 2023a, b; Google, 2023). In
Self-RAG (Asai et al., 2023), a critic language model is employed to offer a
cost-effective reward strategy compared to RLHF, utilizing reflective tokens.
The critic model determines whether a given task necessitates retrieval,
evaluates the appropriateness of the retrieved context, assesses if the
generated rationale aligns with the retrieved context, and ultimately judges
the overall utility of the output. Our Self-BioRAG follows the approach of
Self-RAG to create a domain-specific critic language model that not only
maintains the aforementioned capabilities but is also well-versed in
biomedical text.

###  2.3 Retrieval-Augmented Generation

The retrieval-augmented generation (RAG) significantly enhances performance in
knowledge-intensive tasks and open-domain question-answering by providing
context as input to the language model (Lewis et al., 2020; Mao et al., 2021;
Kang et al., 2023). The retriever also plays a crucial role in language models
by providing evidence for pre-training and few-shot fine-tuning (Guu et al.,
2020; Izacard et al., 2022a). With the recent advancements in instruction
language models, the combination of retriever and language models involves
either using the retriever in advance to fetch evidence or iteratively
retrieving it when needed (Jiang et al., 2023; Shao et al., 2023). Our base
framework, Self-RAG (Asai et al., 2023), deviates from these approaches by
being designed to perform retrieval on-demand, resulting in better cost
efficiency compared to scenarios where retrieval is always active. However, in
domain-specific fields like biomedical or clinical domains, the general method
of retrieving context may not be applicable. Therefore, Self-BioRAG utilizes
retrieval methods and documents tailored to specific domains, retrieving
meaningful context that aligns with the intended field.

Dataset Name | # of instances | Tasks Types
---|---|---
Mol-Instructions (Fang et al., 2023) | 38,156 | Information Extraction, Question Answering, Multi-Choice Question
MedInstruct (Zhang et al., 2023) | 36,429 | Question Answering, Summarization, Text Classification, Multi-Choice Question
Biomedical Instructions (Ours) | 10,143 | Text Generation, Question Answering, Relation Extraction, Text Classification, Summarization

Table 1: Statistics of biomedical instruction sets used to train the generator
language model $M$.

##  3 Self-BioRAG

In this section, we outline the process of creating our Self-BioRAG framework
using various biomedical components. First, we leverage three datasets
consisting of biomedical instructions, which are used to train language models
to align with human intentions for biomedical text (Section 3.1). To
supplement scarce knowledge via relevant documents in the biomedical or
clinical domains, we employ an off-the-shelf MedCPT (Jin et al., 2023)
retriever, known for its effectiveness in retrieving relevant documents in
biomedical and clinical domains (Section 3.2). Subsequently, we develop a
critic language model $C$ to annotate the instruction sets which will contain
information for facilitating an autonomous assessment of reflective criteria
(Section 3.3). Lastly, we perform training on our generator model $M$ using
the instruction sets created with diverse biomedical components (Section 3.4).
We depict our processes of data generation, training, and inference in Figure
2.

###  3.1 Biomedical Instruction Datasets

#### List of Instruction Datasets for Biomedical and Clinical Domains.

To train the self-reflection language model (LM), also referred to as the
critic LM $C$, we utilize diverse text triplets (instruction, input, output).
Specifically, we collect two off-the-shelf instruction sets (Mol-Instructions
(Fang et al., 2023) and MedInstruct (Zhang et al., 2023)), which include tasks
like open-generation, true or false, and multi-choice questions. In addition
to the distributed instruction sets, we synthetically generate an additional
18k biomedical and clinical instructions following the Self-Instruct (Wang et
al., 2022). In total, we construct 120k biomedical instruction sets addressing
diverse tasks: information extraction, question answering, and summarization.
For instance, illustrated in Figure 2, the example instruction was set to
classify the given radiology report according to which part of the body is
related and answer with the lumbar spine. The statistics of biomedical
instruction sets are provided in Table 1. Detailed statistics of our generated
instruction sets can be found in Appendix A.

###  3.2 Biomedical Retriever

In the fields of biomedical and clinical domains, researchers and doctors
addressing challenging issues typically supplement their knowledge with
additional information. Similarly, for a language model to solve problems, it
needs to retrieve relevant documents as needed. To achieve this, we use the
off-the-shelf MedCPT (Jin et al., 2023)
retriever111https://github.com/ncbi/MedCPT, which is contrastively trained on
an unprecedented scale of 255M query-article pairs from PubMed search logs. To
retrieve relevant documents, we compile data from four sources: PubMed
Abstract222https://pubmed.ncbi.nlm.nih.gov/, PMC Full-
text333https://www.ncbi.nlm.nih.gov/pmc/tools/textmining/, Clinical
Guidelines444A publicly released subset of 35,733 guideline articles from
MEDITRON (Chen et al., 2023), extracted from 8 sources: CCO, CDC, CMA, ICRC,
NICE, SPOR, WHO, and WikiDoc, and English Medical Textbooks555Medical
textbooks widely used by medical students and takers of the United States
Medical Licensing Examination (USMLE), https://github.com/jind11/MedQA. We
encode these data offline to make it computationally effective. The documents
are segmented into chunks of 128 words with 32-word overlaps to form evidence
following previous works (Wang et al., 2019; Karpukhin et al., 2020). We first
retrieve top-$k$ ($k$=10, in our case) evidence from each source data (total
4$k$ evidence) and then use the reranking module to obtain the final top-$k$
evidence relevant to the query. Table 2 presents the overall statistics of
biomedical corpus and how many documents are indexed.

Data | # Documents | # Chunks | Embedding Size
---|---|---|---
PubMed | 36,533,377 | 69,743,442 | 400GB
PMC | 1,060,173 | 46,294,271 | 160GB
CPG | 35,733 | 606,785 | 3.5GB
Textbook | 18 | 133,875 | 0.7GB
Table 2: Statistics of the indexed biomedical corpus. CPG stands for Clinical
Practice Guideline.

###  3.3 Self-Reflection Language Model (Critic Language Model)

#### Data Construction of Critic LM $C$.

We collect a total of 120k biomedical instruction sets and randomly sample 5k
examples to train the critic LM $C$. We use GPT-4 API Calls to generate
reflective tokens, guiding the critic model $C$ in learning how to predict
these tokens. We follow the usage of four types of reflective tokens employed
in Self-RAG (Asai et al., 2023), as described in Table 3. Detailed statistics
and prompts used in generating each type of reflective token are provided in
Appendix G. Exploring other reflective tokens suitable for specific domains is
left for future work.

#### Process of Training Critic LM $C$.

We observe that a critic LM released by Self-RAG (Asai et al.,
2023)666https://github.com/AkariAsai/self-rag is unsuitable for predicting
reflective tokens in biomedical domains777See Appendix B for a comparison of
critic LMs’ performances. Thus, we decide to develop a domain-specific critic
LM $C$ using our biomedical instruction sets. We split the sampled instruction
sets into train and dev to train and assess the performance of the critic LM
$C$. We train the model using four types of reflective tokens annotated with
GPT-4 API calls. Note that training the critic LM $C$ signifies it to predict
pre-defined reflective tokens given instruction, output, and optionally
evidence. We use trained LM $C$ to annotate whole instruction sets and filter
out instances when it mispredicts the reflective tokens that are not pre-
defined such as [Continue Generation]. We provide detailed hyperparameters
used to train the critic LM $C$ in Appendix C.

#### Annotating Biomedical Instruction Sets Using Critic LM $C$.

After training, the model $C$ predicts four types of reflective tokens: (1)
identifying whether a question requires retrieval (RET); (2) determining if
retrieved evidence provides useful information to solve a question (REL); (3)
assessing whether all statements of answers can be supported by evidence
(SUP); (4) evaluating whether all statements of answers are a useful response
to the question (USE). For example, in Figure 2, the model $C$ predicts the
retrieval of factual content related to the role of BRCA1 and BRCA2 gene
mutation ([Retrieval]). Then, the model predicts that the retrieved evidence
provides a fact that BRCA1 and BRCA2 play similar roles in breast cancer and
sporadic cancer ([Relevant]). By comparing a statement of the answer and
retrieved evidence, the model $C$ predicts that the answer could be supported
by evidence ([Fully supported]). Finally, the model $C$ suggests that all
statements of answers are useful responses to the question ([Utility:4]).
After annotating each type of reflective token, we aggregate all results to
construct a complete instance as above. We provide detailed instructions to
annotate the biomedical instruction dataset using the critic LM in Appendix H.

Type | Input | Output | Definitions
---|---|---|---
RET | $x/x,y$ | {yes, no, continue} | Decides when to retrieve using $R$
REL | $x,e$ | {relevant, irrelevant} |  $e$ provides useful information to solve $x$
SUP | $x,e,y$ |  | {fully supported, partially
---
supported, no support}
| All of the verification-worthy statement
---
in $y$ is supported by $e$
USE | $x,y$ | {5, 4, 3, 2, 1} |  $y$ is a useful response to $x$

Table 3: Reflective tokens used in Self-BioRAG. $x$, $y$, and $e$ respectively
indicate input, output, and evidence. Specific reflective tokens highlighted
in bold are desirable during data construction as they contribute to
preserving the existing instruction data possible.

###  3.4 Domain-Specific Instruction-Tuned Language Model (Generator Language
Model)

#### Data Construction Using Critic LM $C$ & Training Generator LM $M$.

We use MedCPT to retrieve top-$k$ evidence following an instruction that
necessitates retrieval of biomedical context. After retrieving relevant
documents, we use the critic LM $C$ to predict each reflective token as
described in Table 3. Consequently, we preserve 84k filtered instances of
biomedical instruction sets annotated with pre-defined reflective tokens,
instruction, and output triplets to train generator LM $M$. We want to point
out that the critic LM $C$ is only used to annotate reflective tokens to
generate biomedical instruction sets to train generator LM $M$. Then, we fine-
tune these biomedical instructions on the generator model released by the
Self-RAG framework. This enhances generalizability in the biomedical and
clinical domains preserving the abilities of text generation and self-
assessment of its generated explanations with reflective tokens.

#### Inference Process of Self-BioRAG.

In Figure 2, we present a MedQA (Jin et al., 2021) example to illustrate our
Self-BioRAG inference offline. For instance, the question is inquiring about
the diagnosis of a female patient exhibiting symptoms of obesity, acne, and
has a history of type 2 diabetes mellitus. The generator model $M$ determines
the need to retrieve a relevant document and selects the best evidence from
the top-$k$ retrieved documents based on a score $S$, calculated as the
weighted sum of reflective tokens, using the same hyperparameters as Self-RAG,

$\displaystyle S(Critique)$
$\displaystyle=\sum_{G\in\mathbb{G}}w^{G}s^{G},~{}~{}\mathbb{G}={REL\cup
SUP\cup USE}$ $\displaystyle s^{G}$
$\displaystyle=\frac{p(\hat{r})}{\sum_{i=1}^{N^{G}}p(r_{i})}$

where $s^{G}$ denotes the generation probability of the most desirable
reflective token $\hat{r}$ (e.g., [Fully supported]) for reflective token type
$G$ (e.g., SUP) and $w^{G}$ represents the hyperparameter providing weight for
$s^{G}$. We can set the weight $w^{G}$ to adjust our behavior at inference
time. For example, to find the most relevant document $e$ related to question
$x$, we can set a weight term $REL$ score higher. Self-BioRAG is tailored to
conditionally generate text without any additional training which could need
balancing the trade-off between multiple preferences (Touvron et al., 2023; Wu
et al., 2023).

The prioritized evidence includes information on the family history of type 2
diabetes mellitus and the patient’s diagnosis of polycystic ovarian syndrome
(PCOS). Due to space limitations, we display partial information in the
figure; please refer to the complete case in Table 7. Consequently, the
generator model $M$ generates the following text: (1) the patient has acne and
obesity, typical symptoms of PCOS; (2) the patient has a family history of
type 2 diabetes mellitus, often associated with PCOS; (3) the patient
underwent a glucose tolerance test, a characteristic feature of PCOS. Overall,
our Self-BioRAG identifies clues in the question, retrieves factual content if
necessary, and responds with encoded knowledge, similar to how doctors
approach such cases.

##  4 Experimental Details

|  | Open-domain Biomedical Benchmark
---|---|---
| Model
---
| Params.
---
| MedQA
---
(Acc.)
| MedMCQA
---
(Acc.)
| MMLU-Med
---
(Acc.)
Average
Proprietary LM |  |  |  |  |
Med-PaLM (Singhal et al., 2022) | 540B | 60.3 | 56.5 | 75.6 | 64.1
GPT-3.5 (OpenAI, 2023a) | - | 53.6 | 51.0 | 67.3 | 57.2
GPT-4-base (OpenAI, 2023b) | - | 86.1 | 73.7 | 89.9 | 83.2
Open LM |  |  |  |  |
Alpaca (Taori et al., 2023) | 7B | 23.6 | 30.4 | 34.4 | 29.5
FLAN-T5 (Chung et al., 2022) | 3B | 36.1 | 20.0 | 44.2 | 33.4
Galactica (Taylor et al., 2022) | 6.7B | 29.5 | 32.7 | 39.4 | 33.9
MEDITRON (Chen et al., 2023) | 7B | 36.5 | 37.1 | 41.1 | 38.3
LLaMA2 (Touvron et al., 2023) | 7B | 35.2 | 36.3 | 46.3 | 39.3
Open LM + Retrieval |  |  |  |  |
RAG | 7B | 36.2 | 38.3 | 47.7 | 40.7
Self-RAG (Asai et al., 2023) | 7B | 31.2 | 36.5 | 45.7 | 37.8
Self-BioRAG (Ours) | 7B | 43.6 | 42.1 | 53.9 | 46.5
Self-BioRAG (Ours) | 13B | 48.6 | 44.0 | 57.2 | 49.9

Table 4: Experimental results on biomedical benchmark datasets. We use 3-shot
examples as guidelines for language models to address benchmark instances.
These examples are chosen from each training dataset using k-nearest-neighbor
(Guo et al., 2003). Since the MMLU dataset lacks training data, we employ the
same examples detailed in the appendix of MedPALM (Singhal et al., 2022). The
score of GPT-3.5 and GPT-4-base models are from the following paper (Nori et
al., 2023). We use biomedical corpus (e.g., PubMed, PMC, CPG, and Textbook) as
evidence during inference on the RAG setting. The best score is highlighted in
bold for the parameter size of 7B or less.

#### Benchmark Evaluations.

We evaluate the downstream task performance of Self-BioRAG fine-tuned on Self-
RAG. Our experiments encompass multi-choice question-answering benchmarks
designed to gauge biomedical and clinical knowledge encoded in language
models. Following the approach of previous work, such as MedPaLM (Singhal et
al., 2022), we use three datasets for few-shot evaluation on open-domain
question-answering: MedQA (Jin et al., 2021), MedMCQA (Pal et al., 2022), and
MMLU (Hendrycks et al., 2020). In MMLU, we extract six clinical topics:
anatomy, clinical knowledge, college biology, college medicine, medical
genetics, and professional medicine.

#### Baselines.

In Table 4, we compare Self-BioRAG with proprietary, open foundation, and open
foundation with retrieval-augmented language models. We report the Med-PaLM
score as presented in Med-PaLM (Singhal et al., 2022) and the GPT-3.5 and
GPT-4-base scores as presented in Nori et al. (2023) to establish the upper
bound of benchmark datasets (Row 1-3). Open foundation models, pre-trained for
sequence-to-sequence generation with instruction tuning, such as Alpaca (Taori
et al., 2023) and Flan-T5 (Chung et al., 2022), are reported (Row 4 & 5), as
well as models fine-tuned on the specific vertical domains (e.g., biomedical
and clinical), like Galactica (Taylor et al., 2022) and Meditron (Chen et al.,
2023) (Row 6 & 7). LLaMA2 (Touvron et al., 2023) demonstrates state-of-the-art
performance in open-foundation 7B models in our experiment (Row 8). Therefore,
we employ LLaMA2 for the result of retrieval-augmented generation (RAG) and
provide the top-10 evidence collected from the biomedical corpus using the
MedCPT retriever (Row 9). Due to the length limit of RAG for input, we can
only leverage the top 1 evidence in input and few-shot examples. Additionally,
we report Self-RAG (Asai et al., 2023) using Contriever (Izacard et al.,
2022b) fine-tuned on MSMARCO (Bajaj et al., 2016) with the Wikipedia corpus
(Row 10). We compare these baselines with our Self-BioRAG framework which is
trained with biomedical components.

#### Training and Inference Settings.

Self-BioRAG is trained with 84k biomedical instruction sets filtered using a
trained critic language model (LM). We adopt the Self-RAG critic LM as our
base model and fine-tune it with 5k sampled instruction sets annotated by
GPT-4 API calls. As training on the Self-RAG generator LM yields better
results, we fine-tune our biomedical instruction sets instead of training
directly on LLaMA2 (Touvron et al., 2023) or Meditron (Chen et al., 2023). For
the retriever, we use the off-the-shelf MedCPT (Jin et al., 2023) retriever,
specialized in retrieving documents based on biomedical queries and retrieving
up to ten evidence for each input.

For inference, we use vllm (Kwon et al., 2023) to speed up our inference time.
Following Self-RAG (Asai et al., 2023), we assign the same weight terms for
reflective tokens (e.g., REL, SUP, USE) in decoding. We adopt adaptive
retrieval by default which dynamically decides when to retrieve the evidence
by predicting a reflective token [Retrieval]. we retrieve the top ten evidence
from the biomedical corpus processed offline. We provide details of the
retrieved percentage of source data used to evaluate biomedical benchmark
datasets in Section 5.2.

##  5 Results and Analysis

###  5.1 Experimental Results

#### What Contributes to the Performance Improvements in Self-BioRAG?

In Table 4, we compare our Self-BioRAG with open foundation language model
(LM) and retrieval augmented generation (RAG). With a parameter size of 7B or
less, our Self-BioRAG outperforms other open foundation LMs (Row 4-8) in all
three biomedical benchmark datasets (MedQA, MedMCQA, and MMLU-Med). We also
compare our model with baselines using retrieval evidence. The RAG pipeline
faces two challenges: it struggles to identify crucial evidence and encounters
limitations in incorporating numerous pieces of evidence due to constraints on
input length. However, our Self-BioRAG outperforms the RAG baseline and can
prioritize important evidence via the values of reflective tokens, which is
useful for analyzing all the retrieved evidence (Row 9 & 11). Although Self-
RAG is fine-tuned on LLaMA2, we observe that Self-RAG cannot generalize to
biomedical benchmark datasets, resulting in a performance drop (Row 8 & 10).
By providing a biomedical critic LM and corpus to train a biomedical generator
LM, our Self-BioRAG achieves state-of-the-art performance on 7B parameters in
MedQA, MedMCQA, and MMLU-Med datasets. We also provide the 13B performance of
our Self-BioRAG model to demonstrate the effectiveness of our framework works
in other model parameters (Row 12). We provide the detailed performance of
specific MMLU datasets in Appendix F. We aim to analyze the step-by-step
process through which our Self-BioRAG achieves its state-of-the-art
performance in the following subsection.

Experiment Detail |  | MedQA
---
(Acc.)
| MedMCQA
---
(Acc.)
| MMLU-Med
---
(Acc.)
Average
Self-RAG (Asai et al., 2023) | 31.2 | 36.5 | 45.7 | 37.8
$+$ Biomedical Corpora | 33.5 | 38.9 | 46.7 | 39.7 (+1.9)
$+$ MedCPT Retriever | 35.6 | 40.1 | 49.3 | 41.7 (+3.9)
$+$ Biomedical Instruction Sets | 43.6 | 42.1 | 53.9 | 46.5 (+8.7)

Table 5:  Effect of each domain-adaptaion component.

###  5.2 Analysis

#### Which Domain-Adaptation Components Show the Improvements Compared to
Self-RAG?

In Table 5, each experiment involves sequentially adding biomedical components
in Self-RAG. The goal is to identify the factors that significantly
contributed to the performance improvement, ultimately leading to the final
performance of Self-BioRAG. We observe that using four biomedical corpora
(PubMed, PMC, CPG, and Medical Textbook) to retrieve appropriate evidence
shows performance improvement compared to Wikipedia evidence (Row 2). We also
use domain-specific MedCPT retriever instead of the Contriever (Izacard et
al., 2022b) fine-tuned on MSMARCO (Bajaj et al., 2016) (Row 3). Ultimately,
the most effective approach was the collection and processing of biomedical
instruction sets to create both a critic language model and a generation
language model (Row 4). We recommend readers collect their domain-specific
instructions to address corresponding instructions.

Figure 3: Ratio of retrieved evidences from each of the four biomedical
corpora (PubMed, PMC, CPG, Medical Textbook). The RAG statistics refer to the
top-1 evidence usage ratio, while Self-BioRAG selects the most useful evidence
from the top-10 retrieved evidence.

#### In Biomedical Corpora, What Evidence is Used to Solve Open-Domain
Question-Answering Benchmarks?

In Figure 3, we compare the ratio of retrieved evidence using the MedCPT (Jin
et al., 2023) retriever on four biomedical corpora (PubMed, PMC, CPG, and
Medical Textbook). Even though the index sizes of Medical Textbook and CPG are
much smaller than PubMed or PMC, retrieved evidences show even distribution.
Specifically, our Self-BioRAG only retrieves small portions to solve three
datasets (MedQA (12%), MedMCQA (8%), and MMLU-Med (11%)) meaning that these
open-domain benchmarks don’t require that much evidence than expected. We
depict these portions up to 100% in Figure 3. We observe a trend in which
Self-BioRAG retrieves a higher proportion of information from the Medical
Textbook, similar to the approach used in solving USMLE-style questions. This
is also aligned with previous facts that retrieving documents from Medical
Textbook can achieve higher performance in clinical questions (Wang et al.,
2023; Li et al., 2023).

#### Does Evidence Truly Help to Supplement Limited Knowledge in Self-BioRAG?

In Table 6, we conduct three different experimental settings using open-domain
question-answering benchmark datasets: (1) No Retrieve: Answer without
utilizing any provided evidence. (2) Only Retrieve: Retrieve top-10 evidence
for each question and predict an answer. (3) Adaptive Retrieve: Use a
criterion to decide whether to retrieve or not based on the question. The
criterion is set as follows,

$\frac{p(retrieve=Yes)}{p(retrieve=Yes)+p(retrieve=No)}>\delta.$ (1)

where we set $\delta$ hyperparameter as 0.2 for the Adaptive Retrieve
experiment setting. Our findings indicate that retrieving relevant documents
indeed aids in solving benchmark datasets. Additionally, we observed that
adaptively retrieving shows comparable performance on average with the Only
Retrieve setting. This is attributed to the small portion of retrieved
evidence used to answer the questions. While the Only Retrieve setting
exhibits a substantial improvement in MedMCQA, it shows a performance drop in
MMLU-Medical datasets compared to the No Retrieve setting, indicating its
instability. As a result, we recommend readers to use the adaptive retrieval
setting.

Methods |  | MedQA
---
(Acc.)
| MedMCQA
---
(Acc.)
| MMLU-Med
---
(Acc.)
Average
Only [No Retrieval] | 39.7 | 41.9 | 52.8 | 44.8
Only [Retrieval] | 40.1 | 47.2 | 51.3 | 46.2
Adaptive Retrieval (Ours) | 43.6 | 42.1 | 53.9 | 46.5

Table 6:  Effect of adaptive retrieval in Self-BioRAG. ‘Only [No Retrieval]’
refers to not retrieving any evidence, and ‘Only [Retrieval]’ refers to
forcing the retrieval of top 10 evidences.  Figure 4:  Performance of LLaMA2,
RAG, and Self-BioRAG on examples split into [No Retrieval] and [Retrieval]
based on Self-BioRAG using the MedQA test dataset. Self-BioRAG consistently
outperforms other baselines whether or not retrieved evidence is used.

#### Distinguishing When to Retrieve Documents in Self-BioRAG.

In Figure 4, we evaluate the performance of LLaMA2, RAG (LLaMA2 with MedCPT
and biomedical corpora), and Self-BioRAG on examples predicted as [No
Retrieval] and [Retrieval] by Self-BioRAG. To show an overall trend, we use
the MedQA dataset here and the rest of the two datasets are in Appendix E.
Notably, Self-BioRAG retrieves small portions to solve three biomedical
benchmarks. Still, the results demonstrate that Self-BioRAG consistently
outperforms other baselines, whether or not retrieved evidence is used. In
situations where retrieval is not necessary (left column), Self-BioRAG $>$ RAG
$\approx$ LLaMA2. The overall trend in the retrieved situation (right column)
indicates Self-BioRAG $>$ RAG $\geq$ LLaMA2. Intuitively, we identify that
Self-BioRAG distinguishes well on situations to use retrieved evidence or not
depending on questions.

MedQA Dataset
---
| Query: A 27-year-old woman presents to the office with concerns about her
long struggle with her physical appearance since adolescence. She
---
says she has always been $``$large$"$ and was constantly targeted by her
classmates and coworkers for being so. Her main concern at the moment is
her acne and unwanted facial hair on her upper lip, for which she often visits
a local spa. She has tried numerous diet plans, exercise regimens, and
cosmetic products with little to no effect. Recently, she underwent a glucose
tolerance test that showed a plasma glucose level of 160 mg/dL (8.9
mmol/L) after 2 hours of a 75 g dose of oral glucose. She has a family history
of type 2 diabetes mellitus and a menstrual cycle that occurs every
45 days. Her pulse is 72/min and the blood pressure is 138/80 mm Hg. On
physical examination, her height is 160 cm (5 ft 3 in) and her weight is 85
kg (187 lb), and she has severe inflammatory acne over the cheeks and forehead
and dark coarse hairs on the back. What is the most likely
diagnosis of this patient?
Option A: Hypothyroidism
Option B: Idiopathic hirsutism
Option C: Polycystic ovarian syndrome (PCOS)
Option D: Ovarian hyperthecosis
Evidence: Early Clinical Expressions of Insulin Resistance: The Real Enemy to
Look For. Today, a very common clinical scenario is a 17-year
-old female with a family history of type 2 diabetes mellitus (T2DM) and hypertension in her mother and two first-degree relatives. Three years ago
she was diagnosed with polycystic ovarian syndrome (PCOS). She is overweight
with her body weight increasing 10 kg in the last 2 years. Addition-
ally, she has acanthosis nigricans (AN) in the axillae and knuckles. Blood
tests were unremarkable with a fasting plasma glucose value of 82 mg/dl
and an HbA1c of 5.0%.
Prediction: The most likely diagnosis of this patient is Option C: Polycystic
ovarian syndrome (PCOS). The patient has the typical symptoms of
PCOS, such as acne, hirsutism, and obesity. Additionally, she has a family
history of type 2 diabetes mellitus, which is often associated with
PCOS. The glucose tolerance test result of 160 mg/dL (8.9 mmol/L) after 2
hours confirms impaired glucose tolerance, which is another charac-
teristic feature of PCOS.

Table 7: Case report of Self-BioRAG prediction using evidence in MedQA
dataset. Retrieved evidence is written in italics. Blue-colored text comprises
segments connected to key information from retrieved evidence, while red-
colored text consists of segments tied to the model’s parametric knowledge.
Our model leverages both types of segments to address the given query.

#### Result of Closed-domain Question-answering dataset.

We evaluate Self-BioRAG on the PubMedQA (Jin et al., 2019) dataset, providing
a golden context to solve questions. Our Self-BioRAG achieves 54.6% accuracy,
which is slightly lower than open foundation LMs such as LLaMA2 (56.0%) or
MEDITRON (58.1%). Additionally, RAG (46.2%) and Self-RAG (52.1%) exhibit lower
performance in this trend. We attribute this performance drop to the use of
retrieved evidence during generation. In other words, PubMedQA already
provides a golden context to solve questions, and Self-BioRAG further
introduces the top-10 evidence, potentially introducing errors due to excess
noisy information.

#### Case Report of Using Retrieved Evidence.

In Table 7, we present a MedQA dataset example to illustrate how Self-BioRAG
addresses a problem. For instance, a patient exhibits symptoms of physical
appearance changes, acne, and a family history of type 2 diabetes mellitus
(T2DM). Self-BioRAG determines the need to retrieve relevant documents
containing information on a female diagnosed with polycystic ovarian syndrome
(PCOS) and similar symptoms (e.g., T2DM and obesity). Self-BioRAG determines
the patient’s diagnosis as PCOS by integrating all three: patient’s symptoms,
retrieved evidence, and parametric knowledge. Throughout the query, evidence,
and prediction, we color-code using blue and red to distinguish two categories
of related snippets: 1) key information extracted from retrieved evidence and
2) the model’s essential parametric knowledge, both of which are leveraged by
Self-BioRAG to address the given problem. Overall, Self-BioRAG finds the clues
in the question, retrieves relevant documents if needed, and understands how
to answer with information of retrieved evidence and encoded knowledge same as
a medical expert would do.

##  6 Conclusion

In this manuscript, we introduce the Self-BioRAG framework, enabling a Self-
RAG (Asai et al., 2023) to generalize to biomedical and clinical domains of
instructions. This framework enhances the generation capacity, facilitates the
retrieval of factual content on demand, and enables self-assessment of
generated rationales. Our experimental results cover three multi-choice
question-answering benchmark datasets widely used in biomedical and clinical
domains. Self-BioRAG achieves a 7.2% absolute improvement compared to the
state-of-the-art model among the open foundation 7B models. We demonstrate the
necessity of domain-specific components, such as retriever, domain-related
document corpus, self-reflection model, and generator model, to address
domain-related instructions. We provide diverse analyses: (1) Self-BioRAG
retrieves a larger portion of evidence from Medical Textbook than other
corpora to solve USMLE-style questions; (2) Self-BioRAG can distinguish when
to retrieve evidence depending on instruction and question; (3) Provided
evidence from biomedical corpora genuinely helps supplement scarce knowledge.
However, we observe a limitation in closed-domain question-answering datasets,
such as PubMedQA, where, given the golden context, our retrieved evidence
could be erroneous due to an excessive amount of noisy information. In future
works, we aim to explore domain-specific reflective tokens, which could
enhance assessment and generation abilities with domain knowledge. Rather than
evaluating multi-choice question answering, we intend to generate long-form
text or tune our model for chat dialogue.

## References

  * Asai et al. (2023) Asai, A. et al. (2023).  Self-rag: Learning to retrieve, generate, and critique through self-reflection.  arXiv preprint arXiv:2310.11511.
  * Bajaj et al. (2016) Bajaj, P. et al. (2016).  Ms marco: A human generated machine reading comprehension dataset.  arXiv preprint arXiv:1611.09268.
  * Cao et al. (2022) Cao, M. et al. (2022).  Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
  * Chen et al. (2023) Chen, Z. et al. (2023).  Meditron-70b: Scaling medical pretraining for large language models.  arXiv preprint arXiv:2311.16079.
  * Christiano et al. (2017) Christiano, P. F. et al. (2017).  Deep reinforcement learning from human preferences.  Advances in neural information processing systems.
  * Chung et al. (2022) Chung, H. W. et al. (2022).  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416.
  * Dao et al. (2022) Dao, T. et al. (2022).  Flashattention: Fast and memory-efficient exact attention with io-awareness.  Advances in Neural Information Processing Systems.
  * Fang et al. (2023) Fang, Y. et al. (2023).  Mol-instructions: A large-scale biomolecular instruction dataset for large language models.  arXiv preprint arXiv:2306.08018.
  * Guo et al. (2003) Guo, G. et al. (2003).  Knn model-based approach in classification.  In On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003. Proceedings.
  * Guu et al. (2020) Guu, K. et al. (2020).  Retrieval augmented language model pre-training.  In International conference on machine learning.
  * Hendrycks et al. (2020) Hendrycks, D. et al. (2020).  Measuring massive multitask language understanding.  arXiv preprint arXiv:2009.03300.
  * Izacard et al. (2022a) Izacard, G. et al. (2022a).  Few-shot learning with retrieval augmented language models.  arXiv preprint arXiv:2208.03299.
  * Izacard et al. (2022b) Izacard, G. et al. (2022b).  Unsupervised dense information retrieval with contrastive learning.  Transactions on Machine Learning Research.
  * Ji et al. (2023) Ji, Z. et al. (2023).  Survey of hallucination in natural language generation.  ACM Computing Surveys.
  * Jiang et al. (2023) Jiang, Z. et al. (2023).  Active retrieval augmented generation.  arXiv preprint arXiv:2305.06983.
  * Jin et al. (2021) Jin, D. et al. (2021).  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  Applied Sciences.
  * Jin et al. (2019) Jin, Q. et al. (2019).  Pubmedqa: A dataset for biomedical research question answering.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  * Jin et al. (2023) Jin, Q. et al. (2023).  Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval.  Bioinformatics (Oxford, England).
  * Kang et al. (2023) Kang, M. et al. (2023).  Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks.  arXiv preprint arXiv:2305.18395.
  * Karpukhin et al. (2020) Karpukhin, V. et al. (2020).  Dense passage retrieval for open-domain question answering.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * Kitaev and Klein (2018) Kitaev, N. and Klein, D. (2018).  Constituency parsing with a self-attentive encoder.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  * Kitaev et al. (2019) Kitaev, N. et al. (2019).  Multilingual constituency parsing with self-attention and pre-training.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  * Kwon et al. (2023) Kwon, W. et al. (2023).  Efficient memory management for large language model serving with pagedattention.  In Proceedings of the 29th Symposium on Operating Systems Principles.
  * Lewis et al. (2020) Lewis, P. et al. (2020).  Retrieval-augmented generation for knowledge-intensive nlp tasks.  Advances in Neural Information Processing Systems.
  * Li et al. (2023) Li, B. et al. (2023).  Meddm: Llm-executable clinical guidance tree for clinical decision-making.  arXiv preprint arXiv:2312.02441.
  * Mao et al. (2021) Mao, Y. et al. (2021).  Generation-augmented retrieval for open-domain question answering.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
  * Nori et al. (2023) Nori, H. et al. (2023).  Capabilities of gpt-4 on medical challenge problems.  arXiv preprint arXiv:2303.13375.
  * OpenAI (2023a) OpenAI (2023a).  Chatgpt.
  * OpenAI (2023b) OpenAI (2023b).  Openai gpt-4 technical report.
  * Ouyang et al. (2022) Ouyang, L. et al. (2022).  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems.
  * Pal et al. (2022) Pal, A. et al. (2022).  Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.  In Conference on Health, Inference, and Learning.
  * Paszke et al. (2019) Paszke, A. et al. (2019).  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems.
  * Rajbhandari et al. (2020) Rajbhandari, S. et al. (2020).  Zero: Memory optimizations toward training trillion parameter models.  In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE.
  * Schulman et al. (2017) Schulman, J. et al. (2017).  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347.
  * Shao et al. (2023) Shao, Z. et al. (2023).  Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.  arXiv preprint arXiv:2305.15294.
  * Singhal et al. (2022) Singhal, K. et al. (2022).  Large language models encode clinical knowledge.  arXiv preprint arXiv:2212.13138.
  * Taori et al. (2023) Taori, R. et al. (2023).  Alpaca: A strong, replicable instruction-following model.  Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html.
  * Taylor et al. (2022) Taylor, R. et al. (2022).  Galactica: A large language model for science.  arXiv preprint arXiv:2211.09085.
  * Google (2023) Google (2023).  Bard.
  * Touvron et al. (2023) Touvron, H. et al. (2023).  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288.
  * Wang et al. (2022) Wang, Y. et al. (2022).  Self-instruct: Aligning language model with self generated instructions.  arXiv preprint arXiv:2212.10560.
  * Wang et al. (2023) Wang, Y. et al. (2023).  Augmenting black-box llms with medical textbooks for clinical question answering.  arXiv preprint arXiv:2309.02233.
  * Wang et al. (2019) Wang, Z. et al. (2019).  Multi-passage bert: A globally normalized bert model for open-domain question answering.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  * Wei et al. (2022) Wei, J. et al. (2022).  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems.
  * Wolf et al. (2019) Wolf, T. et al. (2019).  Huggingface’s transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771.
  * Wu et al. (2023) Wu, Z. et al. (2023).  Fine-grained human feedback gives better rewards for language model training.  arXiv preprint arXiv:2306.01693.
  * Zhang et al. (2023) Zhang, X. et al. (2023).  Alpacare: Instruction-tuned large language models for medical application.  arXiv preprint arXiv:2310.14558.

## Acknowledgements

We thank Gangwoo Kim, Hyeon Hwang, Chanhwi Kim, and Akari Asai for the
valuable discussions and feedback on our work. This work was supported in part
by the National Research Foundation of Korea [NRF-2023R1A2C3004176], the
Ministry of Health & Welfare, Republic of Korea [HR20C0021(3)], the Ministry
of Science and ICT (MSIT) [RS-2023-00262002], and the ICT Creative Consilience
program through the Institute of Information & Communications Technology
Planning & Evaluation(IITP) grant funded by the MSIT [IITP-2024-2020-0-01819].
This work was supported by a grant from the Korea Health Technology R&D
Project through the Korea Health Industry Development Institute (KHIDI),
funded by the Ministry of Health & Welfare, Republic of Korea [HR22C1302].

##  Appendix A Statistics of our generated Biomedical Instruction Set

In Figure 6, we depict a ROUGE-L score distribution and show that more various
instructions were generated relative to the biomedical seed tasks. We compute
the ROUGE-L overlap score with the seed tasks’ instructions to prevent
generating similar instructions as much as possible. Additionally, we observe
that the generated instruction, input, and output triplets show diverse length
distribution depending on the condition of instruction. In Table 8, we provide
other statistics such as the average length of instructions, inputs, and
outputs.

To guarantee the diverse range of instruction types, we analyze the biomedical
instructions using Berkeley Neural Parser888https://parser.kitaev.io/ (Kitaev
and Klein, 2018; Kitaev et al., 2019). We parse the generated instructions and
extract the top 20 root verbs in instructions and their direct noun objects.
Due to the requirement of the instruction-generation process, our instruction
is entirely related to biomedical terms such as report, hypothesis, and
diagnosis. We depict this information in Figure 5. Additionally, we provide a
prompt to contain a requirement of generating biomedical/medical/clinical
topics of instruction. We random sample 100 instances to manually measure the
category of instruction topics. Our instruction dataset covers various
biomedical topics such as text generation (42%), classification (15%), symptom
identification (7%), risk value prediction (6%), clinical note editing (5%),
etc.

Statistics |
---|---
# of total instances | 18,854
avg. instruction length (words) | 24.2
avg. input length (words) | 35.4
avg. output length (words) | 47.8
Table 8: Statistics of our generated Biomedical Instruction Set. Figure 5:
Nested piechart of our biomedical instructions root verbs (inner circle) and
their four noun objects (outer circle). It signifies the diversity of
generated instructions and relates to biomedical terms such as hypothesis,
proteins, diagnosis, and symptoms.

Figure 6: (a) Distribution of ROUGE-L scores between generated instructions
and their top-1 similar biomedical seed task instruction. (b) Length
distribution of generated instructions. (c) Length distribution of generated
inputs. (d) Length distribution of generated outputs. We set the max length of
the generated output length as 150.

##  Appendix B Comparison of Critic LM’s Performance

We split 5k instruction sets into train and dev for training and evaluating
critic language model performance. In Table 9, we use GPT-4 (OpenAI, 2023b) as
ground-truth predictions and identify how the critic language model of Self-
RAG (Asai et al., 2023) and Self-BioRAG performs. Due to the uncommon domain
of questions, Self-RAG exhibits lower performance compared to our Self-BioRAG.
The critic model of Self-BioRAG trained on Self-RAG predicts reflective tokens
of utility and relevance well. We find that predicting reflective tokens of
retrieval and supportiveness shows far lower than we expected. However, we
want to note that distilling knowledge of proprietary language models such as
GPT-4 to LLaMA2 (Touvron et al., 2023) can perform well in predicting
reflective tokens.

Model | Params. |  | Utility
---
(Acc.)
| Retrieval
---
(Acc.)
| Relevant
---
(Acc.)
| Supportive
---
(Acc.)
Self-RAG (Asai et al., 2023) | 7B | 78.2 | 65.8 | 88.5 | 69.9
Self-BioRAG (Ours) | 7B | 89.3 | 73.1 | 91.2 | 75.9
Self-BioRAG (Ours) | 13B | 91.2 | 75.3 | 91.8 | 77.2

Table 9: Critic LM performance on dev set using GPT-4 (OpenAI, 2023b)
predictions as ground-truth predictions.

Figure 7: Performance of LLaMA2, RAG, and Self-BioRAG on examples predicted as
[No Retrieval] and [Retrieval] by Self-BioRAG. Self-BioRAG consistently
outperforms other baselines whether or not retrieved evidence is used.

##  Appendix C Hyperparameters of Training Details

We use 8 Nvidia A100 with 80GB memory to train our Self-BioRAG models. We
train our critic model for 3 epochs with a learning rate of 2e-5 with a 1%
warmup rate. We also train our generator model for 5 epochs with a learning
rate of 2e-5 with a 3% warmup rate, and linear decay afterward. We set our
model maximum input length as 2048 for the 7B and 13B models to address at
least 3 shots in training and inference. We use Deepspeed stage 3 (Rajbhandari
et al., 2020) to implement multi-GPU settings and FlassAttention (Dao et al.,
2022) for efficient training. Our code is based on Self-RAG (Asai et al.,
2023), which is written in PyTorch (Paszke et al., 2019) and HuggingFace (Wolf
et al., 2019). We only try once to train our models, thus it could be useful
to search for better hyperparameters of our framework.

##  Appendix D How to Set Weight of Reflective Tokens

We compute scores of each critique type as normalized probabilities of desired
tokens same as Self-RAG (Asai et al., 2023). For the $REL$ token type, we
compute as below.

$S(REL)=\frac{p(relevant=yes)}{p(relevant=Yes)+p(relevant=No)}$ (2)

We compute $SUP$ using weighted scores to desirable tokens of $Fully$
$supported$ as 1 and $Partially$ $supported$ as 0.5. We compute our $SUP$
token type as follows:

$S(SUP)=\frac{p(sup=Fully)+0.5*p(sup=Partial)}{p(sup=Fully)+p(sup=Partial)+p(sup=No)}$
(3)

We set a weighted scores $w$ of {-1, -0.5, 0, 0.5, 1} to the tokens of
$USE=${1, 2, 3, 4, 5}. We compute $USE$ token type as follows:

$S(USE)=\sum_{i=1}^{5}w_{i}\frac{p(utility=i)}{\sum_{t=1}^{5}p(utility=t)}$
(4)

##  Appendix E Distinguishing When to Retrieve in MedMCQA and MMLU-Med
Datasets

In Figure 7, we evaluate the performance of LLaMA2, RAG (LLaMA2 with MedCPT
and biomedical corpora), and Self-BioRAG on examples predicted as [No
Retrieval] and [Retrieval] by Self-BioRAG. We could also observe that a
similar trend is found in MedMCQA (Pal et al., 2022) and MMLU-Med (Hendrycks
et al., 2020) datasets. Experimental results demonstrate that Self-BioRAG
consistently outperforms other baselines, whether or not retrieved evidence is
used. Interestingly, we observe that the performance of the MedMCQA dataset
shows 91.1% accuracy on retrieved questions. This also aligns with the result
that the hard constraint of using retrieved evidence in all questions shows
high performance compared to not retrieving any evidence or adaptively
retrieving as shown in Table 6. We posit that the MedMCQA dataset contains
questions requiring external knowledge to retrieve.

##  Appendix F Detailed performance of MMLU-Med Datasets

To measure medical knowledge, we evaluate the Massive Multitask Language
Understanding (MMLU) (Hendrycks et al., 2020) dataset and extract six medical
and clinical domains to construct MMLU-Med benchmark as follows: Anatomy,
Clinical Knowledge, College Biology, College Medicine, Medical Genetics, and
Professional Medicine. In Table 10, we provide detailed performances of the
MMLU-Med benchmark. Our model Self-BioRAG outperforms other baselines used in
the open foundation language model and open foundation model with language
model on the MMLU-Med benchmark.

|  | MMLU Benchmark
---|---|---
| Model
---
| Params.
---
MMLU-AN | MMLU-CK | MMLU-CB | MMLU-CM | MMLU-MG | MMLU-PM
Open LM |  |  |  |  |  |  |
Alpaca | 7B | 44.4 | 35.5 | 23.6 | 31.2 | 37.0 | 34.9
FLAN-T5 | 3B | 42.2 | 53.6 | 41.7 | 41.6 | 43.0 | 43.0
Galactica | 6.7B | 40.7 | 41.5 | 48.6 | 34.1 | 48.0 | 23.5
MEDITRON | 7B | 45.1 | 39.5 | 43.2 | 41.1 | 38.0 | 39.8
LLaMA2 | 7B | 45.9 | 52.5 | 38.9 | 45.6 | 55.0 | 40.0
Open LM + Retrieval |  |  |  |  |  |  |
Self-RAG | 7B | 42.2 | 49.1 | 51.0 | 39.9 | 52.0 | 40.0
Self-RAG† | 7B | 45.2 | 55.1 | 52.8 | 39.9 | 55.0 | 39.7
Self-BioRAG (Ours) | 7B | 51.1 | 60.8 | 60.4 | 50.9 | 52.0 | 48.2
Self-BioRAG (Ours) | 13B | 51.9 | 60.8 | 66.7 | 51.4 | 60.0 | 52.2

Table 10: Experimental results of biomedical benchmark datasets. We use 3-shot
examples to guide how to solve a benchmark example to language models. We use
the same examples described in the appendix of MedPALM (Singhal et al., 2022)
for the MMLU dataset. We refer to each dataset with the abbreviation of full
name: Anatomy (AN), Clinical Knowledge (CK), College Biology (CB), College
Medicine (CM), Medical Genetics (MG), and Professional Medicine (PM). † use
biomedical corpora (e.g., PubMed, PMC, Clinical Guideline, and Medical
Textbook) as evidence during inference.

##  Appendix G Prompt of Data Generation for Self-Reflection Language Model

We provide prompts to generate data for the self-reflection language model
(critic language model). In Table 11, utility reflective tokens are
distinguished in 5 scores, ranging from 1 to 5. We use the utility to check
that the output is a useful response to the instruction. In Table 12, we
decide if the output can be verified solely with suggested evidence on
retrieval reflective tokens. In Table 13, we use relevant reflective tokens to
determine if the evidence is relevant to the instruction and the preceding
context. Additionally, we use supportive reflective tokens to check an output
sentence is supported by the evidence, described in Table 14.

Prompt of Reflective token type Utility (USE)
---
Given an instruction and an output, rate whether the response appears to be a
helpful and informative answer to the query, from
1 (lowest) - 5 (highest). We call this score perceived utility.
The detailed criterion is as follows:
5: The response provides a complete, highly detailed, and informative response
to the query, fully satisfying the information needs.
4: The response mostly fulfills the need in the query, while there can be some
minor improvements such as discussing more detailed
information, having better structure of the response, or improving coherence.
3: The response is acceptable, but some major additions or improvements are
needed to satisfy users’ needs.
2: The response still addresses the main request, but it is not complete or
not relevant to the query.
1: The response is barely on-topic or completely irrelevant.
### Instruction: Generate a description of the structure and mechanism of
action of the enzyme nicotinamide adenine dinucleotide
phosphate (NADPH) oxidase.
Output: NADPH oxidase is a membrane-bound enzyme that plays an important role
in the production of reactive oxygen species. It
is composed of six distinct subunits and is found in both the cytosol and the
membrane. The subunits are composed of two copies
each of the cytosolic subunits: p47 phox, p67 phox, p40 phox, and two copies
of the membrane-associated subunits: gp91phox and
p22phox. NADPH oxidase functions by donating electrons to the active site of
the enzyme, thus allowing it to reduce oxygen mole
-cules to form superoxide anion. The end product of this reaction is the reactive oxygen species, which is used to fight infection or
for other body functions. NADPH oxidase is regulated by various signals such
as calcium, nicotinamide adenine dinucleotide phos
-phate (NADP), and calcium binding proteins. In addition, it is inhibited by certain compounds, such as apocynin, that prevent its
activation. NADPH oxidase plays an important role in the body’s defenses
against infection, and it is thus essential for many biolo
-gical processes.
Perceived utility: 5
Explanation: This response provides a descriptive explanation of the structure
and function of NADPH oxidase, giving full and
comprehensive information. Thus, this is deemed a high utility response
satisfying the user’s information needs.
### Instruction: Was the enhancement of Lck activity immediate?
Output: No, the enhancement of Lck activity did not have an immediate
relevance to TCR-mediated signal transduction, as judged
from the lack of detectable de novo phosphorylation of zeta-chain and the
absence of functional responses of T cells.
Perceived utility: 3
Explanation: The response robustly addresses the aspect of Lck activity
enhancement not being immediate. However, the details
provided thereafter are quite dense and technical, and without proper context,
they may not be easy to interpret for all readers.
Simplifying this information or providing some additional context could
improve the perceived utility.
### Instruction: Alcohol intake during pregnancy causes, all, except?
Output: Brachycephaly
Perceived utility: 5
Explanation: This is a concise and correct answer to the posed question,
focusing on the exclusion of one potential effect of alcohol
intake during pregnancy. The response specifically points out that
brachycephaly is not caused by alcohol intake during pregnancy,
thus correctly answering the query. There’s not much room for more elaborate
discussion in this case, making this answer complete
and highly useful for the user’s information needs in this context.
### Instruction: {Example Instruction}
Output: {Example Output}
Perceived utility:

Table 11: Prompt of Utility Reflective token. We use 3-shots to generate
reflective tokens via GPT-4 API calls.

Prompt of Reflective token type Retrieval (RET)
---
You will be provided with an instruction, evidence, output sentence, and
preceding sentences (optional). If the preceding sentence
is given, the output should be the sentence that follows those preceding
sentences. Your task is to determine whether the informat
-ion in the output sentence can be fully verified by the evidence or if it requires further external verification.
If the output sentence can be verified solely with the evidence or doesn’t
require any verification, respond with [No Retrieval].
If additional information is needed to verify the output sentence, respond
with [Retrieval]. Please provide explanations for your
judgments.
### Instruction: What is the mechanism of 72T4Cl activation under
physiological conditions?
Evidence: Timing mechanism and effective activation energy concerned with
aging and lifespan in the long-lived and thermosens
-ory mutants of Caenorhabditis elegant.. The lifespans of many poikilothermic animals, including the nematode Caenorhabditis
elegans, depend significantly on environmental temperature. Using long-living,
thermosensory mutants of C.elegans, we tested
whether the temperature dependency of the mean lifespan is compatible with the
Arrhenius equation, which typically represents
one of the chemical reaction rate theories. The temperature dependency of C.
elegans was the Arrhenius type or normal, but daf-
2(e1370) mutants were quite different from the others. However, taking into
account the effect of the thermal denaturation of DA
-F-2 with the temperature, we showed that our analyzed results are compatible with previous ones. We investigated the timing me
-chanism of one parameter (the onset of biodemographic aging (t(0))) in the lifespan equation by applying the RNAi feeding met
-hod to daf-2 mutants in order to suppress daf-16 activity at different times during the life cycle. In summary, we further deepened
the biological role of two elements, t(0) and z (the inverse of the aging
rate), in the lifespan equation and mean lifespan formulated
by our diffusion model z(2) = 4Dt(0), where z is composed of t(0) and D (the
diffusion constant).
Output: The mechanism of 72T4Cl activation under physiological conditions is
not known.
Rating: [No Retrieval]
Explanation: The output sentence does not provide any factual information that
needs to be verified, it is stating a fact that the mec
-hanism is not known. The evidence does not mention 72T4Cl at all, supporting the claim that the mechanism of its activation is
not known.
### Instruction: Pigment providing colour to stool is
Evidence: The importance of stool tests in diagnosis and follow-up of
gastrointestinal disorders in children. Stool samples should be
evaluated macroscopically in terms of color, consistency, quantity, form,
odor, and presence of mucus. The presence of a small amount
of mucus in stool is normal. However, the presence of copious mucus or bloody
mucus is abnormal. The normal color is tawny due to
the presence of bilirubin and bile. In infants, the stool may be green, its
consistency may be watery or pasty. Stool color varies greatly
depending on diet. Clay-colored or putty colored stool is observed in biliary
obstructions. If more than 100 mL blood is lost from the
upper gastrointestinal system, black, tarry stool is observed. Besides
bleeding, black-colored stool may also be observed due to iron or
bismuth treatment. Red-colored stool is observed in lower gastrointestinal
tract bleeding.
Output: Stercobilinogen. The characteristic brown colour of feces is due to
the presence of pigment stercobilin (derived from
stercobilinogen). So, actually it is stercobilin (not stercobilinogen) which
is responsible for the brown color of feces. Stercobilin is
derived from stercobilinogen and therefore stercobilinogen is the best answer
amongst the options provided.
Rating: [Retrieval]
Explanation: The provided evidence does not mention stercobilin or
stercobilinogen providing the color to stool, although it does
mention bilirubin and bile contribute to the color. Therefore, additional
evidence would be necessary to verify this output sentence.
### Instruction: {Example Instruction}
Preceding sentences: {Example Preceding sentences}
Evidence: {Example Evidence}
Output: {Example Output}
Rating:

Table 12: Prompt of Retrieval Reflective token. We use 2-shots to generate
reflective tokens via GPT-4 API calls.

Prompt of Reflective token type Relevance (REL)
---
You’ll be provided with an instruction, along with evidence and possibly some
preceding sentences. When there are preceding
sentences, your focus should be on the sentence that comes after them. Your
job is to determine if the evidence is relevant to the
initial instruction and the preceding context, and provides useful information
to complete the task described in the instruction.
If the evidence meets this requirement, respond with [Relevant]; otherwise,
generate [Irrelevant].
### Instruction: Based on the provided genetic information, classify whether
the mutation described predisposes carriers to
breast cancer. The provided dataset is a genetic report showing a mutation in
the BRCA1 gene.
Evidence: TBRCA mutations: is everything said? BACKGROUND: Mutations in the
BRCA1 and BRCA2 genes constitute a
risk factor for breast cancer development. BRCA mutation research has been an
active field since the discovery of the genes,
and new mutations in both genes are constantly described and classified
according to several systems. AIM: We intend to provide
an overview of the current state of BRCA1 and BRCA2 mutation description and
classification. We wanted to know whether
there was a trend towards a more frequently described mutation type and what
the proportion of pathogenic mutations was.
RESULTS: We found that, although new mutations are described each year as
reflected in current database records, very few of
them are reported in papers. Classification systems are highly heterogeneous
and a consensus among them
Rating: [Relevant]
Explanation: The evidence mentions BRCA1/2 mutation probability models and
their potential role in predicting breast cancer
risk, directly relevant to the instruction about classifying the
predisposition to breast cancer based on a mutation.
### Instruction: {Example Instruction}
Evidence: {Example Evidence}
Rating:

Table 13: Prompt of Relevant Reflective token. We use 1-shot to generate
reflective tokens via GPT-4 API calls.

Prompt of Reflective token type Supportive (SUP)
---
You will receive an instruction, evidence, and output, and optional preceding
sentences. If the preceding sentence is given, the
output should be the sentence that follows those preceding sentences. Your
task is to evaluate if the output is fully supported by
the information provided in the evidence, and provide explanations on your
judgement
Use the following entailment scale to generate a score:
$[$Fully supported$]$ \- All information in output is supported by the
evidence, or extractions from the evidence. This is only applicable
when the output and part of the evidence are almost identical.
$[$Partially supported$]$ \- The output is supported by the evidence to some
extent, but there is major information in the output that is
not discussed in the evidence. For example, if an instruction asks about two
concepts and the evidence only discusses either of them,
it should be considered a $[$Partially supported$]$.
$[$No support / Contradictory$]$ \- The output completely ignores evidence, is
unrelated to the evidence, or contradicts the evidence.
This can also happen if the evidence is irrelevant to the instruction.
Make sure to not use any external information/knowledge to judge whether the
output is true or not. Only check whether the output is
supported by the evidence, and not whether the output follows the instructions
or not.
### Instruction: Ask about your symptoms and potential cardiovascular-related
issues. I’ve been feeling intense discomfort in my
chest during physical exertion. It sometimes shoots up to my left shoulder.
There’s also some mild shortness of breath.
Output: Based on the symptoms you have described, it is important to consider
potential cardiovascular-related issues. Chest
discomfort during physical exertion, shooting pain to the left shoulder, and
mild shortness of breath can be indicative of a heart
problem. These symptoms could potentially be related to angina, which is chest
pain or discomfort caused by reduced blood flow
to the heart. It commonly occurs during exercise or periods of stress. Another
possibility is a myocardial infarction (heart attack),
where there is a blockage in one of the coronary arteries supplying blood to
the heart muscle. Given the seriousness of these
symptoms, I would strongly advise you to seek immediate medical attention and
consult with a healthcare professional for a proper
evaluation and diagnosis. They will be able to conduct a thorough examination,
ask additional questions about your medical history,
possibly order tests such as an electrocardiogram (ECG) or stress test, and
provide appropriate treatment recommendations based on
their findings.
Evidence: Disparities in Awareness of Myocardial Infarction and Stroke
Symptoms and Response Among United States– and Foreign‐
Born Adults in the National Health Interview Survey. Knowledge of MI symptoms
was assessed based on responses to the question,
“Which of the following would you say are the symptoms that someone may be
having a heart attack?” Responses were pain or
discomfort in the jaw, neck, or back; feeling weak, lightheaded, or faint;
chest pain or discomfort; pain or discomfort in the arms or
shoulder; and shortness of breath. Similarly, knowledge of stroke symptoms was
assessed by an individual’s response to the question,
“Which of the following would you say are the symptoms that someone may be
having a stroke?” Responses were sudden numbness
or weakness of face, arm, or leg, especially on one side; sudden confusion or
trouble speaking; sudden trouble seeing in one or both
eyes; sudden trouble walking, dizziness, or loss of balance; and sudden severe
headache with no known cause. Respondents who answered
“yes” to the 5 queries on MI and stroke symptoms were classified as knowing
the symptoms of MI and stroke. Those who answered “no,”
“don’t know,” or refused to answer were classified as not knowing the
symptoms.
Score: [Fully supported]
Explanation: The output correctly identifies potential cardiovascular-related
issues including angina and myocardial infarction (heart
attack) based on the described symptoms. The evidence supports these concerns,
mentioning commonly recognized symptoms of a
heart attack including chest pain or discomfort, shooting pain in the
shoulder, and shortness of breath. The output also correctly advises
seeking immediate medical attention and consultation with a healthcare
professional, which is an appropriate advice given the severity
of these conditions.
### Instruction: {Example Instruction}
Preceding sentences: {Example Preceding entences}
Output: {Example Output}
Evidence: {Example Evidence}
Score:

Table 14: Prompt of Supportive Reflective token. We use 1-shot to generate
reflective tokens via GPT-4 API calls.

##  Appendix H Examples of Biomedical Instructions to Train Domain-Specific
Instruction-Tuned Language Model

We collect 84k biomedical instruction sets to train a domain-specific
instruction-tuned language model. For example, in Table 15, we provide several
cases used in training the language model: (1) Our critic language model
decides to not retrieve any evidence about generating an explanation of the
human cardiovascular system. (2) Retrieved evidence contains information on
symptoms of Type 1 diabetes mellitus: urination, loss of weight, and stomach
pain. Therefore, our critic model predicts that the output sentence (e.g.,
"answer is Type 1 diabetes mellitus") fully supports the evidence. (3) The
critic model decides that more evidence is needed to answer the following
question. Even if relevant evidence is retrieved (e.g., information about
cystic fibrosis), the evidence does not support solving the original question.
Overall, our critic language model predicts whether to retrieve or not based
on the example instruction

Cases of Instruction used at Training Generator Model
---
No Retrieval
Instruction: Write a simple explanation of the human cardiovascular system
suitable for a non-medical audience. <noinput>
Output: [No Retrieval] It consists of the heart, blood vessels, and blood. [No
Retrieval] Think of the heart as a powerful pump
that never stops working. [No Retrieval] Its main job is to pump blood
throughout the body. [No Retrieval] The blood vessels
are like roads or highways that carry the blood to different parts of the
body. [No Retrieval] There are three types of blood
vessels: arteries, veins, and capillaries. [No Retrieval] Arteries carry
oxygen-rich blood away from the heart to the body’s organs
and tissues. [No Retrieval] Veins bring back oxygen-poor blood from the organs
and tissues to the heart. [No Retrieval] Capillaries
are tiny blood vessels that connect the arteries and veins, allowing oxygen
and nutrients to pass through to the cells and removes
waste products from them. [No Retrieval] Blood is the fuel that keeps
everything running smoothly. [No Retrieval] It carries
oxygen and nutrients from the lungs and digestive system to all the cells in
your body. [No Retrieval] It also carries waste products,
such as carbon dioxide, away from the cells so they can be eliminated. [No
Retrieval] So, the heart pumps the blood, the blood
vessels deliver it, and the blood carries important substances to where
they’re needed, keeping the body healthy. [No Retrieval]
In summary, the human cardiovascular system is like a transportation network
that transports blood, nutrients, and oxygen to all
parts of our body and removes waste products. [No Retrieval] The heart acts as
a pump, blood vessels are the roads, and blood is
the fuel that keeps everything running smoothly. [Utility:5]
Retrieval (Fully Supported)
Instruction: Classify whether this description corresponds to Type 1 or Type 2
diabetes mellitus. The patient showed onset of
symptoms during childhood, including excessive thirst, frequent urination,
rapid weight loss, and stomach pain. The tests
confirmed low insulin levels and absence of insulin resistance
Output: $[$Retrieval$]$ $<$paragraph$>$ beta cells have been destroyed. About
two-thirds of body cells require insulin to absorb glucose
and in its absence, they will not be able to use the glucose (blood sugar) for
energy. Type 1 diabetes usually comes on abruptly,
although the damage to the beta cells may begin much earlier. Typical signs of
Type 1 diabetes are a great thirst, hunger, a need to
urinate often, and loss of weight. To treat the disease, the person must
inject insulin, follow a diet plan, exercise daily, and test blood
glucose several times a day. Type 1 diabetes usually occurs in children and
adults who are under age 30. This type of diabetes used
to be known as "insulin-dependent diabetes," "juvenile diabetes," "juvenile-
onset diabetes" and "ketosis-prone diabetes." - Diabetes
mellitus $<$$/$paragraph$>$ $[$Relevant$]$ Based on the description provided,
this corresponds to Type 1 diabetes mellitus. $[$Fully supported$]$
$[$No Retrieval$]$ Type 1 diabetes mellitus is typically characterized by an
early onset of symptoms, often during childhood or adolescence.
$[$No Retrieval$]$ Common symptoms include excessive thirst (polydipsia),
frequent urination (polyuria), rapid weight loss, and
abdominal pain. $[$No Retrieval$]$ In addition, individuals with Type 1
diabetes typically have low insulin levels and no insulin resistance.
$[$No Retrieval$]$ Therefore, based on the symptoms and test results
described, the answer is Type 1 diabetes mellitus. $[$Utility:5$]$
Retrieval (No Support / Contradictory)
Instruction: A seven year old child with recurrent chest infections and
exocrine pancreatic insufficiency is suspected of having cystic
fibrosis. Sweat chloride levels have been observed to be between 40-60 mmol/l
on two separate occasions. Which of the following test
should be performed next to suppo the diagnosis of Cystic fibrosis? What of
the following is the right choice?
(A) Repeat sweat chloride levels on a different day
(B) Demonstrate an abnormal 72 hour fecal fat collection
(C) Demonstrate an abnormal nasal potential difference
(D) Demonstrate an abnormal F508 mutation by DNA analysis
Output: $[$Retrieval$]$ $<$paragraph$>$ to expert centers where bioassays of
CFTR function like nasal potential difference measurement or
intestinal current measurement can be done. Still, in some patients, despite
symptoms compatible with CF and some indication of CFTR
dysfunction (e.g. only intermediate sweat chloride value), diagnostic criteria
are not met (e.g. only 1 CFTR mutation identified). For these
subjects, the term CFTR related disorder (CFTR-RD) is used. Patients with
disseminated bronchiectasis, congenital bilateral absence of
the vas deferens and acute or recurrent pancreatitis may fall in this
category. CF has a very wide disease spectrum and increasingly the
diagnosis is being made during adult life, mainly in subjects with milder
phenotypes. In many countries, nationwide CF newborn
screening (NBS) has been introduced. In screen positive babies, the diagnosis
of CF must be $<$$/$paragraph$>$ $[$Relevant$]$ The final answer
is (C). $[$No support / Contradictory$]$ $[$Utility:4$]$

Table 15: Cases of Instruction used at Training Generator Model. The Critic LM
$C$ predicts retrieval of any evidence depending on the instruction. Given
biomedical evidence, the model $C$ predicts whether the original output
supports the evidence or not.