# Appendix for PromptCBLUE

Wei Zhu∗, Xiaoling Wang, Huanran Zheng,
East China Normal University
Mosha Chen
Holoflow Digital Technology, Hangzhou
&Buzhou Tang
Harbin Institute of Technology, Shenzhen
Equal contributions. Corresponding author<EMAIL_ADDRESS>Work done at
Alibaba Group.

##  Appendix A Appendix

###  A.1 Prompt and target formats for PromptCBLUE

We now present one example of the prompt and response formats for each sub-
task of PromptCBLUE. For complete set of prompt templates, please refer to our
project page https://github.com/michael-wzhu/PromptCBLUE.

CMeEE-v2 As a standard named entity recognition task in the medical domain,
when given a input text sequence, one has to extract the medical entities,
including the text mentions’ spans and entity types, in the sequence.
Recently, with the prevailing of generative LM, a series of literature no
longer requires the model to predict the entity spans. Thus, we formulate
CMeEE-v2 task as the task of generating all the entity mentions. With the idea
of Wei2022ChainOT, the prompt will ask the LLMs to generate entities type-by-
type. In comparison, without COT, the LLMs are asked to generate entities one
by one.

Prompt Template with COT:

    : (Medical named entity recognition: )
    [sentence]
    : [entity_types] (Choices of entity types: [entity_types])
     (Instruction: please generate
    named entities type-by-type. )
    : (Answer: )

Prompt Template without COT:

    : (Medical named entity recognition: )
    [sentence]
    : [entity_types] (Choices of entity types: [entity_types])
    : (Answer: )

Target Template with COT:

    : (The above sentence have the following entities: )
    [entity\_type]: [entity_mentions] ([entity_type] entities:
    [entity_mentions])

Target Template without COT:

    : (The above sentence have the following entities: )
    [entity\_mention]:[entity\_type]
    [entity\_mention]:[entity\_type]

Example prompt with COT:

    : (Medical named entity recognition: )
     (The peripheral blood white blood cell
    count is often significantly elevated, with left nuclear shift.)
    :  (Choices of entity types: diseases, medical test items, hospital
    departments, body parts, microorganisms, clinical manifestations,
    drugs)
     (Instruction: please generate
    named entities type-by-type. )
     (Answer: )

Example prompt without COT:

    : (Medical named entity recognition: )
     (The peripheral blood white blood cell
    count is often significantly elevated, with left nuclear shift.)
    :  (Choices of entity types: diseases, medical test items, hospital
    departments, body parts, microorganisms, clinical manifestations,
    drugs)
     (Answer: )

Example target with COT:

    : (The above sentence have the following entities: )
    :  (medical test items entities:
    peripheral blood white blood cell count)
    :  (disease entities: None)
    :  (hospital department entities: None)
    :  (drug entities: None)
    :  (microorganism entities: None)

Example target without COT:

    : (The above sentence have the following entities: )
    :

Note that we also want LLM to determine whether there are entities for a given
type in the sentence.

CMeIE The CMeIE task is a triple extraction task, that is, the model has to
predict the head entity information (including mention span and entity type),
tail entity information (including mention span and entity type), and the
relation type between the two entities. Similar as the CMeEE task, we
reformuate this task as the task of generating the head-tail entity pair under
the given relation types. The following are the prompt and target templates,
in which [relations] denotes the relation names, and [sentence] denotes the
input text. An example pair of prompt and target response is also presented.

Prompt Template:

    [relations]: (Find the entity pairs that
    have the [relations] relations: )
    [sentence]

Target Template:

    : (the entity pairs and their relations are:)
    [triple]
    [triple]

Example prompt:

     (Find the entity pairs that have the clinical finding or
    synonym relations:)
    @ (systemic vascular resistance
    SVR) SVR (Hypertension during pregnancy@healthy pregnant women can be
    balanced by a decrease in systemic vascular resistance (SVR) and
    cardiac output, whereas women with gestational hypertension have a
    lower SVR and tend to have an increase in cardiac output.)

Example target:

     (the entity pairs and their relations are
    as follows: )
    SVR (head entity: hypertension during pregnancy, tail entity: SVR
    is lower; relation: clinical finding)
     (head entity: hypertension during pregnancy, tail entity: SVR
    is lower; head entity: hypertension during pregnancy, tail entity:
    cardiac output tends to increase; relation: clinical finding)

With the idea of COT, LLMs will deal with this task with a more suitable
strategy: first predict the relations in a given sentence, and then extract
the triples type by type.

Prompt Template with COT:

    [relations]: (Find the entity pairs that
    have the [relations] relations: )
    [sentence]
     (Instruction: Please determine whether there is a certain relation
    in the sentence. If so, extract the entity pair with this
    relationship. )

Target Template with COT:

    : [existing-relations]
    [relation]: [triples] (the entity pairs having
    the [relation] relation are as follows: [triples])
    [relation]: [triples] (the entity pairs having
    the [relation] relation are as follows: [triples])

Example prompt with COT:

     (Find the entity pairs that have the clinical finding or
    synonym relations:)
    @ (systemic vascular resistance
    SVR) SVR (Hypertension during pregnancy@healthy pregnant women can be
    balanced by a decrease in systemic vascular resistance (SVR) and
    cardiac output, whereas women with gestational hypertension have a
    lower SVR and tend to have an increase in cardiac output.)
     (Instruction: Please determine whether there is a certain relation
    in the sentence. If so, extract the entity pair with this
    relationship. )

Example target with COT:

    :
     (the entity pairs having
    the clinical finding relation are as follows: )
    SVR (head entity: hypertension during pregnancy, tail entity: SVR
    is lower; head entity: hypertension during pregnancy, tail entity:
    cardiac output tends to increase;)

CHIP-CDEE In this task, one needs to extract clinical findings as medical
events in a given medical report. An clinical finding event consists of a 主体词
(trigger), and three attributes, 发生状态 (occurrence status), 描述词 (descriptor)
and 解剖部位 ( anatomical part). In PromptCBLUE, this task is reformulated as
generating event descriptions for clinical findings given a medical report.

Prompt Template:

    [sentence]
     (Question: What are the clinical
    findings and their attributes in the sentence?)
     (Note: a clinical finding event consists of trigger, occurrence
    status, descriptor and anatomical part.)

Target Template:

     (The clinical finding events in the above
    sentence are as follows:)
    [str][str][str][str] (trigger: [str]; occurrence status: [str]; descriptor:
    [str]; anatomical part: [str])

Example prompt:

    7() 10mg 146 20mg bid*145() 10mg 143fishpml/rara(15/17)(Before July, patients were given 10mg of sodium arsenite chloride
    (Itada) injection intravenously for 14 days. Before June, they were
    given oral retinoic acid 20mg bid * 14 days for maintenance treatment.
    Before May, patients were given 10mg of sodium arsenite chloride
    (Itada) injection intravenously for 14 days for maintenance
    chemotherapy. After more than 3 months, bone marrow examination
    showed proliferative myelogram; Fish: pml/rara (dual color dual
    fusion) (15/17): no abnormalities found; No abnormal cells were
    found in the cerebrospinal fluid through lumbar puncture. I am
    currently admitted for maintenance chemotherapy.)
     (Question: What are the clinical
    findings and their attributes in the sentence?)
     (Note: a clinical finding event consists of trigger, occurrence
    status, descriptor and anatomical part.)

Example target:

     (The clinical finding events in the above
    sentence are as follows:)
    fishpml/rara(15/17) (trigger: Fish: pml/rara (dual color dual fusion) (15/17) anomaly;
    occurrence status: not found; descriptor: none; anatomical part: none)
     (trigger: bone marrow imaging; occurrence status: none; descriptor:
    proliferative; anatomic site: bone marrow)

With COT, we ask the LLMs to conduct event extraction step-by-step. That is
,we first ask the LLMs to detect trigger words, and then we will extract the
attributes for each trigger word to complete the events.

Prompt Template with COT:

    [sentence]
     (Question: What are the clinical
    findings and their attributes in the sentence?)
     (Note: a clinical finding event consists of trigger, occurrence
    status, descriptor and anatomical part.)

Target Template with COT:

     (The trigger words of the clinical
    finding events in the above sentence are as follows:)
    [triggers]
    [trigger][str][str][str] (the attributes of the trigger word [trigger]: [str];
    occurrence status: [str]; descriptor: [str]; anatomical part: [str])

Example prompt with COT:

    7() 10mg 146 20mg bid*145() 10mg 143fishpml/rara(15/17)(Before July, patients were given 10mg of sodium arsenite chloride
    (Itada) injection intravenously for 14 days. Before June, they were
    given oral retinoic acid 20mg bid * 14 days for maintenance treatment.
    Before May, patients were given 10mg of sodium arsenite chloride
    (Itada) injection intravenously for 14 days for maintenance
    chemotherapy. After more than 3 months, bone marrow examination
    showed proliferative myelogram; Fish: pml/rara (dual color dual
    fusion) (15/17): no abnormalities found; No abnormal cells were
    found in the cerebrospinal fluid through lumbar puncture. I am
    currently admitted for maintenance chemotherapy.)
     (Question: What are the clinical
    findings and their attributes in the sentence?)
     (Note: a clinical finding event consists of trigger, occurrence
    status, descriptor and anatomical part.)

Example target with COT:

     (The trigger words of the clinical
    finding events in the above sentence are as follows:)
    fishpml/rara(15/17)
    \"pml/rara(15/17)\" (trigger word \"Fish: pml/rara (dual color dual fusion) (15/17)
    anomaly\" has the following attributes: occurrence status: not
    found; descriptor: none; anatomical part: none)
    \"\" (trigger: bone marrow imaginghas the following attributes:
    occurrence status: none; descriptor: proliferative; anatomic site:
    bone marrow)

CHIP-CDN This task aims to map the diagnosis terms (query terms) written by
doctors to standardized disease terms (target terms) according to a certain
medical terminology standard jmir. The number of standardized terms may exceed
10 thousands. The standard system adopted in this task is usually the
International Statistical Classification of Diseases and Related Health
Problems 10th Revision
(ICD-10)111https://www.who.int/standards/classifications/classification-of-
diseases. ICD-10 has more than 30 thousand disease terms, thus it is
prohibitive to feed all the disease terms into ChatGPT. ICD-10 has different
versions in different countries, and in this work we adopt the ICD-10 Beijing
Clinical Trial (version v601)222http://www.cips-chip.org.cn/2021/eval3. We
will refer to this Chinese version of ICD-10 as ICD-10-Beijing.

Recently, this task is usually addressed by a retrieve-and-rank system
Park2019AnIR. Firstly, a small set of candidate standard terms are retrieved
via inverted index (e.g., ElasticSearch) or a semantic index toolkit like
Faiss, then a ranking model will score and rank the relevances of each query-
target term pair. To construct the data samples in CBLUE, for each query term
in the task datasets, we first retrieve candidate target terms using BM25
Robertson2009ThePR. The candidate pool consists of the top 10-30 terms
retrieved by BM25 not are not in the ground truth, and the ground truth terms
(for 80% of the cases). The candidate pool’s order is shuffled so that LLMs
can not obtain the true answers by just selecting the first few candidates.
Thus, under PromptCBLUE, LLMs act as a ranking model, and choose the final
target terms among the candidates or determine no proper target terms are
present in the candidate pool.

Prompt Template:

    [sentence]
    [candidate-terms] (choices of terms[candidate-terms])
    ICD-10 (Task explanation: select the terms that match the original
    diagnostic description from several candidate ICD-10 diagnostic
    standard terms)
     (Answer)

Target template:

    [selected-terms]

Example prompt:

     (Aortic arch constriction and cardiac dysfunction)
     (choices of termspseudoaneurysm of the anterior tibial artery,
    coarctation of the aorta, male hypogonadism, male hypogonadism,
    pituitary dysfunction, and cardiac insufficiency)
    ICD-10 (Task explanation: select the terms that match the original
    diagnostic description from several candidate ICD-10 diagnostic
    standard terms)
     (Answer)

Example target:

     (coarctation of the aorta, cardiac insufficiency)

CHIP-CTC The main objective of this task is to classify the text input into a
type of clinical trial screening criteria (44 types). Under PromptCBLUE, we
transform the task to the task of generate the type of clinical trial
screening criteria for the give text input and the given label set. LLMs need
to respond with ”Not the above type” to indicate that the given label types in
the prompt does not match the sentence.

Prompt template:

    [sentence]
     (What is the type of clinical trial
    screening criteria for this sentence?)
    [candidate-types] (Choices of categories: [candidate-types])

Target template:

    [selected-type]

Example prompt:

    8.3 (8. Patients who have undergone intraocular
    surgery within the past 3 months;)
     (What is the type of clinical trial
    screening criteria for this sentence?)
    ()( (Choices of categories: addiction behavior, smoking status, sexual
    orientation, disabled population, reading and writing ability, tumor
    progression, participation in other trials, disease staging, ability,
    disease, medication, diagnosis, educational status, oral related,
    receptor status, healthy population, data accessibility, equipment,
    blood donation, allergic tolerance, special patient characteristics,
    sleep, pregnancy related, researcher decision, organ and tissue status,
    symptoms (patient perception), treatment or surgery, Nursing, gender,
    race, laboratory examination, informed consent, diet, age, residential
    status, source of case, alcohol use, physical signs (doctor tested),
    exercise, risk assessment, life expectancy, ethical review, dependency)

Example target:

     (treatment or surgery)

KUAKE-QIC This task asks one to classify the intent labels for an online
medical search query. There is a relatively high proportion of samples with
the the ”other” category. Thus, similar to the KUAKE-QIC task, we drop this
label and reformulate this task as generating the intent type of a given
search query with the candidate labels given in the prompt, or respond with
”Not the above type” to indicate that the given label types in the prompt does
not match the sentence.

Prompt Template:

     (Determine the intention of the following
    search term)
    [query]
    [candidate-types] (Choices: [candidate-types])
     (Answer: )

Target template:

    [selected-type]

Example prompt:

     (Determine the intention of the following search
    term)
     (Treatment methods for infectious genital
    warts in Wuhan)
     (Choices: Indicator interpretation, treatment plan, efficacy,
    precautions, condition diagnosis, medical advice, disease description)
     (Answer: )

Example target:

     (treatment plan)

CHIP-STS The aim of this task is to determine whether two disease-related
questions express the same semantics. Under PromptCBLUE, the task is to
respond 是 (_yes_), or 相同 (the same), or ”不是” (_no_), or 不同 (different) to
indicate whether the two input medical questions in the prompt have the same
semantics.

Prompt Template:

     (Are the following two sentences
    semantically the same or different?)
    [sentence-1][sentence-2]
    [candidate-types]
    (Answer: )

Target template:

    [selected-type]

Example prompt:

     (Are the following two sentences
    semantically the same or different?)
     ("What is diabetes’ three more and one less ", "Can the
    limitless" Ganoderma lucidum emperor "and" Sangtang drink "cure
    diabetes?".)
     (Option: the ame, different)
    (Answer: )

Example target:

     (different)

KUAKE-QTR Under CBLUE, this task asks one to determine to what degree the
online medical search query and the title of a web page are related
semantically by outputing a relevance score. The relevance score is one of the
followng four values: 0 (means ”完全不匹配”, _completely mismatched or without any
reference value_), 1 (means ”很少匹配有一些参考价值”, _slightly matched but having some
reference value_), 2 (means ”部分匹配”, _partially matched_) and 3 (means ”完全匹配”,
_perfectly matched_). Under PromptCBLUE, this task is formulated as outputing
the label names indicating to what degree a query and a page title are
related.

Prompt Template:

     (How do the following search terms
    and page tags have the same meaning?)
    [query] (Search term: [query])
    [web-page-title] (Page Title: [web-page-title])
     (Option: Completely mismatched or without reference value, rarely
    matched with some reference value, partially matched, completely
    matched)

Target template:

    [selected-type]

Example prompt:

     (How do the following search terms
    and page tags have the same meaning?)
     (Search term: The baby has been running
    a fever for three weeks, he doesn’t play and just sleep)
     (Page Title: The child has been running a fever in his hands, feet,
    and mouth for three weeks, but after a day, he stops burning and
    goes to bed)
     (Option: Completely mismatched or without reference value, rarely
    matched with some reference value, partially matched, completely
    matched)

Example target:

     (partially matched)

KUAKE-QQR This task is designated to differentiate the semantic relations
between two medical queries. And the semantic relation labels are: -1 (meaning
”语义无直接关联”, _not semantically related_), 0 (meaning ”后者是前者的语义父集”, _the latter
is the semantic superset of the former_), 1 (meaning ”后者是前者的语义子集”, _the latter
is the semantic subset of the former_), 2 (meaning ”完全一致”, _completely
equivalent_).

Prompt template:

     (What is the semantic relationship between the
    following two sentences?)
    [sentence-1][sentence-2]
    :  (Option: completely consistent, the latter is a semantic subset
    of the former, and the latter is a semantic superset of the former,
    with no semantic correlation)

Target template:

    [selected-type]

Example prompt:

     (What is the semantic relationship between
    the following two sentences?)
     (What kind of medicine is good
    for wounds quickly? What kind of medicine is good for wounds.)
    :  (Option: completely consistent, the latter is a semantic subset
    of the former, and the latter is a semantic superset of the former,
    with no semantic correlation)

Example target:

     (completely consistent)

KUAKE-IR This is originally a standard information retrieval (IR) task in the
medical domain. It asks one to retrieve relevant documents for a online
medical query among a corpus of 1000 thousand documents. Apparently, one can
not input all the documents in the corpus into LLMs. Thus, we reformulate this
task to the task of determine whether the given query and document are
relevant.

Prompt template:

    [query] (medical search: [query])
    [document] (content: [document])
    :  (Option: related, unrelated)
     (Answer: )

Target template:

    [selected-type]

Example prompt:

     (medical search: correction surgery for
    nasal bridge collision)
    1012000 (content: Hello, in this case, you usually need to undergo nasal
    bone osteotomy and nasal septum joint correction. The surgery requires
    hospitalization, which takes about 10 days and costs around 12000
    yuan. Our hospital does not provide medical insurance. If you want
    medical insurance reimbursement, you need to go through referral
    procedures and then return to your location with relevant documents
    for reimbursement.)
    :  (Option: related, unrelated)
     (Answer: )

Example target:

     (unrelated)

CHIP-MDCFNPC Given a dialogue history between a patient and a doctor, this
task asks one to extract the clinical finding entities and their statuses.
Here the status of a clinical finding entity is defined as one of the four
categories: ”阳性” (”已有症状疾病或者假设未来可能发生的疾病”, _existing symptoms/diseases or
potential future symptoms/diseases_), ”阴性” (meaning ”未患有的症状疾病”,
_symptoms/diseases not currently experienced_), 其他
(”没有回答、不知道、回答不明确或者模棱两可不好推断”, _no answer, don’t know, unclear answer, or
ambiguous response that is not conducive to inference._), 不标注
(”无实际意义的不标注或者和病人当前的状态独立不标注”, _irrelevant and not specified or unrelated to the
patient’s current condition and not specified_). Under PromptCBLUE, we ask
LLMs to output the descriptions of the status label for extracted clinical
finding entities.

Prompt template:

    [dialogue-history]
     (Question: What are the clinical findings in the above consultation
    dialogue? What are the statuses of these entities?)
     (Options: Existing symptomatic diseases or assumed future diseases,
    etc., without symptomatic diseases, without answers, unclear answers,
    or ambiguous reasoning, without practical significance, or independent
    of the patient’s current state)
     (Explanation: Clinical findings are a collection of concepts
    describing patient states in clinical medicine)

Target template:

    (The clinical finding entities detected in the above conversation
    and its status are as follows: )
    [clinical-finding-entity][status-option]

Example prompt:

     (Patient: Can I still take Wuji Baifeng
    Pills and Danzhi Xiaoyao Pills after menstruation)
     (Doctor: How long has similar symptoms
    occurred?)
     (Doctor: What is the purpose of taking this
    medicine for)
     (Patient: I went to check a few months ago for an increase in free
    radicals, and my menstrual cycle did not come)
     (Doctor: what is high?)
     (Patient: Increased thyroid free radical hormone)
     (Doctor: Show me the test report.)
     (Patient: Now we’re going to check the normal values)
     (Doctor: If you have a lot of menstrual flow, you can skip
    these medications. If you have a little menstrual flow, you can take them.)
     (Patient: It’s just menstrual irregularities)
     (Patient: Still taking thyroid hormone medication)
     (Doctor: Is it hypothyroidism? Is
    hypothyroidism present?)
     (Patient: I cannot understand the report
    form in Malaysia)
     (Doctor: Hmm, only hypothyroidism requires taking thyroid hormone.)
     (Question: What are the clinical findings in the above consultation
    dialogue? What are the statuses of these entities?)
     (Options: Existing symptomatic diseases or assumed future diseases,
    etc., without symptomatic diseases, without answers, unclear answers,
    or ambiguous reasoning, without practical significance, or independent
    of the patient’s current state)
     (Explanation: Clinical findings are a collection of concepts describing
    patient states in clinical medicine)

Example target:

    (The clinical finding entities detected in the above conversation
    and its status are as follows: )
     (Menstruation not coming: Existing symptomatic diseases or assumed
    future diseases, etc.)
     (Elevated free radical adenosine: Existing symptomatic diseases
    or assumed future diseases, etc.)
     (Increased thyroid free radical hormone: Existing symptomatic
    diseases or assumed future diseases, etc.)
     (Low menstrual flow: No labeling of meaningless or independent of
    the patient’s current state)
     (Menorrhagia: Low menstrual flow: No labeling of meaningless or
    independent of the patient’s current state)
     (Irregular menstruation: Existing symptomatic diseases or assumed
    future diseases, etc.)
     (Hypothyroidism: Not answering, not knowing, unclear or ambiguous
    in answer, making it difficult to infer)
     (Low menstrual flow: No labeling of meaningless or independent of
    the patient’s current state)

IMCS-V2-SR Under CBLUE, this task consists of multiple steps: (a) extract
symptom entities from a dialogue history between a patient and a doctor; (b)
normalize the symptom entity mentions to standardized terms (choosing from the
vocabulary provided by the task); (c) determine the status of a symptom
entity. In this task, the status label set is: ”患有该症状”
(_experiencing/suffering from the symptom_), ”没有患有该症状” (_not
experiencing/suffering from that symptom_), ”无法根据上下文确定病人是否患有该症状” (unable to
determine from the context whether the patient is experiencing that symptom).
Under PromptCBLUE, we discard the second step, that is, we only ask the LLMs
to extract the symptom entities in the current utterance and determine their
statuses.

Prompt template:

     (Identify the symptoms in the current
    conversation and determine their status:)
    (Dialogue history: )
    [dialogue-history]
    (Current utterance: )
    [current-utterance]
     (Options of symptom status: not experiencing/suffering from that
    symptom, experiencing/suffering from the symptom, unable to determine
    from the context whether the patient is experiencing that symptom)

Target template:

    (The symptoms in the current conversation
    and their statuses are: )
    [aymptom][status]

Example prompt:

     (Identify the symptoms in the current
    conversation and determine their status:)
     (Dialogue history: )
     (Patient: I didn’t listen very much)
     (Doctor: According to your description, the baby is coughing,
    roaring in the throat, possibly due to wheezing or wheezing in the
    throat, and the possibility of bronchitis is higher)
     (Current utterance: )
     (Doctor: It is necessary to take the baby to the pediatric
    department of the hospital for treatment, use a stethoscope to
    auscultate the lungs, conduct blood routine chest X-ray and other
    related tests, rule out pneumonia, and provide symptomatic treatment
    such as infection control, cough relief, and phlegm reduction
    based on the results.)
     (Options of symptom status: not experiencing/suffering from that
    symptom, experiencing/suffering from the symptom, unable to determine
    from the context whether the patient is experiencing that symptom)

Example target:

    (The symptoms in the current conversation
    and their statuses are: )
     (Pneumonia: unable to determine from the context whether the patient
    is experiencing that symptom)
     (Infection: experiencing/suffering from the symptom)
     (Cough: experiencing/suffering from the symptom)
     (Phlegm: experiencing/suffering from the symptom)

IMCS-V2-NER The reformulation of this task is similar to that of CMeEE-v2,
except that this task focuses on NER of dialogue utterances.

Prompt template:

    [entity-labels] (What are the [entity-labels]
    entities in the following conversation?)
    [utterance]
     (Answer: )

Target template:

     (The entities in the above sentence include: )
    [entity-mention][entity-type]
    [entity-mention][entity-type]

Example prompt:

     (What are the medical examinations, symptoms, and medical procedures
    in the following conversation?)
     (Does the baby also vomit?)
     (Answer: )

Example target:

     (The entities in the above sentence include: )
    :  (vomiting: Symptom entity)

With the idea of COT Wei2022ChainOT, the prompt will ask the LLMs to generate
entities type-by-type.

Prompt template with COT:

    [entity-labels] (What are the [entity-labels]
    entities in the following conversation?)
    [utterance]
     (Note: Please extract entities according to the entity type)
     (Answer: )

Target template with COT:

     (The entities in the above sentence include: )
    [entity-type][entity-mentions]

Example prompt with COT:

     (What are the medical examinations, symptoms, and medical procedures
    in the following conversation?)
     (Does the baby also vomit?)
     (Note: Please extract entities according to the entity type)
     (Answer: )

Example target with COT:

     (The entities in the above sentence include: )
     (Medical examination and testing entity: None)
     (Symptom entity: vomiting)
     (Medical operation entity: None)

IMCS-V2-DAC This task askes one to classify the intent of the current
utterance in a patient-doctor dialogue for online medical consultations. The
original task only gives label names that are not natural languages. To better
fit this task into LLMs, we re-write the intent label names.

Prompt template:

    : (Determine the intent label of this utterance: )
    [utterance]
    [candidate-labels] (Choices of intent labels: [candidate-labels])

Target template:

    [selected-label]

Example prompt:

    : (Determine the intent label of this utterance: )
     (At that time, the doctor said we
    prescribed medication for simple bronchitis without wheezing)
     (Choices of intent labels: Answers to medical advice, providing
    diagnosis, answering symptoms, asking questions about symptoms,
    asking questions about medical advice, answering questions about
    existing tests and treatments, asking questions about precautions,
    asking questions about existing tests and treatments, asking questions
    about personal basic information, answering questions about personal
    basic information, answering questions about medication advice,
    asking questions about causes, and asking questions about medication
    advice, Answers to precautions and etiology)

Example target:

     (answers regarding existing examinations and
    treatments)

IMCS-V2-MRG In this task, LLMs are asked to summarize a patient-doctor
dialogue and write a diagnostic and treatment report. In the dataset, all the
reports are divided into 6 sections: chief complaint, present illness history,
auxiliary examination, past medical history, diagnosis, recommendations. Thus,
under PromptCBLUE, we also ask the LLMs to generate the summarization report
in the order of these six sections.

Prompt template:

     (Consultation conversation history: )
    [dialogue-history]
     (Based on the above dialogue, a diagnosis and
    treatment report is given)
    , , , , ,  (Note: The diagnosis and treatment report is divided into six
    chapters: chief complaint, history of present illness, auxiliary
    examination, past history, diagnosis, and recommendations.)

Target template:

     (The diagnosis and treatment report of the
    above consultation dialogue is as follows:)
    [str] (Chief complaint: [str])
    [str] (History of current illness: [str])
    [str] (Auxiliary check: [str])
    [str] (Past history: [str])
    [str] (Diagnosis: [str])
    [str] (Suggestion: [str])

Example prompt:

     (Consultation conversation history: )
     (Patient: The baby is just one month old and breastfeeding. He
    has coughed from time to time in the past two days. His appetite
    and energy are okay, but his sleep is not very stable. In the past
    two days at home, the adults and the baby’s sister have also had colds.
    I don’t know if the baby has been infected with a cold or something
    else. How can I treat it?)
     (Doctor: Hello, I am your auxiliary doctor. I need to ask a few
    questions to better evaluate the child’s condition. Are you still here?)
     (Doctor: Is the babys temperature normal?)
     (Doctor: Are you still there?)
     (Patient: in)
     (Doctor: Hello)
     (Doctor: Is the babys temperature normal now?)
     (Patient: Temperature is normal)
     (Doctor: Are you spitting bubbles at the mouth?)
     (Patient: None)
     (Doctor: Do you have a hoarse voice?)
     (Patient: Crying is the same as before)
     (Doctor: OK)
     (Patient: Its just that the nasal sound is heavy)
     (Doctor: Are there any other symptoms?)
     (Patient: Unable to sleep well)
     (Doctor: Do you sweat a lot?)
     (Patients: not many)
     (Doctor: Are you crying?)
     (Patient: more crying than before)
     (Doctor: What does the stool look like?)
     (Patient: I didnt pay attention to this. He didnt have a bowel
    movement yesterday. He had a bowel movement this morning. He slept
    more peacefully after the bowel movement, but he still coughed from
    time to time.)
    ? (Doctor: Is there phlegm?)
     (Patient: There is no phlegm, and there is dry cough)
     (Doctor: Uh-huh)
     (Doctor: Do you have a runny nose?)
     (Patient: None)
     (Doctor: Uh-huh)
     (Doctor: Because the baby is relatively small, it is best to take
    the baby to the pediatrician in a public hospital. If there is no
    problem with the lungs, he can be observed.)
     (Based on the above dialogue, a diagnosis
    and treatment report is given)
    , , , , ,  (Note: The diagnosis and treatment report is divided into six
    chapters: chief complaint, history of present illness, auxiliary
    examination, past history, diagnosis, and recommendations.)

Example target:

     (The diagnosis and treatment report of the
    above consultation dialogue is as follows:)
     (Chief complaint:
    Paroxysmal cough.)
     (History of current illness: The child had
    paroxysmal dry cough for two days.)
     (Auxiliary check: None)
     (Past history: None)
     (Diagnosis: Cough to be examined.)
     (Suggestion: Pediatric visit, auscultation of
    the lungs.)

MedDG This task asks one to generate the responses of doctors give the
dialogue history and the patient’s current utterance. Under the original
setting, MedDG is an entity-centric dialogue generation task, that is, it asks
the model to generate responses that contains certain entities. Under
PromptCBLUE, we currently discard this setting and only asks the LLMs to
generate responses on behlf of the doctors.

Prompt template:

    [dialogue-history]
     (Based on the above conversation
    history, how should a doctor respond?)
     (Answer: )

Target template:

    [generated-response]

Example prompt:

    19 (Patient: Recently, I always feel sick in my stomach at
    night. I’ve been feeling a little nauseous after eating for the past
    few days, but I still have headaches, dizziness, and nausea. My
    stomach still feels uncomfortable (female, 19 years old))
     (Doctor: Hello, how long has this been
    going on?)
     (Patient: Its been half a month.)
     (Doctor: Do you eat regularly?)
     (Based on the above conversation
    history, how should a doctor respond?)
     (Answer: )

Example target:

     (Why does your stomach feel so bad? Acid reflux, heartburn and
    hiccups? Or stomachache or bloating?)

###  A.2 Hyper-parameter settings for the PEFT methods

We now elaborate on the hyper-parameter settings for different PEFT methods,
and the training settings.

PEFT Hyper-parameters For P-tuning, we prepend 128 soft prompt tokens to the
input. The learnable prompts are randomly initialized and use the LSTM prompt
encoder. For P-tuning-V2, each layer’s soft prompt is directly randomly
initialized without prompt encoder. The prompt length is also 128 for
P-tuning-V2. Adapter is added in parallel to the feed-forward and self-
attention module of the LLM, with a bottleneck dimension of 128. LoRA’s rank
is set to 24 and is added to the query, key, value, output weight matrix, and
the two weight matrix in the feed-forward module. AdaLoRA’s initial rank
setting is the same with LoRA.

Training settings Each PEFT method is trained with the same training hyper-
parameters: warm up steps are 100, batch size is 64, and the learning rate is
3e-4, the optimizer is AdamW, the learning schedule is linear decay, gradient
clipping is 1, weight decay is 1e-4, and the maximum epoch is 10. The other
settings follow the huggingface Transformers’ default settings. We find that
training a few steps after the dev loss hits the lowest point is helpful for
generating better responses. Thus we pick the checkpoint that is 200 steps
after the one with the lowest dev loss.

###  A.3 Example dialogue for Table LABEL:tab:error_mrg

The complete patient-doctor conversation for Table LABEL:tab:error_mrg is as
follows:

    701701

And its English translation is:

    Patient: Is there any problem with the platelet 701 of the four-month-old
    baby? The baby had a cough recently. I went to the hospital for a
    blood test. The white blood cells were over 10 and the platelets
    were 701. The doctor said it was probably a blood clot. The baby
    is a little anemic and a little calcium deficient, maybe because
    there is not a lot of milk. I caught a cold yesterday and I dont
    know if it was passed on to my baby. High platelets may also be a
    blood disease. I heard some doctors on the Internet say that this
    kind of situation will heal on its own without intervention? ?
    Please ask a professional doctor to answer. Is the babys problem
    serious? How should I do it?
    Doctor: Hello
    Patient: Hello
    Patient: Is there a problem with high platelets?
    Doctor: This child’s platelets are high, which should be related
    to infection
    Doctor: Viral infection or inflammation can easily lead to
    elevated platelets
    Doctor: This platelet value does not meet the diagnostic criteria
    for thrombocytosis. , generally speaking, it can be diagnosed as
    thrombocytosis if it exceeds 800, or even 900.
    Patient: Is it possible to have a blood clot?
    Doctor: High platelets will increase blood viscosity, and the
    probability of thrombosis is much higher than that of normal
    children, but this probability is relatively low
    Patient: Is it a big problem?
    Doctor: From the medical history you provided, my mother has a
    cold, so I should consider that she was infected by her
    Doctor: The possibility of blood disease is unlikely, and I should
    consider that infection is related to it.
    Doctor: Generally speaking, this platelet value does not require
    special treatment. As the child’s infection is controlled, the
    platelet value will gradually decrease
    Patient: What should I do?
    Doctor: Just treat the child for infection, because the total number
    of white blood cells in this child is high. After the infection is
    controlled, the blood routine will be rechecked
    Patient: Is it not a big problem?
    Doctor: Yes
    Patient: Well, thank you!
    Doctor: Hello
    Patient: Hello, Dr. Qu
    Doctor: You can eat Panax notoginseng, which has the effect of
    promoting blood circulation and removing blood stasis
    Patient: My child has diarrhea tonight. It seems that the
    inflammation is in the intestines and stomach.
    Doctor: Gastrointestinal problems should be considered
    Patient: But he is only four months old, can he take it?
    Doctor: Traditional Chinese medicine preparations for cough medicine
    may cause diarrhea in children
    Doctor: It is not recommended to take it orally because the child
    is too young
    Doctor: If it is taken orally, you can choose aspirin or dipyridamole.
    These two drugs have inhibitory effects. The role of platelet
    aggregation
    Patient: Oh, the child is a little anemic
    Doctor: What is the hemoglobin
    Patient: I havent reviewed it yet
    Doctor: Oh
    Patient: Well, thank you doctor!
    Doctor: The main problem with this child is the increase in platelets
    and white blood cells caused by infection.