# EHRNoteQA: A Patient-Specific Question Answering Benchmark
for Evaluating Large Language Models in Clinical Settings

Sunjun Kweon1∗, Jiyoun Kim1, Heeyoung Kwak2,3, Dongchul Cha2,4
Hangyul Yoon1, Kwanghyun Kim5, Seunghyun Won6, Edward Choi1
KAIST1 NAVER Digital Healthcare LAB2 Naver Cloud3
NAVER Healthcare LAB4, Ewha Womans University College of Medicine5
Seoul National University Bundang Hospital6
{sean0042, jiyoun.kim, hangyulmd<EMAIL_ADDRESS>
{heeyoung.kwak<EMAIL_ADDRESS>{khkim.uro<EMAIL_ADDRESS>Equal
contribution

###### Abstract

This study introduces EHRNoteQA, a novel patient-specific question answering
benchmark tailored for evaluating Large Language Models (LLMs) in clinical
environments. Based on MIMIC-IV Electronic Health Record (EHR) (Johnson et
al., 2023), a team of three medical professionals has curated the dataset
comprising 962 unique questions, each linked to a specific patient’s EHR
clinical notes. What makes EHRNoteQA distinct from existing EHR-based
benchmarks is as follows: Firstly, it is the first dataset to adopt a multi-
choice question answering format, a design choice that effectively evaluates
LLMs with reliable scores in the context of automatic evaluation, compared to
other formats. Secondly, it requires an analysis of multiple clinical notes to
answer a single question, reflecting the complex nature of real-world clinical
decision-making where clinicians review extensive records of patient
histories. Our comprehensive evaluation on various large language models
showed that their scores on EHRNoteQA correlate more closely with their
performance in addressing real-world medical questions evaluated by clinicians
than their scores from other LLM benchmarks. This underscores the significance
of EHRNoteQA in evaluating LLMs for medical applications and highlights its
crucial role in facilitating the integration of LLMs into healthcare systems.
The dataset will be made available to the public under PhysioNet credential
access, and the code will be accessible via GitHub
repository111https://github.com/ji-youn-kim/EHRNoteQA promoting further
research in this vital field.

EHRNoteQA: A Patient-Specific Question Answering Benchmark
for Evaluating Large Language Models in Clinical Settings

Sunjun Kweon1∗, Jiyoun Kim1††thanks:  Equal contribution, Heeyoung Kwak2,3,
Dongchul Cha2,4 Hangyul Yoon1, Kwanghyun Kim5, Seunghyun Won6, Edward Choi1
KAIST1 NAVER Digital Healthcare LAB2 Naver Cloud3 NAVER Healthcare LAB4, Ewha
Womans University College of Medicine5 Seoul National University Bundang
Hospital6 {sean0042, jiyoun.kim, hangyulmd<EMAIL_ADDRESS>{heeyoung.kwak<EMAIL_ADDRESS>{khkim.uro<EMAIL_ADDRESS>

##  1 Introduction

The advance of generative Large Language Models (LLMs), exemplified by the GPT
series (Brown et al., 2020; Ouyang et al., 2022; OpenAI, 2023) and open-source
models such as LLaMA (Touvron et al., 2023a, b), has significantly progressed
the field of LLM research. These models exhibit comprehensive world knowledge,
reasoning capabilities, and fluent language generation, paving the way for
their potential integration into healthcare (Cascella et al., 2023; Eysenbach
et al., 2023; Patel and Lam, 2023; Javaid et al., 2023). Still, there remains
a significant challenge in their application due to the lack of specialized
benchmarks for evaluating LLMs in clinical environments.

Figure 1: For each patient record, EHRNoteQA consists of all discharge
summaries across multiple admissions, a clinically relevant question
formulated to reflect clinician inquiries, multi-choice answer options, and
the correct answer.

Dataset | Questions | Documents | Patients | Source | Answer Type | Single/Multiple Documents
---|---|---|---|---|---|---
Raghavan et al. (2018) | 5,696 | 71 | 71 | Clinical Notes (Cleveland Clinic) | Text Span | Single
Pampari et al. (2018) | 73,111 | 303 | 303 | Discharge Summaries (n2c2) | Text Span | Single
Fan (2019) | 245 | 138 | 138 | Discharge Summaries (n2c2) | Text Span | Single
Yue et al. (2020) | 50 | - | - | Clinical Notes (MIMIC-III) | Text Span | Single
Yue et al. (2021) | 1,287 | 36 | 36 | Clinical Notes (MIMIC-III) | Text Span | Single
Soni et al. (2022) | 3,074 | 1,009 | 100 | Radiology Notes (MIMIC-III) | Text Span | Single
Lehman et al. (2022) | 2,029 | 114 | 114 | Discharge Summaries (MIMIC-III) | No Answer | Single
Moon et al. (2023) | 96,939 | 505 | 505 | Discharge Summaries (n2c2) | Text Span | Single
Fleming et al. (2023) | 983 | 37,264 | 276 | EHRs (Stanford University) | Free Text | Single
EHRNoteQA (ours) | 962 | 1,659 | 962 | Discharge Summaries (MIMIC-IV) | Multi Choice | Multiple

Table 1: Comparison of EHRNoteQA with existing patient-specific benchmarks on
EHRs. EHRNoteQA stands out as the first dataset to implement a multi-choice
question answering format, enabling reliable and accurate automatic
evaluation. It is also unique in explicitly demanding the use of two or more
clinical notes for answering a question.

Existing clinical benchmarks for LLMs (Jin et al., 2019; Hendrycks et al.,
2020; Jin et al., 2021; Pal et al., 2022) primarily focus on general medical
questions derived from either medical exams or biomedical articles. These
benchmarks, while effective in assessing LLMs’ clinical reasoning
capabilities, often fall short in capturing the complexities inherent in
individual patient cases. Our study aims to fill this gap by building a
patient-specific LLM benchmark, to address queries based on patients’
Electronic Health Records (EHRs).

In this study, we present EHRNoteQA, the first dataset for patient-specific,
multi-choice question answering dataset derived from EHR clinical notes. This
dataset, leveraging the publicly available MIMIC-IV EHR database (Johnson et
al., 2023), was initially created using GPT-4 and thoroughly reviewed and
refined by a team of three clinicians to ensure both accuracy and clinical
relevance. EHRNoteQA includes 962 unique questions, each linked to patient-
specific clinical notes, specifically discharge summaries, that occurred
during multiple admissions. The dataset demands the examination of multiple
notes to formulate accurate answers (refer to Figure 1), mirroring the complex
nature of real-world medical decision-making where clinicians gather
information across a patient’s accumulated hospitalization record.

Using EHRNoteQA, we conduct experiments with diverse large language models to
show its effectiveness as an EHR-based benchmark for LLMs. We start by
examining the rationale behind adopting a multiple-choice format for question
answering, as opposed to the free-text format which is more common in clinical
environments. In practice, when clinicians use LLMs in medical settings, they
are unlikely to present the model with predefined answer options as done in a
multiple-choice format. Despite this deviation from practical usage, our
experiment results indicated that the multi-choice format yields a more
consistent and reliable outcome for automatic evaluation when compared to the
free-text approach. Furthermore, we conduct additional tests to see if our
dataset can reflect how clinicians evaluate LLMs in actual clinical scenarios.
To do this, three clinicians evaluated and ranked the models using a
dataset222Note that we utilized DiSCQ (Lehman et al., 2022), which only
contains questions from actual doctors, but not the answers. Therefore it
cannot be used as a proper LLM benchmark unless evaluated directly by the
clinicians. that is comprised of open-ended clinical questions sourced from
real clinical environments. We then compared these rankings with those derived
from existing benchmarks and EHRNoteQA. Our results revealed a notable
alignment between the model rankings from EHRNoteQA, and those from the
clinician-led evaluation using real-world clinical questions. In conclusion,
among benchmarks currently available for evaluating LLMs, our dataset stands
out with its ability to provide a reliable evaluation that mirrors closely the
actual assessments of clinicians.

##  2 Related Works

Figure 2: An overview of the construction process for the EHRNoteQA dataset,
which involves three key stages: 1) Sampling Clinical Notes from MIMIC-IV
database, 2) Data Generation using GPT-4, and 3) Modifications by Clinicians.

While there have been several initiatives towards creating patient-specific
question answering datasets using Electronic Health Records (EHRs), each of
these works comes with its own set of limitations. For instance, Raghavan et
al. (2018) has first built a question answering dataset annotated by medical
students based on EMR, but it is not open to public. Pampari et al. (2018)
generated a clinical note reading comprehension dataset using the i2b2 dataset
(Uzuner et al., 2008; Uzuner, 2009; Uzuner et al., 2010, 2011), with questions
based on pre-defined templates related to medical entities. However, this
approach limits the diversity of questions. Fan (2019) generated why-QAs on
clinical notes by identifying sentences with ‘because’ or ‘due to’, then
splitting these sentences into questions and answers, but this method’s
limitation is that both question and answer are derived from the same
sentence, failing to mirror the complexity of real-world clinical inquiries.

Subsequently, Yue et al. (2020) introduced a QA dataset using MIMIC-III
(Johnson et al., 2016) clinical notes, but the size is small and not publicly
available. Yue et al. (2021) constructed multiple QA pairs built upon MIMIC-
III clinical notes, but around 75% were generated by a pre-defined text span
as the answer and creating a question based on it, leading to contextually
limited questions. Soni et al. (2022) proposed a physician-annotated QA
dataset on radiology reports with the answer existing as a text span. Lehman
et al. (2022) created a trigger-based question dataset from MIMIC-III
discharge summaries, but without its annotated answers. Moon et al. (2023)
developed a dataset focusing on drug-reasoning questions with answers
comprising multiple clinical entities, but were constrained by template-based
generation. Lastly, Fleming et al. (2023) proposed a patient-specific EHR QA
dataset using instructions from clinicians, but the answers were in free-text
format, lacking an accurate and reliable evaluation method without clinician
evaluation. Moreover, 80% of their data consisted of context lengths longer
than 32k tokens, exceeding the capacity of current LLMs.

As specified in Table 1, these studies predominantly use textual spans from
clinical notes as answers and evaluated models through F1 and Exact Match
scores. While this method is suitable for extractive models such as BERT
(Devlin et al., 2019), it is less effective for generative large language
models that provide more detailed and complex responses (Kamalloo et al.,
2023). Furthermore, confining answers to text span restricts the ability to
craft in-depth questions necessary in real medical settings. Such settings
often require collecting information from numerous clinical segments within
and across documents. Even in cases where the answer exists as multiple text
spans, this does not fulfill the need in real-world applications for a
complete, single response. Other datasets face their own challenges: either
lacking annotated answers (Lehman et al., 2022) or providing answers in free
text (Fleming et al., 2023), which complicates evaluation due to subjectivity
and the need for human review, often involving clinicians. This poses a
significant challenge, as consistent clinician-led evaluation is both costly
and time-consuming.

In our work, we propose a patient-specific EHR QA dataset on MIMIC-IV
discharge summaries (Johnson et al., 2023), inspected by clinicians and
reflecting real-world medical scenarios. Our dataset is unique in requiring
references to two or more clinical notes to answer a single question. Moreover
by employing a multi-choice format, our dataset serves as a clinical benchmark
that enables accurate and consistent automatic evaluation of LLMs.

##  3 Data Construction

In this section, we describe the construction of the EHRNoteQA dataset, which
consists of three main phases: Document Sampling (Section 3.1), Question-
Answer Generation (Section 3.2), and Clinician Modification (Section 3.3).
Figure 2 shows an overview of our EHRNoteQA construction process.

###  3.1 Document Sampling

For the construction of EHRNoteQA, we utilized clinical notes sourced from the
MIMIC-IV (Medical Information Mart for Intensive Care IV) EHR database
(Johnson et al., 2023), a rich source of real patient records from Beth Israel
Hospital spanning from 2008 to 2019. Specifically, we formulated our data
using discharge summaries, which are detailed records prepared when a patient
is discharged from the hospital. These summaries are crucial for clinical note
question answering, as they encapsulate the extensive information generated
from a patient’s admission to discharge.

Category | MIMIC-IV | Final (Sampled)
---|---|---
Level | # D.S. | # Patients | # Tokens | # Patients | # Tokens
1 | 1 | 38,926 | 1,819 | 264 (275) | 1,787
2 | 437 | 2,147 | 265 (275) | 2,146
2 | 1 | 44,645 | 3,514 | 145 (150) | 3,501
2 | 14,176 | 4,470 | 144 (150) | 4,581
3 | 1,161 | 4,956 | 144 (150) | 5,030
Total | 99,345 | - | 962 (1000) | -

Table 2:  Quantitative analysis of patient counts and average token length per
admission in Level 1 and Level 2 of the MIMIC-IV dataset and EHRNoteQA. Values
in parentheses denote the initially sampled patient counts. D.S. indicates
Discharge Summaries.

The MIMIC-IV database encompasses 331,794 discharge summaries for 145,915
unique patients, with an average of 2.3 notes per patient. However, these
summaries are typically lengthy, with the average length of all discharge
summaries for a patient being around 8k tokens. This presents a challenge for
current LLMs, as only a limited number of them can process contexts that
exceed 8,000 tokens, making it difficult to handle such extensive clinical
notes.

To address this challenge while incorporating lengthy notes into our data, we
initially reduced the overall length of the notes without altering their
content or structure. By minimizing excessive white spaces, such as removing
spaces or tabs around newlines, we achieved an average reduction of 10% in
note length. Subsequently, we categorized patients into two levels based on
the length of their clinical notes, ensuring compatibility with the processing
capabilities of existing LLMs. The first level (Level 1) consists of patients
whose cumulative note length in the database does not exceed 3,500 tokens,
aligning with models designed to process up to 4k tokens. The second level
(Level 2) is for patients whose notes consist of 3,500 tokens to 7,500 tokens,
suitable for models that can handle up to 8k tokens.

As shown in Table 2, the first category encompasses instances involving one to
two hospital admissions, as indicated by the number of discharge summaries,
whereas the second category can accommodate cases of one to three admissions.
The remaining cases, which require models capable of handling extremely long
context lengths, are not covered in this study. For the construction of the
EHRNoteQA dataset, we randomly selected 1,000 patients—550 from Level 1 and
450 from Level 2—and prepared their discharge summaries for the next step.

###  3.2 Question-Answer Generation

Based on the documents selected as described in Section 3.1, we aimed to
construct a question-answering dataset that closely emulates the types of
questions a clinician would ask in a real-world clinical context, while also
operating as a reliable performance indicator. To achieve this, we employed
GPT-4 OpenAI (2023)333To conduct experiments with MIMIC-IV data alongside
online API-based language models such as GPT, we carried out our work on
Azure’s HIPAA-compliant platform, in accordance with the regulations set by
PhysioNet. to generate a dataset for multi-choice question answering,
utilizing the clinical notes from each patient. This involved designing five
answer choices for each question: one correct answer and four distractors. To
ensure the generated data met our objectives, we closely collaborated with
clinicians during the prompt tuning phase.

Category | Example | Percentage
---|---|---
Treatment |  | What was the treatment provided for the patient’s left breast cellulitis during her second admission?
---
19%
Assessment | What was the change in the patient’s left knee condition from the first hospital admission to the second one? | 20%
Problem |  | What was the patient’s chief complaints and major surgical procedures carried out during his admissions in 2167 and 2176?
---
22%
Etiology | What is the most probable cause of the patient’s diarrhea during her admission? | 10%
Sign/Symptom |  | What were the main presenting symptoms during the patient’s first recorded hospital visit?
---
5%
Vitals | What was the range of the patient’s blood pressure during her second hospital stay? | 9%
Others | Was the patient’s pregnancy full-term during her second recorded admission, and why was a cesarean section required? | 5%

Table 3: Distribution of question categories across 100 sampled questions from
EHRNotQA, including examples.

The question-answer generation process unfolded in two primary steps.
Initially, we provided GPT-4 with the patients’ discharge summaries to
generate clinically meaningful questions that can be answered within the given
text. This step was crucial not only for generating clinically relevant
questions but also for creating questions with clear answers. In the
subsequent step, we re-inputted the discharge summaries along with the
formulated questions into GPT-4, in order to extract a set of answer choices
for each question: one correct and four incorrect choices, along with the
correct answer index.

The decision to employ a two-step approach, instead of a more straightforward
single-step process, was a deliberate decision made through iterative
refinement of question and answer generation prompts by incorporating
extensive feedback from medical professionals. Collaborative insights from
these clinicians underscored that this two-step methodology not only yielded
questions and answer choices that were more realistic and accurate but also
ensured that the incorrect options were sophisticated enough to avoid being
trivially dismissible. The sample data generated using this approach with
GPT-4 is illustrated in Figure 2. To support reproducibility, we have
disclosed the specific model used, its associated costs, and the exact prompts
in the Appendix A.

###  3.3 Clinician Modification

Despite incorporating feedback from clinicians during the prompt tuning phase
of GPT-4’s question-answer generation, the machine-generated data still
exhibited certain imperfections. These included questions that were trivial or
uncharacteristic of those typically asked by real clinicians, as well as
incorrect or irrelevant answers, and overly simplistic incorrect answer
choices. To address these issues, a refinement phase was conducted involving
three doctors over a period of two months, reviewing all 1,000 QA data points
generated in Section 3.2. The modifications consist of:

Data Removal: Instances where GPT-4 produced questions that are too trivial or
unrepresentative of typical clinical inquiries were identified and removed by
the doctors. A total of 38 data points were removed from the dataset of 1,000
entries.

Question Revision: Questions that were clinically meaningful but unclear,
overly detailed, or not in the typical format of clinicians’ inquiries were
directly revised. In total, 206 out of the 1,000 questions were modified by
the doctors.

Correct Answer Choice Revision: In cases where the generated correct answers
were found to be incorrect and unclear, revisions were made to ensure the
answers were completely correct, leaving no aspect of the question unanswered.
A total of 338 out of 1,000 correct answers were modified.

Increasing the Difficulty of Incorrect Answer Choices: If the incorrect answer
choices included content not present in the clinical notes, they were revised
to be plausible distractors, instead of being straightforwardly incorrect
options. Out of the 4,000 wrong answers (1,000 questions with 4 wrong answers
each), 962 were revised.

Our final dataset, refined by clinicians, now consists of 962 unique
questions, each linked to a specific patient. The final patient counts
corresponding to each level are indicated in Table 2.

We conduct further analysis of our dataset, by categorizing the question
content types according to the question classification scheme proposed by
Lehman et al. (2022). This categorization process was carried out on a subset
of 100 questions and conducted by the authors. The distribution of question
content types, as presented in Table 3, shows that our dataset encompasses a
comprehensive representation of data within each category.

##  4 Experiments

Size Model Level 1 (multi-choice) Level 2 (multi-choice) Level 1 (free-text)
Foundation Reference GPT4-turbo-preview (1106) 95.384  $\pm 0.104$ 94.368
$\pm 0.126$ 86.990  $\pm 0.740$ - OpenAI (2023) GPT4 (0613) 97.124  $\pm
0.080$ 95.104  $\pm 0.192$ 91.060  $\pm 0.800$ - OpenAI (2023)
GPT3.5-turbo-16k (0613) 88.280  $\pm 0.233$ 84.990  $\pm 0.000$ 80.980  $\pm
1.110$ - Brown et al. (2020) 70B Llama-2-70b-chat-hf 84.652  $\pm 0.282$ -
71.080  $\pm 0.670$ LLama-2-70b Touvron et al. (2023b) qCammel-70-x 85.630
$\pm 0.134$ - 72.820  $\pm 1.270$ LLama-2-70b Toma et al. (2023) Camel-
Platypus2-70B 89.790  $\pm 0.233$ - 76.030  $\pm 1.330$ LLama-2-70b Lee et al.
(2023a) Platypus2-70B-instruct 90.322  $\pm 0.159$ - 78.940  $\pm 1.820$
LLama-2-70b Lee et al. (2023a) 30B MPT-30b-instruct 79.660  $\pm 0.250$ 75.382
$\pm 0.206$ 58.100  $\pm 1.160$ MPT-30b-8k MosaicML (2023) 13B
Llama-2-13b-chat-hf 73.196  $\pm 0.309$ - 62.270  $\pm 1.460$ LLama-2-13b
Touvron et al. (2023b) vicuna-13b-v1.5 82.116  $\pm 0.217$ - 64.840  $\pm
1.050$ LLama-2-13b Chiang et al. (2023) WizardLM-13B-V1.2 80.758  $\pm 0.159$
- 64.800  $\pm 1.570$ LLama-2-13b Xu et al. (2023) qCammel-13 71.106  $\pm
0.596$ - 54.330  $\pm 0.910$ LLama-2-13b Toma et al. (2023) OpenOrca-
Platypus2-13B 85.896  $\pm 0.288$ - 72.020  $\pm 1.420$ LLama-2-13b Lee et al.
(2023b) Camel-Platypus2-13B 77.958  $\pm 0.510$ - 58.830  $\pm 1.060$
LLama-2-13b Lee et al. (2023a) Synthia-13B-v1.2 79.284  $\pm 0.213$ - 71.270
$\pm 1.240$ LLama-2-13b Tissera (2023a) 7B Llama-2-7b-chat-hf 65.672  $\pm
0.365$ - 50.700  $\pm 1.230$ LLama-2-7b Touvron et al. (2023b) vicuna-7b-v1.5
78.222  $\pm 0.510$ - 50.440  $\pm 1.580$ LLama-2-7b Chiang et al. (2023)
Mistral-7B-Instruct-v0.1 81.926  $\pm 0.170$ 65.038  $\pm 0.126$ 66.120  $\pm
0.820$ Mistral-7B-v0.1 Jiang et al. (2023) MPT-7b-8k-instruct 59.512  $\pm
0.159$ 51.362  $\pm 0.385$ 43.670  $\pm 1.380$ MPT-7B-8k MosaicML (2023)
dolphin-2.0-mistral-7b 76.218  $\pm 0.159$ - 66.840  $\pm 0.900$
Mistral-7B-v0.1 Cognitive (2023) Mistral-7B-OpenOrca 87.074  $\pm 0.104$ -
78.940  $\pm 1.720$ Mistral-7B-v0.1 Lian et al. (2023) SynthIA-7B-v1.3 78.412
$\pm 0.085$ - 73.530  $\pm 1.300$ Mistral-7B-v0.1 Tissera (2023b)

Table 4: Average scores and their standard deviations of Level 1 multi-choice,
Level 2 multi-choice scores, and Level 1 free-text. Evaluation for each model
output is conducted five times on the same prompt.

In our experiments using EHRNoteQA, we conducted a comprehensive evaluation
across a wide array of large language models, spanning from GPT series to
various instruction-tuned open-source models. We selected 22 models for this
purpose, all of which are equipped to manage a context length of 4,000 tokens,
allowing us to evaluate Level 1 data. Furthermore, within this group, 6 models
possess the capability to handle contexts up to 8,000 tokens, which enabled
our analysis of Level 2 data.

To evaluate these models on our dataset, we executed model generation followed
by subsequent assessment of the generated outputs. Traditionally, a common
approach for evaluating multi-choice questions on large language models
involves using probability-based scoring, where each model output is evaluated
based on the probability of the answer choice letter or the full answer
sequence (Brown et al., 2020; Hendrycks et al., 2020; Liang et al., 2022; Gao
et al., 2023). This approach requires providing few-shot examples to guide the
model toward generating outputs in the desired format, in order to ensure
accurate evaluation. However, the challenge arises from the limited context
length capacities of existing LLMs, which cannot accommodate multiple
patients’ discharge summaries. As a result, using a probability-based scoring
system for our dataset with a few-shot approach is infeasible.

Therefore, instead of relying on probability-based metrics, we generated each
model output by inputting discharge summaries, questions, and answer choices.
We then assessed the correctness of the model responses by providing these
outputs along with the gold answer to GPT-4-turbo, instructing it to evaluate
them. For each model, correct outputs were assigned of 1 point, while
incorrect responses were given 0 points. Across 962 questions, the scores of
each model were normalized to a 100-point scale for comparison. Despite our
efforts to eliminate stochastic factors that might affect GPT-4-turbo’s
evaluation (e.g., setting the temperature to 0), there were cases where the
same model output was inconsistently evaluated as correct or incorrect,
particularly in cases of ambiguous responses. Given the impracticality of
manually examining each instance, we had GPT-4-turbo evaluate the same output
five times, and averaged the results.

The results obtained from the evaluation of 22 distinct models utilizing this
approach is presented in Table 4. This table presents the average scores from
five repeated evaluations conducted with GPT-4-turbo and includes the standard
deviation for each score. It’s important to note that for Level 2 data, the
evaluation focused on a subset of 6 models specifically designed to support
context lengths of 8,000 tokens.

One of the key observation from the table is the performance variation within
models of the same size (e.g. Within the 7B size category, scores range from
59 to 87). These disparities can be attributed to the underlying foundation
model (such as LLaMA-2 (Touvron et al., 2023b), Mistral (Jiang et al., 2023),
or MPT (MosaicML, 2023)) or the instruction-tuning dataset that was employed
to enhance the foundation model. This highlights the significant role of the
choice of foundation models and their fine-tuned instructions in determining
performance outcomes on our dataset.

In order to facilitate future studies using our dataset for evaluating LLMs
for clinical applications, we provide GPT-4-turbo prompt used for evaluation
in Appendix A.3. In the following sections, we further conduct more detailed
analyses using our dataset. In Section 4.1, we evaluate our dataset in both
multi-choice and free-text formats to compare which format yields more
consistent and reliable outcomes in automated assessments. Section 4.2
examines the influence of the length and quantity of clinical notes on model
performance. Section 4.3 evaluates the correlation between model performances
on our dataset and physician evaluations of model responses to real-world
clinical questions, assessing our dataset’s fidelity to real-world clinical
evaluations.

###  4.1 Multi-Choice V.S. Free-Text

When engaging with LLMs in practice, clinicians typically would not present
predefined answer options to LLMs, as in multi-choice question answering.
However, we experimentally demonstrate the effectiveness of the multi-choice
format compared to the free-text format when using EHRNoteQA for automatic LLM
evaluation.

In addition to solving EHRNoteQA in a multiple-choice manner, we further
assessed our dataset in a free-text format. Given the absence of a reliable
automatic metric for free-text response assessment, many studies delegate this
evaluation to GPT Zhou et al. (2023); Sottana et al. (2023). Consequently, we
also employed GPT for the scoring of free-text responses in our dataset.
Similar to scoring multi-choice, we provided GPT-4-turbo with both the model’s
output and the correct answer for evaluating the free-text format.

The evaluation was conducted on our Level 1 data, and the results are
summarized in Table 4’s Level 1 (free-text) column. An important observation
comes from the standard deviations: while the scores’ average standard
deviation across 22 models was 0.24 when scoring with multiple-choice format,
the average standard deviation increased to 1.21 when assessing free-text
responses. This suggests that when automatically evaluating the same model,
the scores can vary significantly in free-text format. Additionally, we
computed the models’ rankings over five iterations, represented as $r_{j}^{i}$
(where $j$ denotes each model and $i$ each iteration), and identified the most
frequently occurring rankings (mode) as the gold rankings, $\hat{r_{j}}$, for
each model. A comparison of these gold rankings against those from each of the
five iterations, calculated using the formula
$\sum_{j=1}^{22}\sum_{i=1}^{5}|r_{i}-\hat{r}|$, revealed 12 instances of
ranking differences in multi-choice evaluations versus 29 in free-text
evaluations. This indicates that rankings for models can vary significantly
when scoring free-text responses, suggesting that this method might not serve
as a reliable measure for evaluating models. The specific results of each of
the five scores and rankings can be found in the Appendix B.

This experiment leads us to conclude that while free-text-based evaluation may
more closely mimic real clinical environments, using a multiple-choice format
in assessing our dataset offers a more consistent and reliable scoring system,
making it an essential choice for the automatic evaluation of LLMs.

Figure 3: Model scores on EHRNoteQA with differing number of clinical notes
for each level. For model scores in which the number of notes range from 1 to
2 denotes performance of models on Level 1 data. For model scores in which the
number of notes range from 1 to 3 denotes performance of models on Level 2
data.

Clinician Measure EHRNoteQA MedQA PubMedQA MMLU* MedMCQA ARC HellaSwag MMLU
TruthfulQA Winogrande GSM8K AVG A Spearman 0.74 0.50 0.07 0.65 0.51 0.52 0.25
0.57 0.65 0.38 0.26 0.60 Kendall 0.58 0.35 0.06 0.50 0.38 0.37 0.18 0.41 0.54
0.28 0.17 0.42 B Spearman 0.81 0.68 0.16 0.80 0.73 0.58 0.36 0.65 0.75 0.47
0.24 0.62 Kendall 0.65 0.52 0.09 0.62 0.58 0.45 0.25 0.50 0.59 0.33 0.16 0.47
C Spearman 0.77 0.59 0.12 0.68 0.67 0.53 0.28 0.58 0.65 0.44 0.20 0.58 Kendall
0.66 0.45 0.10 0.54 0.51 0.42 0.21 0.44 0.48 0.31 0.16 0.43

Table 5: Correlations between model scores from various benchmarks and
physician-evaluated model performance on real-world clinical questions, with
bold indicating the highest correlation and underlined the second highest.

###  4.2 Impact of Note Length and the Number of Notes on the Model
Performances

In this section, we conduct a comparative analysis of model performance based
on the length and the number of clinical notes in the EHRNoteQA dataset. As
outlined in Section 3.1, Level 1 data includes patient notes up to 3.5k
tokens, while Level 2 data comprises notes ranging from 3.5k to 7.5k tokens,
indicating that Level 2 involves longer note lengths. The results depicted in
Table 4 show that models consistently achieve lower scores on Level 2 data
compared to Level 1 data. This discrepancy highlights the increased complexity
associated with Level 2, due to the requirement to comprehend and interpret
more extensive clinical contexts presented in longer patient notes. Moreover,
we examine the impact of the number of notes on model performance. Level 1
data consists of clinical notes from 1 to 2 admissions, and Level 2 data
encompasses notes from 1 to 3 admissions. Figure 3 illustrates how model
scores change as the number of notes increases for both Level 1 and Level 2.
The results consistently indicate a diminishing trend in model performance as
the number of notes per question increases. This trend underscores the
challenge posed by our multi-note dataset in assessing a model’s ability to
collect and understand the cumulative content of patient discharge summaries
across multiple hospital admissions.

###  4.3 Representativeness of EHRNoteQA for Actual Clinical Assessments

In our final analysis, our aim is to assess how accurately our dataset
reflects the assessments made by physicians in real clinical settings. To
achieve this, we conducted an analysis measuring the correlation between 19
models’ scores 444Among the 22 models we tested, the 3 models from the GPT
series were overwhelmingly superior in performance, making their ranking
obvious. Therefore, we excluded them from the experiment measuring
correlation. obtained from our dataset, and the physician-assessed scores on
model responses to real-world medical questions. For a comprehensive
evaluation, we expanded our comparison to include four other widely utilized
clinical benchmarks and six general domain benchmarks, and measured the
correlation between these benchmark scores and the assessments made by
physicians as well.

Recently, Lehman et al. (2022) introduced a dataset called DiSCQ, which is
composed of questions written by medical experts under a patient handoff
scenario, utilizing MIMIC-III discharge summaries (Johnson et al., 2016). This
dataset contains 1,089 questions that stem from 114 discharge summaries. We
leveraged DiSCQ as a source of real-world questions for measuring the
correlation, because it represents patient-specific medical inquiries, without
any overlap with our dataset or other benchmark datasets. However, DiSCQ
dataset lacks predetermined correct answers, and thus it is not possible to
conduct automated evaluation. To overcome this limitation, the three
clinicians evaluated the free-form responses of the models to these questions.
From the total 1,089 DiSCQ questions, we randomly selected a sample of 300
questions, assigning 100 questions to each of the three clinicians. Each
clinician was asked to assess a total of 1,900 responses from 19 models for
his/her set of 100 questions, where the order of the model responses for each
question was shuffled to prevent any bias. With these results, we calculated
the correlation between clinician-evaluated DiSCQ scores and scores obtained
from our dataset and various benchmarks.

The correlations are provided in Table 5. Individual scores given by each
clinician and scores of the evaluated benchmarks, along with their references,
are listed in the Appendix C. Notably, the correlation with our dataset
consistently surpassed that of other benchmarks. This observation indicates
that the model scores evaluated using our dataset are more likely to reflect
the model performance on real-world medical questions compared to when using
other benchmark datasets for LLM evaluation.

##  5 Conclusion

In this paper, we introduce EHRNoteQA, a novel benchmark curated by clinicians
for assessing LLMs within the clinical domain. Unlike other patient-specific
datasets, EHRNoteQA uniquely employs a multi-choice question format that
facilitates more reliable automated evaluations of LLMs. Our dataset also
encompasses clinical notes from multiple admissions for each patient, thereby
capturing the complex nature of real-world clinical settings. Through our
research, we found that, although the multiple-choice format deviates from
practical medical situations, adopting this format is a crucial choice for
ensuring reliability. Furthermore, we address the potential limitations of the
format by demonstrating that the correlation between model scores assessed by
physicians in real clinical settings and those obtained from our dataset is
higher than with other datasets. By making EHRNoteQA available to the wider
research community, we aim to facilitate LLM integration into healthcare
services, thereby enhancing clinicians’ decision making and patient care.

## Limitations

Our study primarily focuses on the analysis of discharge summaries, excluding
other clinical documentations or data formats. Hospital EHR databases contain
not only discharge summaries but also other types of unstructured clinical
notes, structured data, and non-textual elements such as images and signals.
Although our research is the first in creating a dataset of discharge
summaries across multiple admissions, there is a need for benchmarks that
encompass a broader spectrum of clinical notes and data types.

Another constraint of our work pertains to the composition of our dataset,
which only includes questions that are answerable. In real-world clinical
settings, questions posed by healthcare professionals to an EHR system can
include both answerable or unanswerable questions. Assessing a model’s
capability to determine if a question is answerable or not is crucial for its
application in real-world scenarios. However, exploring this aspect of
question-answering falls into a distinct research topic, extending beyond the
scope of our current research.

Last limitation arises in measuring the correlation between model evaluations
using EHRNoteQA and the scores from DiSCQ dataset assessed by clinicians.
Firstly, the limited number of available models poses challenges for
statistically robust correlation measurement. Nonetheless, given the scarcity
of models supporting a context window of over 4k, this was the most feasible
option. Secondly, each clinician only evaluated 100 questions when assessing
DiSCQ questions. Due to the necessity for clinicians to review responses from
all models, employing a larger question set was not practical. Consequently,
we made efforts to ensure diverse evaluations by assigning each clinician a
unique set of 100 non-overlapping questions. While expanding the scale of our
analysis to include a wider array of models and a larger question set per
clinician would be desirable, such an expansion presents challenges requiring
a substantial increase in the number of clinicians involved. Thus, addressing
these challenges comprehensively remains a topic for future work.

## References

  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cascella et al. (2023) Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. 2023.  Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios.  _Journal of Medical Systems_ , 47(1):33.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Cognitive (2023) Cognitive. 2023.  Dolphin-2.0-mistral-7b.  https://huggingface.co/cognitivecomputations/dolphin-2.0-mistral-7b,.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eysenbach et al. (2023) Gunther Eysenbach et al. 2023.  The role of chatgpt, generative language models, and artificial intelligence in medical education: a conversation with chatgpt and a call for papers.  _JMIR Medical Education_ , 9(1):e46885.
  * Fan (2019) Jungwei Fan. 2019.  Annotating and characterizing clinical sentences with explicit why-qa cues.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 101–106.
  * Fleming et al. (2023) Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo P Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. 2023.  Medalign: A clinician-generated dataset for instruction following with electronic medical records.  _arXiv preprint arXiv:2308.14089_.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.  A framework for few-shot language model evaluation.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Javaid et al. (2023) Mohd Javaid, Abid Haleem, and Ravi Pratap Singh. 2023.  Chatgpt for healthcare services: An emerging stage for an innovative perspective.  _BenchCouncil Transactions on Benchmarks, Standards and Evaluations_ , 3(1):100105.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  _Applied Sciences_ , 11(14):6421.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.  Pubmedqa: A dataset for biomedical research question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2567–2577.
  * Johnson et al. (2023) Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. 2023.  Mimic-iv, a freely accessible electronic health record dataset.  _Scientific data_ , 10(1):1.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9.
  * Kamalloo et al. (2023) Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023.  Evaluating open-domain question answering in the era of large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5591–5606, Toronto, Canada. Association for Computational Linguistics.
  * Lee et al. (2023a) Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. 2023a.  Platypus: Quick, cheap, and powerful refinement of llms.  _arXiv preprint arXiv:2308.07317_.
  * Lee et al. (2023b) Ariel N. Lee, Cole J. Hunter, Nataniel Ruiz, Bleys Goodson, Wing Lian, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023b.  Openorcaplatypus: Llama2-13b model instruct-tuned on filtered openorcav1 gpt-4 dataset and merged with divergent stem and logic dataset model.  https://huggingface.co/Open-Orca/OpenOrca-Platypus2-13B,.
  * Lehman et al. (2022) Eric Lehman, Vladislav Lialin, Katelyn Edelwina Legaspi, Anne Janelle Sy, Patricia Therese Pile, Nicole Rose Alberto, Richard Raymund Ragasa, Corinna Victoria Puyat, Marianne Katharina Taliño, Isabelle Rose Alberto, et al. 2022.  Learning to ask like a physician.  In _Proceedings of the 4th Clinical Natural Language Processing Workshop_ , pages 74–86.
  * Lian et al. (2023) Wing Lian, Bleys Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023.  Mistralorca: Mistral-7b model instruct-tuned on filtered openorcav1 gpt-4 dataset.  https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252.
  * Moon et al. (2023) Sungrim Moon, Huan He, Heling Jia, Hongfang Liu, Jungwei Wilfred Fan, et al. 2023.  Extractive clinical question-answering with multianswer and multifocus questions: Data set development and evaluation study.  _JMIR AI_ , 2(1):e41818.
  * MosaicML (2023) MosaicML. 2023.  Introducing mpt-7b: A new standard for open-source, commercially usable llms.  Accessed: 2023-05-05.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pal et al. (2022) Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022.  Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.  In _Conference on Health, Inference, and Learning_ , pages 248–260. PMLR.
  * Pampari et al. (2018) Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018.  emrqa: A large corpus for question answering on electronic medical records.  _arXiv preprint arXiv:1809.00732_.
  * Patel and Lam (2023) Sajan B Patel and Kyle Lam. 2023.  Chatgpt: the future of discharge summaries?  _The Lancet Digital Health_ , 5(3):e107–e108.
  * Raghavan et al. (2018) Preethi Raghavan, Siddharth Patwardhan, Jennifer J Liang, and Murthy V Devarakonda. 2018.  Annotating electronic medical records for question answering.  _arXiv preprint arXiv:1805.06816_.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106.
  * Soni et al. (2022) Sarvesh Soni, Meghana Gudala, Atieh Pajouhi, and Kirk Roberts. 2022.  Radqa: A question answering dataset to improve comprehension of radiology reports.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6250–6259.
  * Sottana et al. (2023) Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023.  Evaluation metrics in the era of gpt-4: reliably evaluating large language models on sequence to sequence tasks.  _arXiv preprint arXiv:2310.13800_.
  * Tissera (2023a) Migel Tissera. 2023a.  Synthia-13b-v1.2b: Synthetic intelligent agent.  https://huggingface.co/migtissera/Synthia-13B.
  * Tissera (2023b) Migel Tissera. 2023b.  Synthia-7b-v1.3: Synthetic intelligent agent.  https://huggingface.co/migtissera/Synthia-13B.
  * Toma et al. (2023) Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry B Rubin, and Bo Wang. 2023.  Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding.  _arXiv preprint arXiv:2305.12031_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Uzuner (2009) Özlem Uzuner. 2009.  Recognizing obesity and comorbidities in sparse data.  _Journal of the American Medical Informatics Association_ , 16(4):561–570.
  * Uzuner et al. (2008) Özlem Uzuner, Ira Goldstein, Yuan Luo, and Isaac Kohane. 2008.  Identifying patient smoking status from medical discharge records.  _Journal of the American Medical Informatics Association_ , 15(1):14–24.
  * Uzuner et al. (2010) Özlem Uzuner, Imre Solti, and Eithon Cadag. 2010.  Extracting medication information from clinical text.  _Journal of the American Medical Informatics Association_ , 17(5):514–518.
  * Uzuner et al. (2011) Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011.  2010 i2b2/va challenge on concepts, assertions, and relations in clinical text.  _Journal of the American Medical Informatics Association_ , 18(5):552–556.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_.
  * Yue et al. (2020) Xiang Yue, Bernal Jimenez Gutierrez, and Huan Sun. 2020.  Clinical reading comprehension: A thorough analysis of the emrQA dataset.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4474–4486, Online. Association for Computational Linguistics.
  * Yue et al. (2021) Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon Lin, and Huan Sun. 2021.  Cliniqg4qa: Generating diverse questions for domain adaptation of clinical question answering.  In _2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pages 580–587. IEEE.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_.

##  Appendix A Prompts

When generating the EHRNoteQA data, the version of the GPT-4 model used was
gpt-4 (0613), and this was carried out on Azure’s HIPAA-compliant
platform555https://learn.microsoft.com/en-
us/azure/compliance/offerings/offering-hipaa-us. The total cost for creating
the data was $3,000. In terms of hyperparameters during data generation, we
set the temperature to 1 and left the rest at their default settings. When
evaluating the models using our dataset, we utilized GPT-4 from OpenAI API,
and the model version was gpt-4 (1106-preview). Scoring the entire EHRNoteQA
of 962 items once cost $4.5 per model. During model scoring, we adjusted the
temperature setting to 0, while keeping the other settings as default.
Appendices A.1 and A.2 are the prompts used for generating the data in step1
and step2, respectively. Appendix A.3 is the prompt used for evaluating the
model’s output on our dataset.

###  A.1 Question Generation (Step1)

Situation : When a patient is admitted to the hospital, important clinical
records are summarized in a ’discharge summary’. On the patient’s subsequent
visit, the previous ’discharge summaries’ serve as essential reference for the
doctor’s clinical decision making. Objective : Please formulate one question
that a doctor might actually ask based on the provided ’discharge summaries’.
The questions should have clear answer, and these answer should be found
within the provided ’discharge summaries’. Note : 1\. The ’discharge summary’
is provided between [note 1 start] and [note 1 end]. If there are multiple
notes, they are labeled as [note 1 start], [note 2 start], etc. 2\. The
’discharge summaries’ are provided in chronological order. This means note1 is
a record from before note2. At the beginning of each note, there is an
admission ID and the date it was written, so please refer to that. 3\. Please
refrain from formulating questions that can be answered without referring to a
note. 4\. Do not create question that is too easy to answer. To answer your
question, someone should have the clinical expertise equivalent to a doctor
and must fully understand all provided discharge summaries. 5\. Your answer
should also contain short rationale behind the answer. 6\. When explaining the
answer and its rationale, utilize the chart date of the note. In other words,
instead of saying the first note or the second note, phrase it as ’as per the
note charted on [specific date], …’. 7\. Arrange your output in the following
format: \- Question : [Your Question] \- Answer : [Your Answer] \- Reason :
[Explanation for your answer]

###  A.2 Answer Generation (Step2)

Objective : Please generate a multiple-choice question answering data with
five possible answers (A-E) based on the doctor’s question derived from the
provided ’discharge summary’. Ensure that one answer is correct, and the
remaining four are incorrect answers. Note : 1\. Use the provided doctor’s
question as the basis for the multiple-choice question without modification.
2\. The ’discharge summary’ is enclosed within [note 1 start] and [note 1
end], with additional notes being similarly labeled. 3\. All distractors
(incorrect answer choices) should contain contents that appear in the provided
discharge summary but should be wrong answer to the question. 4\. After
choosing all five choices, paraphrase them so that all choices are consistent
with the format and length. Ensure that longest length answer choice is not an
answer. 5\. The correct answer should be clearly indicated, and the rationale
should explain why this is the answer and why the other options are not
correct but are good distractors. 6\. Arrange your output in the following
format: \- Question: [The doctor’s question] \- Answer Choices: A: [First
option] B: [Second option] C: [Third option] D: [Fourth option] E: [Fifth
option] \- Correct Answer: [The letter of the correct choice] \- Reason:
[Explanation behind your answer and why each other options are incorrect but
can be good distractors]

###  A.3 Evaluation

Your task is to evaluate the provided model output by determining whether it
matches the correct answer from the multiple-choice options provided. The
model output is correct and should be met with a "yes" if it accurately
reflects the content of the correct answer choice, not necessarily its exact
wording. If the content of the model output aligns with the correct answer
choice, despite any additional details or varied phrasing, you are to respond
"yes". Should the model output diverge in meaning or substance from the
correct answer—whether by selecting an alternative choice or providing a
response not aligning with any provided options—a response of "no" is
necessary. Model Output: {output} Answer Choices: {choices} Correct Answer:
{answer} With the given information, do you conclude that the model output
substantively matches the correct answer provided? Respond solely with "yes"
or "no"

##  Appendix B Model Evaluation on EHRNoteQA

Model Multi-Choice Free-Text Take1 Take2 Take3 Take4 Take5 Mode Take1 Take2
Take3 Take4 Take5 Mode GPT4 turbo peview(1106-preview) 95.27 (2) 95.46 (2)
95.27 (2) 95.46 (2) 95.46 (2) 2 87.71 (2) 87.52 (2) 86.20 (2) 86.20 (2) 87.33
(2) 2 GPT4 (0613) 97.16 (1) 97.16 (1) 97.16 (1) 96.98 (1) 97.16 (1) 1 91.02
(1) 92.44 (1) 90.74 (1) 90.55 (1) 90.55 (1) 1 GPT35-turbo-16k (0613) 87.90 (5)
88.28 (5) 88.28 (5) 88.47 (5) 88.47 (5) 5 80.91 (3) 82.42 (3) 80.72 (3) 79.40
(3) 81.47 (3) 3 Llama-2-70b-chat-hf 84.50 (9) 84.69 (9) 85.07 (9) 84.69 (9)
84.31 (9) 9 71.46 (9) 72.02 (11) 70.70 (11) 70.89 (11) 70.32 (10) 11
qCammel-70-x 85.63 (8) 85.82 (8) 85.63 (8) 85.63 (8) 85.44 (8) 8 74.67 (7)
73.35 (9) 71.64 (8) 72.78 (8) 71.64 (8) 8 Camel-Platypus2-70B 89.41 (4) 89.79
(4) 89.98 (4) 89.79 (4) 89.98 (4) 4 76.37 (6) 78.07 (6) 75.61 (6) 75.61 (6)
74.48 (6) 6 Platypus2-70B-instruct 90.36 (3) 90.17 (3) 90.36 (3) 90.17 (3)
90.55 (3) 3 78.26 (5) 82.04 (4) 77.88 (4) 79.02 (4) 77.50 (4) 4
mpt-30b-instruct 79.96 (13) 79.77 (13) 79.40 (14) 79.40 (13) 79.77 (13) 13
56.33 (18) 56.90 (18) 56.71 (18) 54.06 (19) 56.52 (18) 18 Llama-2-13b-chat-hf
73.35 (19) 72.78 (19) 72.97 (19) 73.35 (19) 73.53 (19) 19 62.38 (16) 64.46
(16) 60.68 (16) 61.25 (16) 62.57 (16) 16 vicuna-13b-v1.5 82.42 (10) 82.04 (10)
82.04 (10) 81.85 (11) 82.23 (10) 10 64.27 (15) 65.78 (14) 63.33 (15) 65.78
(14) 65.03 (14) 14 WizardLM-13B-V1.2 80.72 (12) 80.72 (12) 80.53 (12) 80.91
(12) 80.91 (12) 12 67.11 (13) 64.84 (15) 64.27 (14) 62.76 (15) 65.03 (14) 15
qCammel-13 71.27 (20) 71.27 (20) 71.64 (20) 70.08 (20) 71.27 (20) 20 54.82
(19) 55.01 (19) 53.88 (19) 55.01 (18) 52.93 (19) 19 OpenOrca-Platypus2-13B
85.82 (7) 86.39 (7) 85.82 (7) 85.82 (7) 85.63 (7) 7 71.27 (10) 74.10 (8) 71.46
(9) 72.78 (8) 70.51 (9) 8 Camel-Platypus2-13B 78.26 (16) 77.32 (17) 77.88 (17)
77.69 (17) 78.64 (15) 17 58.41 (17) 60.68 (17) 58.60 (17) 58.03 (17) 58.41
(17) 17 Synthia-13B-v1.2 79.40 (14) 79.21 (14) 79.58 (13) 79.02 (14) 79.21
(14) 14 70.32 (11) 73.35 (9) 71.27 (10) 71.08 (10) 70.32 (10) 10
Llama-2-7b-chat-hf 65.41 (21) 65.97 (21) 66.16 (21) 65.41 (21) 65.41 (21) 21
52.36 (20) 49.53 (21) 50.28 (21) 51.61 (20) 49.72 (20) 20 vicuna-7b-v1.5 78.26
(16) 77.69 (16) 79.02 (15) 78.26 (16) 77.88 (17) 16 51.98 (21) 51.61 (20)
50.66 (20) 49.91 (21) 48.02 (21) 21 Mistral-7B-Instruct-v0.1 81.66 (11) 82.04
(10) 82.04 (10) 82.04 (10) 81.85 (11) 10 66.73 (14) 66.73 (13) 64.84 (13)
66.54 (12) 65.78 (13) 13 mpt-7b-8k-instruct 59.36 (22) 59.55 (22) 59.74 (22)
59.55 (22) 59.36 (22) 22 42.91 (22) 44.99 (22) 43.29 (22) 45.18 (22) 41.97
(22) 22 dolphin-2.0-mistral-7b 76.18 (18) 76.37 (18) 76.18 (18) 76.37 (18)
75.99 (18) 18 67.30 (12) 67.67 (12) 65.41 (12) 66.54 (12) 67.30 (12) 12
Mistral-7B-OpenOrca 87.15 (6) 87.15 (6) 86.96 (6) 87.15 (6) 86.96 (6) 6 80.72
(4) 80.53 (5) 77.50 (5) 79.02 (4) 76.94 (5) 5 SynthIA-7B-v1.3 78.45 (15) 78.26
(15) 78.45 (16) 78.45 (15) 78.45 (16) 15 73.53 (8) 75.61 (7) 72.02 (7) 73.35
(7) 73.16 (7) 7

Table 6:  Evaluation results of EHRNoteQA in Multi-Choice format and Free-Text
format on 22 Large Language Models. The score is measured 5 times each, and
the parentheses to the right of the score indicate the rank at that time. The
most frequently occurring rank in the scoring was considered the final rank.

##  Appendix C Model Evaluation on Benchmarks

As described in Section 4.3, our experiment involved three clinicians
assessing 19 models using the DiSCQ questionnaire (Lehman et al., 2022). The
outcomes of their evaluations are presented in Column Clinician A, B, and C of
Table 6. To establish a point of reference for these results, we also
evaluated the same models using 11 benchmarks, including EHRNoteQA. The
medical benchmarks such as MedQA (Jin et al., 2021), PubMedQA (Jin et al.,
2019), MMLU* (Hendrycks et al., 2020) (subset of MMLU whose topics are related
to medical. That subset includes Anatomy, Clinical Knowledge, College Biology,
College Medicine, Medical Genetics), and MedMCQA (Pal et al., 2022) were
assessed using the same method applied to EHRNoteQA. For other benchmarks like
ARC (Clark et al., 2018), Hellaswag (Zellers et al., 2019), MMLU, TruthfulQA
(Lin et al., 2022), Winogrande (Sakaguchi et al., 2021), and GSM8K (Cobbe et
al., 2021), we sourced the scores from the Hugging Face Open Source Large
Language Model leaderboard (Beeching et al., 2023). The ’AVG’ column
represents the average score across these benchmarks listed in the
leaderboard.

Model Clinician A Clinician B Clinician C EHRNoteQA MedQA PubMedQA MMLU*
MedMCQA ARC HellaSwag MMLU TruthfulQA Winogrande GSM8K AVG Llama-2-70b-chat-hf
53 48 37 84.65 44.38 67.8 58.7 37.84 64.59 85.88 63.91 52.8 80.51 26.69 62.40
qCammel-70-x 45 60 52 85.63 55.77 62.6 66.77 43.61 68.34 87.87 70.18 57.47
84.29 29.72 66.31 Camel-Platypus2-70B 62 69 68 89.79 60.02 64 71.7 49.8 71.08
87.6 70.04 58.09 83.82 22.9 65.59 Platypus2-70B-instruct 75 88 77 90.32 56.17
64.2 73.79 50.59 71.84 87.94 70.48 62.26 82.72 40.56 69.30 mpt-30b-instruct 28
41 22 79.66 41.01 68 50.31 39.06 58.45 84.31 49.15 38.05 75.14 15.31 53.40
Llama-2-13b-chat-hf 57 52 45 73.20 36.61 58.8 49.16 32.92 59.04 81.94 54.64
44.12 74.51 15.24 54.92 vicuna-13b-v1.5 60 61 56 82.12 43.21 66.8 57.86 40.21
57.08 81.24 56.67 51.51 74.66 11.3 55.41 WizardLM-13B-V1.2 58 65 57 80.76
39.75 57.2 51.99 33.99 59.04 82.21 54.64 47.27 71.9 13.5 54.76 qCammel-13 31
45 40 71.11 41.48 55.8 51.68 33.66 60.84 83.66 56.73 47.54 76.16 11.37 56.05
OpenOrca-Platypus2-13B 69 76 65 85.90 44.3 60.6 59.12 43.1 62.8 83.15 59.39
53.08 76.24 9.02 57.28 Camel-Platypus2-13B 40 57 49 77.96 46.66 61 59.12 39.3
60.75 83.61 56.51 49.6 75.37 0.08 54.32 Synthia-13B-v1.2 51 52 53 79.28 40.61
54.8 51.47 38.32 61.26 82.93 56.47 47.27 76.48 10.99 55.90 Llama-2-7b-chat-hf
38 33 17 65.67 35.43 58.6 44.23 31.8 52.9 78.55 48.32 45.57 71.74 7.35 50.74
vicuna-7b-v1.5 42 54 52 78.22 38.65 63.4 49.37 35.29 53.24 77.39 51.04 50.34
72.14 8.19 52.06 Mistral-7B-Instruct-v0.1 55 59 55 81.93 41.16 51 55.45 38.99
54.52 75.63 55.38 56.28 73.72 14.25 54.96 mpt-7b-8k-instruct 28 20 23 59.51
33.62 59.1 38.99 35.67 45.9 74.47 41.97 35.21 65.98 20.7 47.37
dolphin-2.0-mistral-7b 55 57 51 76.22 43.99 52.4 56.5 38.58 59.22 80.26 56.9
61.09 75.37 18.65 58.58 Mistral-7B-OpenOrca 61 66 67 87.07 47.29 62 60.06
40.23 64.08 83.99 62.24 53.05 77.74 19.94 60.17 SynthIA-7B-v1.3 51 58 55 78.41
47.13 69 53.77 42.67 62.12 83.45 62.65 51.37 78.85 17.59 59.34

Table 7:  Clinicians and Benchmarks evaluation results on 19 Large Language
Models.


11institutetext: Masahito Hayashi22institutetext: Shenzhen Institute for
Quantum Science and Engineering, Southern University of Science and
Technology, Shenzhen,518055, China, International Quantum Academy (SIQA),
Futian District, Shenzhen 518048, China, Graduate School of Mathematics,
Nagoya University, Nagoya, 464-8602, Japan. 22email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Special functions in quantum phase estimation

Masahito Hayashi

###### Abstract

This paper explains existing results for the application of special functions
to phase estimation, which is a fundamental topic in quantum information. We
focus on two special functions. One is prolate spheroidal wave function, which
approximately gives the maximum probability that the difference between the
true parameter and the estimate is smaller than a certain threshold. The other
is Mathieu function, which exactly gives the optimum estimation under the
energy constraint. It also characterizes the uncertainty relation for the
position and the momentum for periodic functions.

##  1 Introduction

It is well known that quantum system has group symmetry. Therefore, various
quantum information processing can utilize group symmetry to enhance or
optimize various operations. One typical example is the estimation of the
unknown unitary operation. In this problem setting, the set of possible
unitary operations often forms a group representation. When the input state is
fixed to a certain state, this problem can be considered as a special case of
the estimation of the unknown state under the group symmetric model. For this
type of state estimation, Holevo formulated a systematic group symmetric
approach Holevo ; Holevo2 . Holevo’s approach is known as a powerful tool for
state estimation H98 ; Group2 . By using Holevo’s approach, the above
estimation problem of the unknown unitary operation has been formulated in a
general form by CDS ; CMP .

The simplest case of the estimation of the unknown unitary operation is phase
estimation, which is formulated as optimizations of estimating methods of an
unknown element of $\mathop{\rm U}(1)$. A group symmetric approach works well
for this problem. Interestingly, although this problem can be formulated
dependently of the choices of error function and available input systems or
input states, the optimal solution under several special cases can be
characterized by special functions. This paper surveys existing results for
these relations between special functions and the optimal solution under
several examples of phase estimation. In particular, this paper focuses on two
special functions, prolate spheroidal wave function and Mathieu function.

Prolate spheroidal wave function approximately gives an optimal input state to
maximize the probability that the difference between the true parameter and
the estimate is smaller than a certain threshold. Mathieu function gives an
optimal input state under a certain energy constraint. This characterization
can be used for the uncertainty relation between the position and the momentum
on the periodic function space. In this way, these two special functions play
a central role in phase estimation.

The remaining of this paper is organized as follows. Section 2 gives the
formulation of phase estimation. Section 3 discusses the phase estimation
under more specific examples. This section presents the relation between
Prolate spheroidal wave function and phase estimation. Section 4 addresses the
phase estimation under the energy constraint. This section presents the
relation between Mathieu function and phase estimation. Section 5 applies the
result in Section 4 to the uncertainty relation between the position and the
momentum on the periodic function space.

[scale=.45]fig.pdf

Figure 1: This figure expresses the process to estimate the action of the
unknown element of $\mathop{\rm U}(1)$.

##  2 Formulation

We estimate the unknown application of an element of $\mathop{\rm U}(1)$ in
various settings. To cover various settings, this problem is formulated as
follows. First, we consider a fixed unitary representation $\mathsf{f}$ of the
group $\mathop{\rm U}(1)$ on a Hilbert space $\mathcal{H}$, which represents
our physical system. We are allowed to choose the input state $\rho$ and the
quantum measurement on the system $\mathcal{H}$ to get our estimate in
$\mathop{\rm U}(1)$. The quantum measurement on the system $\mathcal{H}$ is
rewritten as a positive operator-valued measure on $\mathcal{H}$, which is
given as ${\cal M}:=(M_{\theta})_{\theta\in[0,2\pi)}$ with the condition

$\displaystyle\int_{0}^{2\pi}M_{\theta}d\theta=I$ (1)

by identifying $\mathop{\rm U}(1)$ with $[0,2\pi)$. Our estimation scheme for
the unknown application $\mathsf{f}(e^{i\theta})$ with
$e^{i\theta}\in\mathop{\rm U}(1)$ is formulated as Fig. 1 CMP ; LP ; BDM ; PLA
; IH09 .

When the true unitary action is $\mathsf{f}(e^{i\theta})$, the output
$\hat{\theta}\in[0,2\pi)$ is generated by the distribution $\mathop{\rm
Tr}\mathsf{f}(e^{i\theta})\rho\mathsf{f}(e^{i\theta})^{\dagger}M_{\hat{\theta}}d\hat{\theta}$.
To evaluate the precision of our estimate, we consider the error function
$R(\theta,\hat{\theta})$. For the symmetry of our problem setting, we impose
the symmetric condition

$\displaystyle
R(\theta,\hat{\theta})=R(0,\hat{\theta}-\theta)=R(0,\hat{\theta}-\theta+2n\pi)$
(2)

with any integer $n$. Then, the average error is calculated as a function of
$\theta,\rho,{\cal M}$ CMP ;

$\displaystyle{\cal R}[\mathsf{f},R,\theta,\rho,{\cal
M}]:=\int_{0}^{2\pi}R(\theta,\hat{\theta})\mathop{\rm
Tr}\mathsf{f}(e^{i\theta})\rho\mathsf{f}(e^{i\theta})^{\dagger}M_{\hat{\theta}}d\hat{\theta}.$
(3)

It is natural to focus on the worst value ${\cal
R}_{\max}[\mathsf{f},R,\rho,{\cal M}]:=\max_{\theta}{\cal
R}[\mathsf{f},R,\theta,\rho,{\cal M}]$ or the average value ${\cal
R}_{av}[\mathsf{f},R,\rho,{\cal M}]:=\frac{1}{2\pi}\int_{0}^{2\pi}{\cal
R}[\mathsf{f},R,\theta,\rho,{\cal M}]d\theta$ with respect to the unknown
parameter $\theta$ CMP . We consider the following minimizations CMP

$\displaystyle{\cal R}_{\max}[\mathsf{f},R]:=\min_{\rho,{\cal M}}{\cal
R}_{\max}[\mathsf{f},R,\rho,{\cal M}],\quad{\cal
R}_{av}[\mathsf{f},R]:=\min_{\rho,{\cal M}}{\cal
R}_{av}[\mathsf{f},R,\rho,{\cal M}].$ (4)

[scale=.45]fig2.pdf

Figure 2: This figure expresses the process to estimate the action of the
unknown element of $\mathop{\rm U}(1)$ with the reference system. The input
system might be an entangled state between the system $\mathcal{H}$ and the
reference system $\mathcal{H}_{R}$.

To discuss the above problems, we consider a detailed structure. An
irreducible representation of $\mathop{\rm U}(1)$ is characterized by an
integer $n\in\mathbb{Z}$ and has a one-dimensional representation space
$\mathcal{H}_{n}$. This representation is denoted as $\mathsf{f}_{n}$ and is
defined as $\mathsf{f}_{n}(e^{i\theta})=e^{in\theta}$.

Now, we consider a general representation $\mathsf{f}$ of $\mathop{\rm U}(1)$
and its representation space $\mathcal{H}$. Let $S$ be the set of indexes $n$
whose corresponding irreducible representation $\mathsf{f}_{n}$ is contained
in $\mathsf{f}$. We denote the multiplicity of $\mathsf{f}_{n}$ in
$\mathsf{f}$ by $m_{n}$, and define an $m_{n}$-dimensional space by
$\mathcal{V}_{n}$. Then, the representation space $\mathcal{H}$ is written as
$\oplus_{n\in S}\mathcal{H}_{n}\otimes\mathcal{V}_{n}$, where the group
$\mathop{\rm U}(1)$ acts only on $\mathcal{H}_{n}$. That is, for
$x=\oplus_{n\in S}x_{n}\otimes v_{n}\in\oplus_{n\in
S}\mathcal{H}_{n}\otimes\mathcal{V}_{n}$, we have

$\displaystyle\mathsf{f}(g)x=\bigoplus_{n\in S}(\mathsf{f}_{n}(g)x_{n})\otimes
v_{n}$ (5)

for $g\in\mathop{\rm U}(1)$.

This formulation contains the case when the input state is an entangled state
between the system $\mathcal{H}$ and a reference system $\mathcal{H}_{R}$ as
Fig. 2 because the joint system $\mathcal{H}\otimes\mathcal{H}_{R}$ has the
form $\oplus_{n\in S}\mathcal{H}_{n}\otimes\mathcal{V}_{n}$.

When the multiplicity $m_{n}$ is one for any $n\in S$, the representation
$\mathsf{f}$ is called multiplicity-free with $S$ and is denoted by
$\mathsf{f}_{S}$. Under the representation $\mathsf{f}_{S}$, we denote a
normalized vector in $\mathcal{H}_{n}$ by $e_{n}$. The representation space of
the representation $\mathsf{f}_{S}$ is the space $\mathcal{H}_{S}$ spanned by
the orthogonal vectors $\\{e_{n}\\}_{n\in S}$. Under the representation
$\mathsf{f}_{S}$, we consider the following types of positive operator-valued
measure. Consider a vector $|w\rangle:=\sum_{n\in S}|e_{n}\rangle$. We choose
$M_{\hat{\theta}}:=\frac{1}{2\pi}\mathsf{f}_{S}(e^{i\hat{\theta}})^{\dagger}|w\rangle\langle
w|\mathsf{f}_{S}(e^{i\hat{\theta}})$, which satisfies the condition (1) for
POVM. This POVM is written as ${\cal M}_{w}$. Also, an element $|\phi\rangle$
of the vector space $\mathcal{H}_{n}$ can be identified with $(\phi_{n})_{n\in
S}$ through the relation $|\phi\rangle=\sum_{n\in S}\phi_{n}|e_{n}\rangle$. We
define the Fourier transform ${\cal F}[\phi](\hat{\theta})$ as $\sum_{n\in
S}\phi_{n}e^{in\hat{\theta}}=\langle
w|\mathsf{f}_{S}(e^{i\hat{\theta}})|\phi\rangle$. Then, as shown in (CMP, ,
Lemma 1 and Theorem 1) CDS , we have

$\displaystyle{\cal R}_{\max}[\mathsf{f},R]=$ $\displaystyle{\cal
R}_{av}[\mathsf{f},R]=\min_{|\phi\rangle\in\mathcal{H}_{S}}{\cal
R}[\mathsf{f}_{S},R,0,|\phi\rangle\langle\phi|,{\cal M}_{w}]$ $\displaystyle=$
$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{S}}\frac{1}{2\pi}\int_{0}^{2\pi}R(0,\hat{\theta})\langle
w|\mathsf{f}_{S}(e^{i\hat{\theta}})|\phi\rangle\langle\phi|\mathsf{f}_{S}(e^{i\hat{\theta}})^{\dagger}|w\rangle
d\hat{\theta}$ $\displaystyle=$
$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{S}}\frac{1}{2\pi}\int_{0}^{2\pi}R(0,\hat{\theta})|{\cal
F}[\phi](\hat{\theta})|^{2}d\hat{\theta}.$ (6)

##  3 Constraint for available irreducible representation

In this section, we consider several examples where available irreducible
representation is restricted. We assume that $R$ is given as
$R_{\sin}(\theta,\hat{\theta}):=2\sin^{2}\frac{\theta-\hat{\theta}}{2}=1-\cos(\theta-\hat{\theta})$.
We consider a typical representation $\mathsf{f}_{\\{0,1\\}}$. We often
consider its $n$-fold tensor product representation
$\mathsf{f}_{\\{0,1\\}}^{\otimes n}$. In this representation, the set of
indexes $S$ is $\\{0,1,\ldots,n\\}$. Hence, it is sufficient to address
$\mathsf{f}_{\\{0,1,\ldots,n\\}}$. Then, the minimization (6) is calculated as

$\displaystyle{\cal R}_{\max}[\mathsf{f}_{\\{0,1\\}}^{\otimes n},R_{\sin}]=$
$\displaystyle{\cal R}_{av}[\mathsf{f}_{\\{0,1\\}}^{\otimes n},R_{\sin}]={\cal
R}_{av}[\mathsf{f}_{\\{0,1,\ldots,n\\}},R_{\sin}]$ $\displaystyle=$
$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{\\{0,1,\ldots,n\\}}}\frac{1}{2\pi}\int_{0}^{2\pi}2\sin^{2}\frac{\hat{\theta}}{2}|{\cal
F}[\phi](\hat{\theta})|^{2}d\hat{\theta}$ $\displaystyle=$
$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{\\{0,1,\ldots,n\\}}}1-\frac{1}{2}\sum_{j=0}^{n-1}(\overline{\phi}_{j}\phi_{j+1}+\phi_{j}\overline{\phi}_{j+1}).$
(7)

For the derivation of the final step, see Holevo ; Holevo2  BDM , (PLA, ,
Section 2) (CMP, , Theorem 7).

In fact, the maximum eigenvalue of the operator
$\frac{1}{2}\sum_{j=0}^{n}(|e_{j}\rangle\langle
e_{j+1}|+|e_{j+1}\rangle\langle e_{j}|)$ is $\cos\frac{\pi}{n+1}$, and its
corresponding eigenvector is
$C\sum_{j=0}^{n}\sin\frac{j\pi}{n+1}|e_{j}\rangle$ with a normalizing constant
$C$ (CMP, , Theorem 7). Hence, the above minimum is

$\displaystyle 1-\cos\frac{\pi}{n+1}=2\sin^{2}\frac{\pi}{2(n+1)},$ (8)

which asymptotically behaves as $\frac{\pi^{2}}{2n^{2}}$. This type of
analysis was extended to the case with the group SU(2) BBM ; CDPS2 ; PLA . In
this case, the error is inverse proportional to $n^{2}$. This scaling is
called Heisenberg scaling.

###### Remark 1

Here, it is better to remark that many papers discussed Heisenberg scaling in
a misleading way GLM ; GLM2 ; NOOST ; OHNOST ; JKFABBM . The above discussion
calculated the minimum error. To discuss the asymptotic behavior of the
minimum error, instead of the above calculation, these papers employ the
relation between the estimation error and Fisher information. The estimation
error is lower bounded by the inverse of Fisher information. The attainability
of this lower bound is not trivial in general. For example, In the case of
state estimation, this lower bound can be attained by a two-step method under
a natural regularity condition HM . However, in the case of unitary
estimation, this lower bound cannot be attained. In particular, the lower
bound given by the maximum Fisher information is strictly smaller than the
optimal minimum estimation error even in the level of the first order
coefficient CMP2 . These papers considered that the maximum Fisher information
gives the estimation error even in this case while Fisher information approach
does not work for the Heisenberg scaling of the estimation error in phase
estimation.

Next, we discuss the asymptotic behavior in another way (IH09, , Section 4).
For simple analysis, we focus on the representation
$\mathsf{f}_{\\{-N,\ldots,N\\}}$ instead of $\mathsf{f}_{\\{0,1,\ldots,n\\}}$.
We consider the function space $L^{2}([-1,1])$ and its dense subset
$L_{c}^{2}([-1,1]):=L^{2}([-1,1])\cap C([-1,1])$, where $L^{2}([-1,1])$ is the
set of square integrable functions on $[-1,1]$ and $C([-1,1])$ is the set of
continuous functions on $[-1,1]$. Given a normalized continuous function
$\psi\in L_{c}^{2}([-1,1])$, we choose
$\phi^{(n)}\in\mathcal{H}_{\\{-N,\ldots,N\\}}$ as the normalized vector of
$(\psi(\frac{j}{N}))_{j=-N}^{N}$. We define the Fourier transform ${\cal F}$
on $L^{2}(\mathbb{R})$ as

$\displaystyle{\cal
F}[\psi](t):=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}e^{itx}\psi(x)dx.$
(9)

Then, using $t=N\hat{\theta}$, we have

$\displaystyle\frac{N^{2}}{2\pi}\int_{0}^{2\pi}2\sin^{2}\frac{\hat{\theta}}{2}|{\cal
F}[\phi](\hat{\theta})|^{2}d\hat{\theta}$ $\displaystyle\cong$
$\displaystyle\frac{1}{2}\int_{-\infty}^{\infty}t^{2}|{\cal
F}[\psi](t)|dt=\frac{1}{2}\langle{\cal F}[\psi]|Q^{2}|{\cal
F}[\psi]\rangle=\frac{1}{2}\langle\psi|P^{2}|\psi\rangle.$ (10)

Here $Q$ is the multiplication operator and $P$ is the momentum operator
defined as $P\psi(x):=i\frac{d}{dx}\psi(x)$. In fact, the minimum eigenvalue
of $P^{2}$ on the function space $L^{2}([-1,1])$ is $\frac{\pi^{2}}{4}$.
Hence, the minimum of (10) is $\frac{\pi^{2}}{8}$, which coincides the
asymptotic behavior of (8) with $n=2N$.

Next, given an real number $T>0$, we maximize the probability satisfying the
condition $-\frac{T}{N}<\hat{\theta}-\theta<\frac{T}{N}$ (IH09, , Section 5).
For this aim, we choose the function $R(\theta,\hat{\theta})$ as the
probability satisfying the condition $|\hat{\theta}-\theta|\geq\frac{T}{N}$,
which is denoted by $R[T](\theta,\hat{\theta})$. Then, we have

$\displaystyle{\cal R}_{\max}[\mathsf{f}_{\\{-N,\ldots,N\\}},R[T]]=$
$\displaystyle{\cal R}_{av}[\mathsf{f}_{\\{-N,\ldots,N\\}},R[T]]={\cal
R}_{av}[\mathsf{f}_{\\{-N,\ldots,N\\}},R[T]]$ $\displaystyle=$
$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{\\{-N,\ldots,N\\}}}1-\frac{1}{2\pi}\int_{-\frac{T}{N}}^{\frac{T}{N}}|{\cal
F}[\phi](\hat{\theta})|^{2}d\hat{\theta}.$ (11)

For simple analysis, we focus on the case when the vector
$|\phi\rangle\in\mathcal{H}_{\\{-N,\ldots,N\\}}$ is given in the above way. As
shown in (IH09, , Section 5), we have

$\displaystyle\frac{1}{2\pi}\int_{-\frac{T}{N}}^{\frac{T}{N}}|{\cal
F}[\phi](\hat{\theta})|^{2}d\hat{\theta}\cong\int_{-T}^{T}|{\cal
F}[\psi](t)|dt.$ (12)

We define the projection $\Pi_{T}$ corresponding to the event that the
spectral of $Q$ belongs to $[-T,T]$. Since $\psi\in L_{c}^{2}([-1,1])$ belongs
to the range of the projection $\Pi_{1}$, we have

$\displaystyle\int_{-T}^{T}|{\cal F}[\psi](t)|dt=\langle{\cal
F}[\psi]|\Pi_{T}|{\cal F}[\psi]\rangle=\langle\psi|{\cal F}\Pi_{T}{\cal
F}|\psi\rangle=\langle\psi|\Pi_{1}{\cal F}\Pi_{T}{\cal F}\Pi_{1}|\psi\rangle.$
(13)

The problem (11) is converted to the maximization of $\langle\psi|\Pi_{1}{\cal
F}\Pi_{T}{\cal F}\Pi_{1}|\psi\rangle$. To discuss the maximum eigenvalue of
the operator $\Pi_{1}{\cal F}\Pi_{T}{\cal F}\Pi_{1}$, we consider the prolate
spheroidal wave function $\psi_{T}$, which is the solution of the differential
equation

$\displaystyle\frac{d}{dx}(1-x^{2})\frac{d\psi}{dx}+(\xi(T)-T^{2}x^{2})\psi(x)=0,$
(14)

where $\xi(T)$ is a real number depending on $T$111For the relation between
$\xi(T)$ and $T$, see Slepian and Pollak SP .. Slepian and Pollak SP  showed
that the function $\psi_{T}$ is the eigenfunction of the operator
$\Pi_{1}{\cal F}\Pi_{T}{\cal F}\Pi_{1}$ with the maximum eigenvalue
$\lambda(T)$, which behaves as Slepian

$\displaystyle 1-\lambda(T)\cong 4\sqrt{\pi
T}e^{-2T}\Big{(}1-\frac{3}{32T}+O(T^{2})\Big{)}.$ (15)

In this way, the asymptotic bahavior of the problem (11) is closely linked to
a special function, the prolate spheroidal wave function.

##  4 Energy constraint

Now, we impose an energy constraint on the input state on $\mathcal{H}$ for a
representation $\mathsf{f}$ (CMP, , Section 11). We define the Hamiltonian $H$
on $\mathcal{H}$ as

$\displaystyle H:=\sum_{j\in S}j^{2}I_{j},$ (16)

where $I_{j}$ is the projection to the subspace
$\mathcal{H}_{j}\otimes\mathcal{V}_{j}$. Then, we impose the following energy
constraint to the input state $\rho$ as

$\displaystyle\mathop{\rm Tr}\rho H\leq E.$ (17)

In the following, we consider the case with $S=\mathbb{Z}$, and denote the set
of states with the condition (17) by ${\cal S}_{E}$. We consider the following
minimizations

$\displaystyle{\cal R}_{\max}[\mathsf{f},R,E]:=$
$\displaystyle\min_{\rho\in{\cal S}_{E},{\cal M}}{\cal
R}_{\max}[\mathsf{f},R,\rho,{\cal M}],$ (18) $\displaystyle{\cal
R}_{av}[\mathsf{f},R,E]:=$ $\displaystyle\min_{\rho\in{\cal S}_{E},{\cal
M}}{\cal R}_{av}[\mathsf{f},R,\rho,{\cal M}].$ (19)

Let $\mathcal{H}_{\mathbb{Z},E}$ be the set of normalized vectors
$\phi\in\mathcal{H}_{\mathbb{Z}}$ to satisfy the condition
$\langle\phi|H|\phi\rangle\leq E$. When the error function $R$ satisfies the
symmetric condition (2), as shown in (CMP, , Theorem 2) as a variant of (6),
we have

$\displaystyle{\cal R}_{\max}[\mathsf{f},R,E]=$ $\displaystyle{\cal
R}_{av}[\mathsf{f},R,E]=\min_{|\phi\rangle\in\mathcal{H}_{S}}{\cal
R}[\mathsf{f}_{S},R,0,|\phi\rangle\langle\phi|,{\cal M}_{w}]$ $\displaystyle=$
$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{\mathbb{Z},E}}\frac{1}{2\pi}\int_{0}^{2\pi}R(0,\hat{\theta})\langle
w|\mathsf{f}_{S}(e^{i\hat{\theta}})|\phi\rangle\langle\phi|\mathsf{f}_{S}(e^{i\hat{\theta}})^{\dagger}|w\rangle
d\hat{\theta}$ $\displaystyle=$
$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{\mathbb{Z},E}}\frac{1}{2\pi}\int_{0}^{2\pi}R(0,\hat{\theta})|{\cal
F}[\phi](\hat{\theta})|^{2}d\hat{\theta}.$ (20)

To consider this problem, we define the function space $L^{2}_{p}((-\pi,\pi])$
as the space of the periodic square integrable functions with the period
$2\pi$. Then, we define the function space $L^{2}_{p,even}((-\pi,\pi])$ as the
space of even functions in $L^{2}_{p}((-\pi,\pi])$. Now, we choose
$R_{\sin}(\theta,\hat{\theta})=2\sin^{2}\frac{\theta-\hat{\theta}}{2}=1-\cos(\theta-\hat{\theta})$.
Then, as shown in (CMP, , Theorem 6 and Eq. (97)), we have

$\displaystyle\min_{|\phi\rangle\in\mathcal{H}_{\mathbb{Z},E}}\frac{1}{2\pi}\int_{0}^{2\pi}R_{\sin}(0,\hat{\theta})|{\cal
F}[\phi](\hat{\theta})|^{2}d\hat{\theta}$ $\displaystyle=$
$\displaystyle\kappa(E):=\min_{\psi\in
L^{2}_{p,even}(-\pi,\pi)}\\{\langle\psi|I-\cos
Q|\psi\rangle|\langle\psi|P^{2}|\psi\rangle\leq E,\|\psi\|=1\\}.$ (21)

To calculate the function $\kappa$, we define the function

$\displaystyle\gamma(s):=$ $\displaystyle\min_{\psi\in
L^{2}((-\pi,\pi]),\|\psi\|=1}\langle\psi|I-\cos Q+sP^{2}|\psi\rangle$
$\displaystyle=$ $\displaystyle\min_{\psi\in
L^{2}((-\pi/2,\pi/2]),\|\psi\|=1}\langle\psi|I-\cos Q+sP^{2}|\psi\rangle.$
(22)

Then, $\kappa(E)$ is given by the Legendre transform of $\gamma(s)$, i.e., as
shown in (CMP, , Lemma 6), we have the formula

$\displaystyle\kappa(E)=\max_{s>0}\gamma(s)-sE.$ (23)

The value $\gamma(s)$ can be characterized as the minimum value of $\gamma$
having the solution in $L^{2}((-\pi/2,\pi/2])$ of the following differential
equation.

$\displaystyle\frac{s}{4}\frac{d^{2}}{d\theta^{2}}\varphi(\theta)+(\gamma-1+\cos(2\theta))\varphi(\theta)=0,$
(24)

which is equivalent to

$\displaystyle\frac{d^{2}}{d\theta^{2}}\varphi(\theta)+(\frac{4(\gamma-1)}{s}+\frac{4}{s}\cos(2\theta))\varphi(\theta)=0.$
(25)

Now, we consider Mathieu equation:

$\displaystyle\frac{d^{2}}{d\theta^{2}}\varphi(\theta)+(a-2q\cos(2\theta))\varphi(\theta)=0.$
(26)

A function $\varphi$ satisfies the above equation if and only if the function
$\varphi$ is the eigenfunction of the differential operator
$P^{2}+2q\cos(2Q)$. The operator $X(q):=P^{2}+2q\cos(2Q)$ preserves the
subspace $L^{2}_{p,even}((-\frac{\pi}{2},\frac{\pi}{2}])$. Then, we denote the
minimum eigenvalue in $L^{2}_{p,even}((-\frac{\pi}{2},\frac{\pi}{2}])$ by
$a_{0}(q)$, which is also the minimum eigenvalue in
$L^{2}_{p}((-\frac{\pi}{2},\frac{\pi}{2}])$ (Wolf, , Section 28.2). Mathieu
function $\mathop{\rm ce}_{0}(\theta,q)$ is defined as the solution of (26)
with $a_{0}(q)$ (Wolf, , Section 28.2(vi)).

Then, since $\gamma(s)$ is $\gamma$ in (26), we have

$\displaystyle\gamma(s)=\frac{sa_{0}(\frac{2}{s})}{4}+1.$ (27)

Hence, using the formula (23), we have

$\displaystyle\kappa(E)=\max_{s>0}\frac{sa_{0}(\frac{2}{s})}{4}+1-sE.$ (28)

The minimum in (21) is attained if and only if ${\cal
F}[\psi](\theta)=\mathop{\rm ce}_{0}(\frac{\theta}{2},-\frac{2}{s_{E}})$,
where $s_{E}:=\mathop{\rm argmax}_{s>0}\frac{sa_{0}(\frac{2}{s})}{4}+1-sE$.

When $s\to 0$, we have the approximation;

$\displaystyle\gamma(s)\cong\sqrt{\frac{s}{2}}-\frac{s}{16}.$ (29)

Then, $\kappa(E)$ is approximated as

$\displaystyle\kappa(E)\cong\frac{1}{8E}-\frac{1}{128E^{2}}.$ (30)

##  5 Application to uncertainty relation

Interestingly, the relation (28) can be used for the uncertainty relation
between the position and the momentum on the periodic function space
$L^{2}_{p}((-\pi,\pi])$. In this function space, the uncertainty of the
position is formulated as the uncertainty for the pair of operators $(\cos
Q,\sin Q)$ as

$\displaystyle\Delta_{\varphi}^{2}(\cos Q,\sin Q):=\Delta_{\varphi}^{2}\cos
Q+\Delta_{\varphi}^{2}\sin Q$ $\displaystyle=$
$\displaystyle\langle\varphi|\cos^{2}Q|\varphi\rangle+\langle\varphi|\sin^{2}Q|\varphi\rangle-\langle\varphi|\cos
Q|\varphi\rangle^{2}-\langle\varphi|\sin Q|\varphi\rangle^{2}$
$\displaystyle=$ $\displaystyle 1-\langle\varphi|\cos
Q|\varphi\rangle^{2}-\langle\varphi|\sin Q|\varphi\rangle^{2}.$ (31)

On the other hand, the uncertainty of the momentum is given as
$\Delta_{\varphi}^{2}P=\langle\varphi|P^{2}|\varphi\rangle-\langle\varphi|P|\varphi\rangle^{2}$.
Thus, the uncertainty relation is formulated as the trade-off between
$\Delta_{\varphi}^{2}(\cos Q,\sin Q)$ and $\Delta_{\varphi}^{2}P$. That is,
this trade-off can be formulated as the following minimization

$\displaystyle\min_{\varphi\in
L_{p}^{2}([-\pi,\pi))}\\{\Delta_{\varphi}^{2}(\cos Q,\sin
Q)|\Delta_{\varphi}^{2}P\leq E,\|\varphi\|=1\\}.$ (32)

Since this problem has symmetry, we can restrict our function $\varphi$ to
satisfy the conditions $\langle\varphi|\sin Q|\varphi\rangle=0$ and
$\langle\varphi|P|\varphi\rangle=0$. Then, our problem is simplified to

$\displaystyle\min_{\varphi\in L_{p}^{2}([-\pi,\pi))}\\{1-\langle\varphi|\cos
Q|\varphi\rangle^{2}|\langle\varphi|P^{2}|\varphi\rangle\leq
E,\|\varphi\|=1\\}$ $\displaystyle=$ $\displaystyle 1-\Big{(}\max_{\varphi\in
L_{p}^{2}([-\pi,\pi))}\\{\langle\varphi|\cos
Q|\varphi\rangle|\langle\varphi|P^{2}|\varphi\rangle\leq
E,\|\varphi\|=1\\}\Big{)}^{2}$ $\displaystyle=$ $\displaystyle
1-\kappa(E)^{2}.$ (33)

By using (28), this trade-off is solved as the following relation (CMP, ,
Theorem 10).

$\displaystyle\min_{\varphi\in
L_{p}^{2}([-\pi,\pi))}\\{\Delta_{\varphi}^{2}(\cos Q,\sin
Q)|\Delta_{\varphi}^{2}P\leq
E,\|\varphi\|=1\\}=\max_{s>0}1-\Big{(}sE-\frac{sa_{0}(2/s)}{4}\Big{)}^{2}.$
(34)

In addition, the minimum in (34) is attained when and only when the function
$\varphi$ is given as a shift of the Mathieu function $\mathop{\rm
ce}_{0}(\theta 2,-\frac{2}{s_{E}})$. Moreover, the right hand side of (34) is
asymptotically expanded as $\frac{1}{4E}-\frac{1}{32E^{2}}$ when $E$ goes to
infinity.

##  6 Conclusion

This paper explains several applications of special functions to phase
estimation. In particular, we have addressed prolate spheroidal wave function
and Mathieu function. Although Mathieu function works for phase estimation
under a certain energy constraint, it also works for the estimation of the
unknown unitary under a certain energy constraint when the set of unknown
unitaries form a group representation of SU(2) CMP .

Another type of energy constraint for phase estimation problem was discussed
in the reference HVK . This problem setting uses a function related to Gamma
function. In this way, special functions have various applications in quantum
information. As another example of special functions to quantum information,
the reference HAY  studied the relation between Askey scheme and quantum state
distinguishability. It is expected that more special functions will be applied
to the analysis on various types of quantum information processings.

All the presented results assume the noiseless case. While the Heisenberg
scaling with the noisy case was studied in HLY , the relations with special
functions were not studied in the noisy case. Therefore, it is an open problem
to extend these relations to the noisy case.

###### Acknowledgements.

The author was supported in part by the National Natural Science Foundation of
China (Grants No. 62171212) and Guangdong Provincial Key Laboratory (Grant No.
2019B121203002).

## References

  * (1) Holevo, A.S.: Covariant measurements and uncertainty relations. Rep. Math. Phys. 16, 385 – 400 (1979).
  * (2) Holevo, A. S.: Probabilistic and Statistical Aspects of Quantum Theory. North-Holland, Amsterdam (1982). Originally published in Russian in 1980.
  * (3) Hayashi, M.: Asymptotic estimation theory for a finite dimensional pure state model. Journal of Physics A: Mathematical and General 31, 4633 – 4655 (1998)
  * (4) Hayashi, M.: A Group Theoretic Approach to Quantum Information, Springer (2017). (Originally published from Kyoritsu Shuppan in 2014 with Japanese.)
  * (5) Chiribella, G., D’Ariano, G.M., Sacchi, M.F.: Optimal estimation of group transformations using entanglement. Phys. Rev. A 72, 042338 (2005)
  * (6) Hayashi, M.: Fourier Analytic Approach to Quantum Estimation of Group Action. Communications in Mathematical Physics 347, 3 – 82 (2016).
  * (7) Luis, A., Perina, J.: Optimum phase-shift estimation and the quantum description of the phase difference. Phys. Rev. A 54, 4564 (1996)
  * (8) Bužek, V., Derka, R., Massar, S.: Optimal quantum clocks. Phys. Rev. Lett. 82, 2207 (1999).
  * (9) Hayashi, M.: Parallel treatment of estimation of SU(2) and phase estimation. Phys. Lett. A 354(3), 183–189 (2006).
  * (10) Imai, H., Hayashi, M.: Fourier analytic approach to phase estimation in quantum systems. New Journal of Physics 11, 043034 (2009).
  * (11) Bagan, E., Baig, M., Munoz-Tapia, R.: Quantum reverse-engineering and reference-frame alignment without nonlocal correlations. Phys. Rev. A 70, 030301(R) (2004).
  * (12) Chiribella, G., D’Ariano, G.M., Perinotti, P., Sacchi, M.F.: Efficient use of quantum resources for the transmission of a reference frame. Phys. Rev. Lett. 93, 180503 (2004).
  * (13) Giovannetti,V., Lloyd, S., Maccone, L.: Quantum-enhanced measurements: beating the standard quantum limit. Science 306, 1330–1336 (2004).
  * (14) Giovannetti, V., Lloyd, S., Maccone, L.: Quantum-enhanced “Quantum metrology. Phys. Rev. Lett. 96, 010401 (2006).
  * (15) Nagata, T., Okamoto, R., O’Brien, J., Sasaki, K., Takeuchi, S.: Beating the standard quantum limit with four-entangled photons. Science 316(5825), 726 (2007).
  * (16) Okamoto, R., Hofmann, H.F., Nagata, T., O’Brien, J.L., Sasaki, K., Takeuchi, S.: Beating the standard quantum limit: phase super-sensitivity of N-photon interferometers. N. J. Phys. 10, 073033 (2008).
  * (17) Jones, J.A., Karlen, S.D., Fitzsimons, J., Ardavan, A., Benjamin, S.C., Briggs, G.A.D., Morton, J.J.L.: Magnetic field sensing beyond the standard quantum limit using 10-spin NOON states. Science 324, 1166–1168 (2009).
  * (18) Hayashi, M., Matsumoto, K.: Statistical model with measurement degree of freedom and quantum physics. RIMS koukyuroku No 1055 (Kyoto: Kyoto University) p 96 (1998) (In Japanese); Hayashi, M., Matsumoto, K.: Asymptotic Theory of Quantum Statistical Inference. ed M Hayashi, Singapore: World Scientific, 2005, p. 162 (reprinted, English translation).
  * (19) Hayashi, M.: Comparison between the Cramer-Rao and the mini-max approaches in quantum channel estimation. Commun. Math. Phys. 304(3), 689–709 (2011).
  * (20) Slepian, D., Pollak, H. O.: Prolate spheroidal wave functions, Fourier analysis and uncertainty-I. Bell Syst. Tech. J. 40, 43 – 63 (1961).
  * (21) Slepian, D.: Some asymptotic expansions for prolate spheroidal functions. J. Math. Phys. 44, 99–140 (1965)
  * (22) Wolf, G.: Mathieu Functions and Hill’s Equation (2013).Available from http://dlmf.nist.gov/28.
  * (23) Hayashi, M., Vinjanampathy, S., Kwek, L. C.: Resolving unattainable Cramer–Rao bounds for quantum sensors. J. Phys. B: At. Mol. Opt. Phys. 52, 015503 (2019).
  * (24) Hayashi, M., Hora, A., Yanagida S., Asymmetry of tensor product of asymmetric and invariant vectors arising from Schur-Weyl duality based on hypergeometric orthogonal polynomial. arXiv:2104.12635 (2021).
  * (25) Hayashi, M., Liu, Z.-W., Yuan, H.: Global Heisenberg scaling in noisy and practical phase estimation, Quantum Science and Technology, 7, 025030 (2022).