# Assertion Detection in Clinical Natural Language Processing using Large
Language Models

1st Yuelyu Ji  Dept. of Computing and Information
University of Pittsburgh
Pittsburgh, US
<EMAIL_ADDRESS>2nd Zeshui Yu  Dept. of Pharmaceutical Sciences
University of Pittsburgh
Pittsburgh, US
<EMAIL_ADDRESS>3rd Yanshan Wang  Dept. of Health Information Management
University of Pittsburgh
Pittsburgh, US
<EMAIL_ADDRESS>
###### Abstract

In this study, we aim to address the task of assertion detection when
extracting medical concepts from clinical notes, a key process in clinical
natural language processing (NLP). Assertion detection in clinical NLP usually
involves identifying assertion types for medical concepts in the clinical
text, namely certainty (whether the medical concept is positive, negated,
possible, or hypothetical), temporality (whether the medical concept is for
present or the past history), and experiencer (whether the medical concept is
described for the patient or a family member). These assertion types are
essential for healthcare professionals to quickly and clearly understand the
context of medical conditions from unstructured clinical texts, directly
influencing the quality and outcomes of patient care. Although widely used,
traditional methods, particularly rule-based NLP systems and machine learning
or deep learning models, demand intensive manual efforts to create patterns
and tend to overlook less common assertion types, leading to an incomplete
understanding of the context. To address this challenge, our research
introduces a novel methodology that utilizes Large Language Models (LLMs) pre-
trained on a vast array of medical data for assertion detection. We enhanced
the current method with advanced reasoning techniques, including Tree of
Thought (ToT), Chain of Thought (CoT), and Self-Consistency (SC), and refine
it further with Low-Rank Adaptation (LoRA) fine-tuning. We first evaluated the
model on the i2b2 2010 assertion dataset. Our method achieved a micro-averaged
F-1 of 0.89, with 0.11 improvements over the previous works. To further assess
the generalizability of our approach, we extended our evaluation to a local
dataset that focused on sleep concept extraction. Our approach achieved an F-1
of 0.74, which is 0.31 higher than the previous method. The results show that
using LLMs is a viable option for assertion detection in clinical NLP and can
potentially integrate with other LLM-based concept extraction models for
clinical NLP tasks.

###### Index Terms:

Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning

##  I Introduction

Figure 1: Examples of assertions in clinical texts. Medical concepts and the
corresponding assertions are highlighted.

Assertion detection is a key task within the area of Clinical Natural Language
Processing (NLP)[1]. It usually involves identifying the assertion types for
medical concepts in the clinical text, namely certainty (whether the medical
concept is positive, negated, possible, or hypothetical), temporality (whether
the medical concept is for present or the previous history), and experiencer
(whether the medical concept is described for the patient or a family member).
Figure 1 shows an example of medical concepts and the corresponding
assertions. This task plays a crucial role in understanding medical concepts
from the free-text Electronic Health Records (EHRs), directly impacting the
accuracy of clinical decision-making and the efficiency of patient care. As a
core component of clinical NLP, assertion detection also holds significant
potential for enhancing information retrieval and automated clinical
reasoning. However, it faces challenges such as class distribution imbalance
and the unstructured nature of clinical notes. Particularly challenging is the
classification of assertions like ’Possible’ and ’Family’, which are often
less frequently occurring and ambiguously expressed. Previous studies have
widely applied rule-based methods such as NegEx[2] and ConText[1] in clinical
NLP software, setting a benchmark in medical informatics with applications in
tools like OHNLP Toolkit[3], MedTagger [4], medspaCy [5], and cTAKES [6].
However, these rule-based approaches are limited by their fixed patterns and
inability to exhaust all possibilities, often leading to low recall rates. To
overcome these limitations, deep learning methods like convolutional neural
networks (CNNs) and Long short-term memory (LSTM) [7, 8, 9] were introduced.
Although these approaches show promise, they still require substantial amounts
of labeled data and tend to underperform when dealing with small or imbalanced
datasets.

To address the above limitations, recent attention has been focused on Large
Language Models (LLMs) for their superior capacity to understand and generate
human-like text. LLMs such as GPT-3[10] and LLaMA[11] are trained on vast
datasets, enabling them to capture intricate patterns in language that rule-
based systems cannot. Furthermore, these models introduce the concept of in-
context learning [12], an idea that enables LLMs to understand and perform new
tasks efficiently by conditioning on a few examples in the input, thereby
grasping the task’s structure and generating response format through these
illustrative examples. Our method employs in-context learning techniques,
including Tree of Thought (ToT) [13], Chain of Thought (CoT)[14], and Self-
Consistency (SC)[15]. These methods leverage a small number of samples to
rapidly understand new tasks and self-reflect, achieving meaningful success in
clinical NLP tasks such as question answering and text generation[16, 17].

In this paper, we treated assertion detection as a generative task, generating
corresponding texts through various in-context learning methods, thereby
utilizing the model’s comprehension to tackle the task. Moreover, we
introduced Low-Rank Adaptation (LoRA) fine-tuning [18] to enhance the LLM’s
understanding of instructions and achieved promising results with minimal data
training. We tested our in-context learning and LoRA fine-tuning techniques on
two datasets, including the public i2b2 2010 assertion dataset[19] and a local
private corpus from the University of Pittsburgh Medical Center (UPMC). The
validation on the local private corpus could validate the generalizability of
the proposed approach. The results indicate that our method outperforms
previous works across all six assertion categories on both datasets.

The three key contributions of this work are as follows:

  * •

We have developed and rigorously evaluated a range of LLMs enhanced by
advanced reasoning methodologies, including ToT, CoT, and SC. These methods
significantly improve the LLMs’ capabilities in assertion detection, providing
deeper insights and more trustworthy interpretations of medical narratives.

  * •

Our study includes fine-tuning the LLaMA2-7B model[11] using LoRA to achieve
greater precision and contextual understanding. This optimization step refines
the model’s performance, making it more adept at handling the specific details
of clinical assertion detection.111We open-source the code and models to the
community for further research at:
https://github.com/JoyDajunSpaceCraft/Assertion_LLM.

  * •

The experiments on both public and private datasets show the fine-tuned LLMs’
effectiveness in carefully detecting and categorizing medical assertions and
highlight their generalizability and adaptability to specialized healthcare
domains.

##  II Related Work

The landscape of clinical assertion detection has significantly evolved, with
methodologies ranging from rule-based systems to advanced machine learning and
deep learning approaches.

One of the earliest and most influential methods is the ConText algorithm [1],
which utilizes hand-crafted patterns for negation and temporality
classification in clinical text [2]. ConText has been integral to various
software applications, demonstrating its enduring impact on the field.
Notably, it’s been incorporated into the OHNLP Toolkit for EHR-based clinical
research [3], MedTagger for cohort identification [4], medspaCy for clinical
text processing [5], and cTAKES for text analysis and knowledge extraction
[6]. These implementations demonstrate ConText’s significant influence and
adaptability in medical informatics.

The i2b2 Challenge Assertions Task [19] further motivated the development of
machine learning models like SVMs and CRFs [1], which provided improvements
but also emphasized the complexity and challenges of clinical data.

The advent of deep learning introduced neural networks, including CNNs and
LSTMs, to the task [7, 8]. These models showed promise but were often limited
by their need for large labeled datasets. This limitation was solved to some
extent by transformer-based models like NegBert [20], which marked significant
advancements but still relied heavily on extensive labeled data.

Recently, prompt-based learning methods utilizing Large Language Models (LLMs)
have emerged as a potent tool for clinical NLP tasks. These methods allow
models to engage in few-shot learning and swiftly adapt to new tasks, as
evidenced by [9]. However, the literature on LLMs in clinical assertion
detection has been scarce. Nonetheless, there is a paucity of literature
regarding utilizing LLMs in clinical assertion detection. Our work expands
upon this innovative foundation by amalgamating prompt engineering with
sophisticated reasoning methodologies, such as ToT[13], CoT[14], and SC [15].
These methods not only facilitate complex inference but also enhance the
interpretability of language models, particularly in specialized tasks like
medical diagnosis [21].

Moreover, applying the LoRA method enables more efficient fine-tuning, thereby
strengthening the model’s capacity to generalize across a diverse range of
clinical narratives [22]. This approach addresses the previous limitations by
reducing the dependency on large annotated datasets and improving model
performance on minority classes.

The field has witnessed a wide array of methodologies, ranging from the rule-
based approach of ConText to the most recent transformer-based models. Our
research extends this evolution by harnessing advanced reasoning and efficient
fine-tuning techniques in LLMs, thereby advancing the frontiers of clinical
assertion detection.

##  III Dataset

Our investigation into clinical assertion detection utilizes two datasets,
namely the i2b2 assertion dataset and a local Sleep dataset, as listed in
Table I

TABLE I: Assertion Distribution in i2b2 and Sleep Datasets

Label | i2b2 | Sleep
---|---|---
Train | Eval | Train | Eval
Family | 185 (5.26%) | 47 (7.79%) | 40 (10.99%) | 12 (13.04%)
Historical | - | - | 81 (22.25%) | 19 (20.65%)
Hypothetical | 317 (9.02%) | 48 (7.96%) | 33 (9.07%) | 9 (9.78%)
Negated | 758 (21.57%) | 127 (21.06%) | 33 (9.07%) | 15 (16.30%)
Possible | 265 (7.29%) | 42 (6.97%) | 46 (12.64%) | 9 (9.78%)
Positive | 1988 (55.58%) | 324 (53.73%) | 131 (35.99%) | 28 (30.43%)
Total | 3513 | 588 | 364 | 92

Figure 2: Methods used in the assertion detection.

1\. i2b2 Dataset: The i2b2 2010 assertion dataset provides annotated data from
discharge summaries and progress notes sourced from three different medical
institutions, as referenced in [19]. It includes manual annotations for six
types of assertions related to medical concepts within clinical documentation:
Present (Positive), Absent (Negated), Possible, Hypothetical, Conditional, and
Family. We only used the category matching our task to fit our assertion
definition, so we didn’t use the Conditional category in the i2b2 dataset.

2\. Sleep Dataset: A curated dataset from UPMC comprises annotated records
related to sleep disorders, specifically targeting snoring and obstructive
sleep apnea (OSA). This choice is motivated by the high prevalence of sleep
disorders, their underdiagnosis, and the negative impact on quality of life,
morbidity, and survival.

Sporadic snoring is often considered harmless, affecting a substantial
proportion of the adult population (44 percent of males and 28 percent of
females aged 30 to 60). However, persistent and loud snoring may suggest a
potential underlying issue like OSA, warranting medical intervention. OSA is a
sleep disorder characterized by obstructive apneas and hypopneas that occur
when upper airway resistance is sufficient to disrupt sleep. Patients with OSA
are at heightened risk of adverse clinical complications, such as metabolic
syndrome, cardiovascular disease, or neuropsychiatric dysfunction [23].

The clinical notes were retrieved from the clinical data warehouse. We used
keyword sampling to identify sleep-related notes and randomly sampled a total
of 456 clinical notes. Annotations in this dataset reflect assertion types
like Positive, Negated, Possible, Hypothetical, Family, and Historical. These
granular categories enable an in-depth analysis of the LLMs’ performance in
recognizing and classifying specific medical assertions. The annotations were
carefully reviewed and adjudicated by a healthcare professional (ZY, co-author
of this study) to ensure their accuracy and reliability. This process involved
a comprehensive examination of relevant clinical notes. This study is approved
by The University of Pittsburgh’s Institutional Review Board (IRB).

The distribution for the data can be found in the Table I.

##  IV Methodology

Figure 2 illustrates the proposed approach of using LLMs for assertion
detection.

###  IV-A In-context Learning

####  IV-A1 Chain-of-Thought Prompting (CoT)

Chain of Thought (CoT) prompting enhances the capability of language models to
address complex tasks by breaking down problems into a series of granular and
progressive subtasks. This approach leads to more structured and
understandable solutions through methodical step-by-step reasoning. For
example, when assessing a patient’s symptoms, the reasoning might unfold as
follows:

> ”The patient has been experiencing persistent migraines, often escalating to
> nausea in the evenings, which have not responded to over-the-counter
> medications for the last month.”

LLM will think stepwisely and generate the question based on understanding the
case.

Guided Question 1: Considering the patient’s description of their migraines as
”persistent” and ”escalating to nausea,” can we assert that the condition is
’Positive’ and currently affecting the patient’s health?

Expected Answer 1: Yes, the description indicates an ongoing and troublesome
condition, confirming a ’Positive’ assertion for the current state of the
patient’s health.

Guided Question 2: Does the lack of response to over-the-counter medications
over the last month strengthen the ’Positive’ assertion for the severity and
presence of the patient’s condition?

Expected Answer 2: Yes, the fact that common treatments have been ineffective
for a duration of ”the last month” suggests that the condition is ’Positive’
and may require further medical evaluation or treatment. It will break down
into different reasoning steps. By answering the questions step by step, LLM
will provide the result of assertion detection.

TABLE II: Performance Comparison Across Models and Methods on the i2b2 and Sleep Datasets. Dataset | Assertion Category | ChatGPT | LLaMA2-7B | BERT | ConText
---|---|---|---|---|---
Simple | CoT | ToT | SC | Simple | CoT | ToT | SC | LoRA
i2b2 | Family | 0.67 | 0.7 | 0.55 | 0.57 | 0.87 | 0.85 | 0.85 | 0.92 | 0.67 | - | 0.72
Historical | - | - | - | - | - | - | - | - | - | - | -
Hypothetical | 0.66 | 0.56 | 0.68 | 0.55 | 0.94 | 0.91 | 0.91 | 0.96 | 0.875 | - | -
Negated | 0.53 | 0.57 | 0.55 | 0.69 | 0.86 | 0.88 | 0.9 | 0.93 | 0.98 | 0.84 | 0.74
Possible | 0.63 | 0.66 | 0.65 | 0.7 | 0.95 | 0.95 | 0.93 | 0.95 | 0.96 | 0.0 | 0.0
Positive | 0.62 | 0.66 | 0.65 | 0.72 | 0.88 | 0.9 | 0.91 | 0.95 | 0.99 | 0.81 | 0.89
Sleep | Family | 0.72 | 0.5 | 0.22 | 0.43 | 0.53 | 0.55 | 0.46 | 0.4 | 0.53 | - | 0.6
Historical | 0.72 | 0.63 | 0.61 | 0.67 | 0.76 | 0.86 | 0.9 | 0.71 | 0.76 | - | 0.7
Hypothetical | 0.44 | 0.55 | 0.11 | 0.44 | 0.11 | 0.11 | 0.11 | 0.0 | 0.88 | - | -
Negated | 0.4 | 0.36 | 0.5 | 0.0 | 0.14 | 0.5 | 0.27 | 0.29 | 0.14 | 0.25 | 0.3
Possible | 0.0 | 0.29 | 0.57 | 0.33 | 0.36 | 0.36 | 0.36 | 0.5 | 0.36 | 0.0 | 0.0
Positive | 0.62 | 0.78 | 0.74 | 0.69 | 0.91 | 0.85 | 0.83 | 0.83 | 0.92 | 0.46 | 0.58

####  IV-A2 Self-Consistency over Diverse Reasoning Paths (SC)

SC over Diverse Reasoning Paths evaluates the consistency of multiple
reasoning pathways, favoring the most common conclusion. It optimizes the
robustness of decision-making processing by considering various possible
solutions and selecting the most frequent one, thereby reducing reliance on a
single CoT. Formalizing the self-consistency is captured by the equation:

$\hat{a}=\arg\max_{a\in A}\sum_{i=1}^{m}(a_{i}=a)$ (1)

The $\hat{a}$ is the final assertion outcome. $A$, which in the context of
medical assertion detection typically includes ’True’ or ’False’. $m$ is the
number of interpretative paths generated, which in Figure 2 self-consistency
is 2. When the assertion outcome $a_{i}$ of the $i_{th}$ path aligns with the
assertion $a$. The method aggregates the outcomes of all paths and selects the
most frequently occurring assertion as the final diagnosis.

####  IV-A3 Tree of Thoughts (ToT) Framework

The ToT framework employs a heuristic-guided decision tree to optimize
problem-solving. It decomposes complex problems, explores multiple reasoning
paths, and uses heuristics to determine the most effective solution. And for
every part, LLM would solve the partial problems.

Formalizing the optimal reasoning path is captured by the equation:

$s^{*}=\arg\max_{s\in S}V(s),$ (2)

where $s^{*}$ is the most supported conclusion, $S$ is the set of reasoning
paths considered, and $V(s)$ represents the heuristic evaluation of each path.
In the context of the medical diagnosis depicted, $s^{*}$ is the diagnosis
affirmed by the process, $S$ includes the LLM’s observations and their
interpretations, and $V(s)$ reflects the evaluation of these hypotheses based
on their alignment with known clinical information. At the same time, it is
essential to ensure that the chosen path in the search process is optimal
within the ToT framework. This guarantees that our model navigates through the
reasoning space efficiently.

###  IV-B Efficient Fine-Tuning Method

For the specific task of clinical assertion detection, we efficiently fine-
tuned our pre-trained language model, LLaMA2-7B with LoRA. LoRA introduces
trainable low-rank matrices $A$ and $B$ into the Transformer layers by
reducing the number of parameters requiring adaptation. This technique allows
for quicker adaptation and resource-efficient customization, which is
particularly beneficial for our medical-focused model that demands high
precision and interpretability.

The adaptation process is formulated in the following equation:

$\Delta W=BA$ (3)

where $\Delta W$ is the update to the pre-trained weight matrix $W$. This
reparameterization strategy enables targeted fine-tuning, which is crucial for
processing complex clinical narratives for the i2b2 dataset and extracting
sleep-related conditions for the Sleep dataset.

##  V Experiments and Results

###  V-A Experimental Setup

We evaluated the performance of two LLMs, LLaMA2-7B, and ChatGPT 3.5 turbo,
across different assertion categories using the F-1 score metric. We employed
CoT, ToT, and SC prompt engineering approaches for both LLMs. In addition, the
open source LLM LLaMA2-7B was fine-tuned with LoRA on an NVIDIA A100 GPU, with
each session lasting one hour. Due to privacy considerations, we used the
Azure OpenAI ChatGPT 3.5 turbo. In comparison, the BERT model [24] and ConText
[1] algorithm are used as baseline approaches. The Simple approach used no in-
context learning method and only asked the LLM whether the medical term
categorized to the assertion.

###  V-B Dataset and Methodological Comparative Analysis

Our analysis across the i2b2 and Sleep datasets(refer to Tables II) showed
notable F1 score variability across different datasets and assertion
categories.

Performance Comparison Across Assertion Categories: We observed that optimal
F1 scores vary across different assertion categories on the two datasets.
Specifically, the ’Family,’ ’Hypothetical,’ and ’Positive’ categories
demonstrated relatively minor disparities in their best F1 scores across the
datasets: 0.72 and 0.92 for the ’Family,’ 0.96 and 0.88 for the
’Hypothetical,’ and 0.99 and 0.92 for the ’Positive’ category, respectively.
The close of these scores suggested that the recognition performance for these
two categories was comparatively stable across datasets. Conversely, the
’Historical’, ’Hypothetical’, ’Negated’, and ’Possible’ categories exhibited
significant differences in their best F1 scores between the datasets. For
example, the best F1 score across datasets was 0.98 and 0.5 for ’Negated’ and
0.95 and 0.57 for ’Possible’.

Performance Comparison Across Approaches: Our examination of the i2b2 and
Sleep datasets revealed distinct patterns in model performance. In the i2b2
dataset, methods like Simple, CoT, ToT, and SC consistently delivered strong
results across categories, with LLaMA2-7B performing better in in-context
learning. Conversely, in the Sleep dataset, ChatGPT’s ToT method excelled in
’Negated’ (0.5) and ’Possible’ (0.57) categories.

Notably, the LLaMA2-7B model, when paired with the ToT approach, achieved an
exceptional F1 score of 0.90 in ’Historical’ within the Sleep dataset.
Moreover, the fine-tuned LoRA technique performed better in the i2b2 dataset,
particularly in ’Negated’ (0.98) and ’Positive’ (0.99), and continued to
perform well in the Sleep dataset, especially in ’Hypothetical’ (0.88) and
’Positive’ (0.92).

Comparatively, baseline models like BERT and ConText also showed commendable
results. For instance, ConText’s ’Family’ score (0.72) outperformed ChatGPT’s
CoT in the i2b2 dataset, and BERT’s performance in ’Negated’ (0.84) outdid
ChatGPT’s CoT, emphasizing the competitive edge of traditional models in
certain contexts.

Performance Comparison Across Dataset: Analysis of the i2b2 and Sleep datasets
revealed notable variances in F1 scores across assertion
categories.’Hypothetical’ and ’Positive’ categories showed relatively stable
performance, with ’Hypothetical’ scoring 0.88 and 0.96 and ’Positive’ scoring
0.92 and 0.99 across Sleep and i2b2 datasets respectively ’Family’ presented
minor variation, scoring 0.72 in Sleep and 0.92 in i2b2. Conversely, the
’Negated’ and ’Possible’ categories showed more significant variations. For
instance, in the ’Negated’ category, the range was from 0.5 in the Sleep
dataset to 0.98 in the i2b2 dataset. Similarly, in the ’Possible’ category, it
varied from 0.57 in Sleep to 0.96 in i2b2. These disparities emphasize the
complexity of achieving consistent recognition across different datasets.

##  VI Discussion

###  VI-A Error Analysis

We conducted an error analysis of the proposed approaches on two datasets.

####  VI-A1 Contextual Ambiguity

Contextual ambiguity presents a significant challenge in clinical assertion
detection. It occurs when models encounter clinical narratives with symptoms
or conditions described ambiguously or indirectly. For example, terms like
’snoring’ and ’sleepiness during the day’ in clinical narratives (Table III)
might suggest various conditions beyond OSA. However, in the absence of a
clear diagnostic label, models may mistakenly think this description indicates
OSA. This results in wrong, where the model incorrectly labels a case as
’Positive’ for OSA without a definitive present diagnosis, as demonstrated in
the clinical narrative below:

TABLE III: Clinical Narrative Demonstrating Contextual Ambiguity. Easy
bruising sleep: snoring, sleepiness during the day ,
---
falling asleep at work, falling asleep easily watching tv, insomnia
psych: depression, anxiety, panic attacks, suicidal thoughts, homicidal
thoughts alcoholic drinks per day: 0 days per week: I have personally
reviewed the nursing note and triage note for this patient.
Model Output: ’Positive’ for OSA
Correct Label: Not Specified

####  VI-A2 Long Dependencies

Long dependencies are characterized by the need to connect entities and their
associated cues that are separated by more than a few tokens. The example is
shown in Table IV. These dependencies are responsible for a substantial
portion of errors, as models might not be configured to consider distant
assertion cues effectively.

TABLE IV: Clinical Narrative Demonstrating Long Dependency After an initial
assessment of joint stiffness, the patient’s
---
symptoms were managed conservatively.…Further evaluation
revealed significant improvement, and the patient’s
previously reported symptoms are no longer present.
Model Output: ’Positive’ for joint stiffness
Correct Label: ’Negated’ for joint stiffness

Here, the resolution of symptoms away from a distant cue should negate the
initial assertion of the condition, but if the model focuses only on adjacent
information, this key information might be missed.

###  VI-B Limitation

This study is subject to several limitations. Firstly, the generalizability of
the study is confined, as the clinical note data was sourced solely from one
institution. Expanding the dataset to include clinical notes from multiple
institutions might enhance the robustness of the study. Secondly, we employed
the LLaMA2-7B model due to computational constraints. While acknowledging that
larger models, such as the 13B and 70B variants, might offer improved results,
their usage was impractical within our computing framework. Future studies
utilizing these larger models could uncover additional performance
enhancements. Thirdly, the domain of LLMs is evolving rapidly.

##  VII Future work

Our results suggest that while traditional tools like ConText have promising
performance, LLMs and new methods such as ToT, CoT, and SC offer significant
improvements in analyzing complex clinical texts. Given the prevalent
application of ConText in diverse clinical NLP tools, LLMs have the potential
to supplant older rule-based approaches effectively. This transition heralds a
promising avenue for future investigations in clinical assertion detection and
healthcare analytics, potentially culminating in more dependable and efficient
results.

## Acknowledgments

This work was supported by the National Institutes of Health under award
number U24 TR004111 and R01 LM014306. The content is solely the responsibility
of the authors and does not necessarily represent the official views of the
National Institutes of Health. We extend our gratitude to Dr. Zeshui Yu for
his diligent work in annotating the dataset.

## References

  * [1] H. Harkema, J. N. Dowling, T. Thornblade, and W. W. Chapman, “Context: an algorithm for determining negation, experiencer, and temporal status from clinical reports,” _Journal of biomedical informatics_ , vol. 42, no. 5, pp. 839–851, 2009.
  * [2] W. W. Chapman, W. Bridewell, P. Hanbury, G. F. Cooper, and B. G. Buchanan, “A simple algorithm for identifying negated findings and diseases in discharge summaries,” _Journal of biomedical informatics_ , vol. 34, no. 5, pp. 301–310, 2001.
  * [3] S. Liu, A. Wen, L. Wang, H. He, S. Fu, R. Miller, A. Williams, D. Harris, R. Kavuluru, M. Liu _et al._ , “An open natural language processing (nlp) framework for ehr-based clinical research: a case demonstration using the national covid cohort collaborative (n3c),” _Journal of the American Medical Informatics Association_ , vol. 30, no. 12, pp. 2036–2040, 2023.
  * [4] H. Liu, S. J. Bielinski, S. Sohn, S. Murphy, K. B. Wagholikar, S. R. Jonnalagadda, K. Ravikumar, S. T. Wu, I. J. Kullo, and C. G. Chute, “An information extraction framework for cohort identification using electronic health records,” _AMIA Summits on Translational Science Proceedings_ , vol. 2013, p. 149, 2013.
  * [5] H. Eyre, A. B. Chapman, K. S. Peterson, J. Shi, P. R. Alba, M. M. Jones, T. L. Box, S. L. DuVall, and O. V. Patterson, “Launching into clinical space with medspacy: a new clinical text processing toolkit in python,” in _AMIA Annual Symposium Proceedings_ , vol. 2021. American Medical Informatics Association, 2021, p. 438.
  * [6] G. K. Savova, J. J. Masanz, P. V. Ogren, J. Zheng, S. Sohn, K. C. Kipper-Schuler, and C. G. Chute, “Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications,” _Journal of the American Medical Informatics Association_ , vol. 17, no. 5, pp. 507–513, 2010.
  * [7] Z. Qian, P. Li, Q. Zhu, G. Zhou, Z. Luo, and W. Luo, “Speculation and negation scope detection via convolutional neural networks,” in _Proceedings of the 2016 conference on empirical methods in natural language processing_ , 2016, pp. 815–825.
  * [8] E. Sergeeva, H. Zhu, P. Prinsen, and A. Tahmasebi, “Negation scope detection in clinical notes and scientific abstracts: a feature-enriched lstm-based approach,” _AMIA Summits on Translational Science Proceedings_ , vol. 2019, p. 212, 2019.
  * [9] S. Wang, L. Tang, A. Majety, J. F. Rousseau, G. Shih, Y. Ding, and Y. Peng, “Trustworthy assertion classification through prompting,” _Journal of biomedical informatics_ , vol. 132, p. 104139, 2022.
  * [10] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020.
  * [11] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [12] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen _et al._ , “In-context learning and induction heads,” _arXiv preprint arXiv:2209.11895_ , 2022.
  * [13] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” 2023.
  * [14] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023.
  * [15] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” 2023.
  * [16] P. M. Arachchige and S. Arosh, “Large language models (llm) and chatgpt: a medical student perspective,” _European Journal of Nuclear Medicine and Molecular Imaging_ , vol. 50, no. 8, pp. 2248–2249, 2023.
  * [17] J. Yuan, R. Tang, X. Jiang, and X. Hu, “Llm for patient-trial matching: Privacy-aware data augmentation towards better performance and generalizability,” _arXiv preprint arXiv:2303.16756_ , 2023.
  * [18] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021.
  * [19] Ö. Uzuner, B. R. South, S. Shen, and S. L. DuVall, “2010 i2b2/va challenge on concepts, assertions, and relations in clinical text,” _Journal of the American Medical Informatics Association_ , vol. 18, no. 5, pp. 552–556, 2011.
  * [20] A. Khandelwal and S. Sawant, “Negbert: A transfer learning approach for negation detection and scope resolution,” _CoRR_ , vol. abs/1911.04211, 2019. [Online]. Available: http://arxiv.org/abs/1911.04211
  * [21] S. Sivarajkumar, M. Kelley, A. Samolyk-Mazzanti, S. Visweswaran, and Y. Wang, “An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing,” 2023.
  * [22] A. Gema, L. Daines, P. Minervini, and B. Alex, “Parameter-efficient fine-tuning of llama for the clinical domain,” _arXiv preprint arXiv:2307.03042_ , 2023.
  * [23] L. R. Kline, N. Collop, and G. Finlay, “Clinical presentation and diagnosis of obstructive sleep apnea in adults,” _Uptodate. com [Internet]_ , 2017.
  * [24] B. van Aken, I. Trajanovska, A. Siu, M. Mayrdorfer, K. Budde, and A. Loeser, “Assertion detection in clinical notes: Medical language models to the rescue?” in _Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations_ , C. Shivade, R. Gangadharaiah, S. Gella, S. Konam, S. Yuan, Y. Zhang, P. Bhatia, and B. Wallace, Eds. Online: Association for Computational Linguistics, Jun. 2021, pp. 35–40. [Online]. Available: https://aclanthology.org/2021.nlpmc-1.5


# Spatially resolved spectroscopy of alkali metal vapour diffusing inside
hollow-core photonic crystal fibres

Daniel R Häupl1,2, Daniel Weller3, Robert Löw3 and Nicolas Y Joly1,2 1
University of Erlangen-Nürnberg, Staudtstraße 7/B2, 91058 Erlangen, Germany  2
Max Planck Institute for the Science of Light, Staudtstraße 2, 91058 Erlangen,
Germany  3 5th Physical Institute, University of Stuttgart, Pfaffenwaldring
57, 70569 Stuttgart, Germany<EMAIL_ADDRESS>
###### Abstract

We present a new type of compact and all-glass based vapour cell integrating
hollow-core photonic crystal fibres. The absence of metals, as in a
traditional vacuum chamber and the much more compact geometry allows for fast
and homogeneous heating. As a consequence we can fill the fibres on much
faster timescales, ranging from minutes to hours. Additionally the all-glass
design ensures optical access along the fibre. This allows live monitoring of
the diffusion of rubidium atoms inside the hollow-core by measuring the
frequency-dependent fluorescence from the atoms. The atomic density is
numerically retrieved using a 5-level system of Bloch-equations.

##  1 Introduction

Hot atomic vapours are an active field of research and have great potential in
the fields of quantum optics [1, 2] quantum communication [3, 4] or metrology
[5, 6]. Optimal atom-light coupling in a vapour cell is limited by the
diffraction limit of Gaussian beams. This can be overcome by confining light
and atoms inside a hollow-core fibre. This led to many applications from
nonlinear optics in noble gases [7], to chemistry [8], and three-photons
spectroscopy of caesium vapour inside pcfs [9]. Although previous works [10,
11, 9] showed the feasibility of filling hc-pcfs with hot vapours, the
experimental setup was not optimal. The fibres were placed inside a
conventional steel vacuum chamber filled with caesium. On the one hand, the
large size of the steel chamber makes a homogeneous heating of the system not
only technically challenging but prone to the creation of cold spots, reducing
the vapour density at the position of the fibre. With this it takes
accordingly longer until the entire fibre is in equilibrium with the
background pressure and homogeneously filled. Uneven filling along the fibre
length additionally creates severe static electric fields, troublesome for
experiments with e.g. Rydberg atoms [12]. In this paper we present a new type
of vapour cell that is entirely made out of glass. This includes a new way to
mount the fibre inside the cell. These cells are not much larger than the
fibre itself and offer an extended stem. Both regions are encapsuled by an
appropriate two-zone oven, preventing condensation in the main part by keeping
the stem slightly cooler. Moreover, since our design allows full optical
access from the side, we can monitor the evolution of the atomic density
distribution along the whole fibre length through the measurement of radially
emitted fluorescence. From spatially resolved fluorescence we can deduce on
what timescale the atoms diffuse into the fibre. Finally we modelled the
frequency-dependent absorption of the light propagating inside the hollow-core
using a 5-level system to obtain a quantitative understanding of the diffusion
process. At first, we present the concept of the gas-cell and how to hold the
fibre properly. To study the dynamics of the filling process, we start in an
equilibrium situation at a certain temperature and then suddenly raise or drop
T and monitor how long it takes to undergo thermalisation.

##  2 Experimental setup

### Optical setup.

Our setup is depicted in Fig. 1a. The pump comes from a linearly polarised
Toptica DL-pro operated at $780\text{\,}\mathrm{nm}$. The pump beam is split
into two independent paths, both being sent to the gas-cell. The power of each
beam line is controlled by a combination of half waveplate and polarisation
beam-splitter cube. We use a second half waveplate to adjust the polarisation
state at the entrance of the cell. One beam is coupled to a
$60\text{\,}\mathrm{\SIUnitSymbolMicro m}$ core diameter Kagomé fibre using a
lens with $100\text{\,}\mathrm{mm}$ focal distance. A scanning electron
micrograph of the end face of the fibre is shown in Fig. 1b. The fibre is
$7.5\text{\,}\mathrm{cm}$ long. The second beam propagates freely through the
gas-cell. At the output of the cell we monitor the transmitted power of the
two beams with two photodiodes. Optical bandpass filters are inserted to
prevent ambient light from perturbing the detected signal. The glass cell
consist of a $\diameter\,\times\,L=30\,\times\,80\,\text{mm}^{2}$ tube made of
borosilicate where the optical fibre is placed. To ensure that the fibre is
kept straight, we shrunk a $5\text{\,}\mathrm{mm}$ thick borosilicate tube
around the bare optical fibre. The fibre-in-tube is not only more sturdy and
can be hold in place with two glass supports, but due to the large thickness
we are able to distinguish between fluorescence emitted in the hollow-core
fibre and the background outside of it. This background light stems from
leakage out of the fibre, non-perfect coupling and emitted fluorescence. Such
a design gives access from the side over the complete length of the fibre,
allowing side-scattering measurements. A secondary
$\diameter\,\times\,L=5\,\times\,45\,\text{mm}^{2}$ tube is connected at
$90\text{\,}\mathrm{\SIUnitSymbolDegree}$ to the main cell and acts as the
reservoir. When the cell is sealed before its first use, liquid alkali metal
is poured into the reservoir. In the current case we filled our cell with
rubidium with its natural isotope distribution. The temperature of the
reservoir and of the main cell can be addressed independently. The temperature
of the reservoir is kept lower than that of the main cell. Therefore, not only
is the atomic density given by the stem temperature but no Rb atoms will stick
to the walls of the main cell. The diffusion of the rubidium atoms from the
reservoir into the main cell is quite fast given the small dimensions and a
steady state is achieved almost instantaneously. To accelerate the diffusion
of the atoms into the fibre, we use elevated temperatures, increasing the
overall atomic density. In principle we could work up to
$300\text{\,}\mathrm{\SIUnitSymbolDegree}$ or even above, but for technical
reason we limit us to the range of 45 to
$80\text{\,}\mathrm{\SIUnitSymbolDegree}$.

Figure 1: (a) Sketch of the experimental setup. The input beam at
$780\text{\,}\mathrm{nm}$ is initially split equally by a 50:50 beam-splitter
BS. Half-wave plates (HWP) and polarisation beam splitter (PBS) ensure the
control of the power and a second HWP the input polarisation. One beam is
free-space propagating through the cell and the other is coupled into a
$60\text{\,}\mathrm{\SIUnitSymbolMicro m}$ core diameter Kagomé hollow-core
pcf shown in (b). At the output of the vapour cell, the beams are separately
focused onto photodiodes, with frequency filters in front. (c) is the all-
glass vapour cell. The temperature of the reservoir and of the main cell can
be addressed independently.

### Filling the vapour cell and the fibre.

Ensuring a constant atomic density inside the fibre is of vital interest for
experiments involving, e.g. Rydberg states. In order to monitor the filling
process, we constantly measure the spectra after our fibre and of a free-space
beam through the glass cell itself, while scanning the probe wavelength
through the complete D2 line of rubidium. We target the 85Rb $\text{F}=2$ line
and model it using a Maxwell-Bloch equation approach, including the hyperfine
states but neglecting the magnetic sublevels. In order to take into account
the small beam size inside our fibre, we include transit time broadening [13]
in our 5-level-system (85Rb; $\text{F}=2,3$; $\text{F}^{\prime}=1,2,3$, Figure
2).

Figure 2: 5-level-system used for the retrieval of the atomic density
distribution.

The evolution of the density matrix is governed by the Lindblad master
equation [14]

$i\hbar\frac{\partial\rho}{\partial t}=\left[H,\rho\right]+i\hbar
L_{D}(\rho),$ (1)

with $H$ the Hamiltonian of the system, $\rho$ the density matrix and
$L_{D}(\rho)$ the Lindblad operator. The measured spectrum corresponds to the
steady state solution of equation (1), which yields the excited state fraction
of the system. The off-diagonal elements lead to the complex valued electric
susceptibility $\chi$ and thus the absorption coefficient. To ensure absolute
calibration of the frequency-axis, part of the pump beam is guided through a
Fabry-Perot interferometer. For intensities below the saturation intensity, we
cross-checked our model to the ElecSus open source software [15] for which
length, temperature, magnetic field, light polarization, etc. can be adjusted.
Note that ElecSus does not take power broadening into account, but it is
implemented in our model. In our case, we work above saturation intensity and
have to include this in our simulations. Beside the intensity, the temperature
of the reservoir is the only fitting parameter, which yields the atomic
density via the perfect gas law.

Figure 3 shows the evolution of the atomic density of rubidium in the vapour
cell and inside the Kagomé fibre for different temperatures of the cell and of
the reservoir. At all times we make sure that the temperature of the reservoir
remains lower than that of the main cell. In the initial stage (Figure 3a) the
vapour cell is heated from room temperature to
$80\text{\,}\mathrm{\SIUnitSymbolCelsius}$ and left there for
$96\text{\,}\mathrm{h}$. During that time, the reservoir is left at ambient
temperature and only heats up due to the conduction of the oven, reaching
$45\text{\,}\mathrm{\SIUnitSymbolCelsius}$. As shown in Figure 3a, as the oven
is switched on, the density of atoms in the main cell rapidly increases (blue
line). Surprisingly, the estimation of the density of rubidium retrieved from
the spectrum at the output of the fibre overshoots the value that is expected
for such a temperature of the reservoir. We reproducibly observe this whenever
we turn on the system. One possible reason being, that even at room
temperature rubidium metal diffused into the fibre and stuck to the core wall.
When we then increase the temperature, those atoms might desorb and after
roughly $5\text{\,}\mathrm{h}$ the system has relaxed. Afterwards the density
inside the fibre increased with a time constant of $45\text{\,}\mathrm{h}$.

Figure 3: Evolution of the atomic density of rubidium in the vapour cell
(blue) and inside the Kagomé fibre (green) vs. the temperature of the cell
(black) and of the reservoir (red). The insets magnify the regions where the
atomic densities rapidly change due to a modification of the temperatures. The
green arrow indicates the apparent overshooting of the density inside the
hollow-core when the system is switched on.

In a second step – after 96 hours – we increased the reservoir temperature to
$65\text{\,}\mathrm{\SIUnitSymbolCelsius}$, while keeping the cell temperature
at $80\text{\,}\mathrm{\SIUnitSymbolCelsius}$. As seen in Figure 3b, the
average density in the cell (blue line) follows almost instantaneously the set
temperature of the reservoir. In case of our Kagomé fibre (green line), we
also observe this rapid change, but the atomic density reached an equilibrium
after $\sim$ $24\text{\,}\mathrm{h}$, with a timescale of
$9.4\text{\,}\mathrm{h}$ for the diffusion process. This should be compared to
the previous experiment using a $60\text{\,}\mathrm{\SIUnitSymbolMicro m}$
core diameter Kagomé fibre. At that time the filling process lasted $>\\!20$
days [16].

To even better understand the diffusion in/out of the fibre for an
increase/decrease of the gas pressure in the main cell, we additionally look
at the radial fluorescence along the fibre. In a simple picture, the atomic
density should be proportional to the emitted fluorescence. As a proof of
principle, we decrease the temperature of the cell, corresponding to a
reduction of the atomic density in the surrounding glass cell and causing a
density gradient between the fibre and the cell. We chose a temperature
decrease, giving us more time for a careful study of the system dynamics.
Specifically, we decreased the reservoir temperature from 65 to
$45\text{\,}\mathrm{\SIUnitSymbolCelsius}$ while keeping the cell temperature
at $80\text{\,}\mathrm{\SIUnitSymbolCelsius}$ and monitored the diffusion
process over $100\text{\,}\mathrm{h}$. While the rubidium diffuses out of the
fibre, we take a series of pictures using a 14-bit ccd camera mounted above
our glass cell. To obtain a full spectrum we scan the laser over the the D2
$\text{F}=2$ transition of 85Rb and image the glass cell every
$17.5\text{\,}\mathrm{MHz}$ of laser detuning. The input polarisation was
chosen so that maximum intensity was recorded by the CCD camera. The complete
scan takes $25\text{\,}\mathrm{s}$, which is well below the diffusion
timescale that we measured previously (Figure 3). Each image is transversally
integrated within the roi to obtain a one-dimensional intensity distribution
along the propagation. By stacking these 160 pictures for all recorded
frequencies, we obtain a two-dimensional map of the emitted fluorescence as a
function of detuning and fibre position as shown in Figure 4. The resolution
of these maps is around $160\,\times\,870$ px2 (detuning $\times$ length),
depending on the acquisition speed of the camera. On Figure 4b and c the black
lines represent the raw experimental data for two specific frequencies and the
blue lines our model.

We extract the evolution of the density inside the fibre for a specific
detuning $\Delta$ of the pump using the Beer-Lambert law. This is done in an
iterative way along the whole length of the fibre, assuming an initial
intensity $I(z=0)$ and an atomic density $\eta(z=0)$. As the light propagates
the intensity decreases according to

$\frac{\mathrm{d}I}{\mathrm{d}z}=\frac{4\pi}{\lambda}\,\mathrm{Im}\left\\{\sqrt{1+\chi\left[\eta(z),I(z),\Delta\right]}\right\\}\,I(z)$
(2)

where the susceptibility $\chi$ depends on the atomic density $\eta(z)$ and on
the light intensity $I(z)$. As we lower the temperature of the reservoir, the
density of atoms inside the cell will decrease, yielding a symmetric diffusion
of atoms out of both ends of the fibre. For simplicity, we assume that the
atomic density distribution follows

$\eta(z)=(\eta_{c}-\eta_{e})\cdot\left\\{1-\tanh^{2}\left[\sigma\cdot\left(z-\frac{L}{2}\right)\right]\right\\}+\eta_{e},$
(3)

with $L$ the fibre length, $\eta_{e}$ and $\eta_{c}$ respectively the atomic
density at both ends and at the center of the fibre. $\sigma$ is chosen so
that $\eta(0)=\eta(L)$ is within 1% of $\eta_{e}$. This approximately
corresponds to a factor of $\sigma=\frac{6}{L}$.

Figure 4: (a) Two-dimensional map of fluorescence emitted by rubidium atoms
inside a hc-pcf as a function of detuning and fibre position at
$65\text{\,}\mathrm{\SIUnitSymbolCelsius}$. Light is entering the cell from
the position ‘0’. Black indicates experimental data and blue the best fit. The
grey area marks the position of the fibre holders, which were ignored for
fitting. (b) and (c) illustrate the fluorescence along $z$ along at
respectively 0 and $250\text{\,}\mathrm{MHz}$ indicated by the dotted lines.
(d) and (e) show the spectrum at the beginning ($z=0$) and end ($z=L$) of the
fibre.

The measured fluorescence (black lines in Figure 4b, c is directly linked with
the atomic density and the excited state fraction. Combining the steady state
solution of (1) with the assumed distribution of (3), we calculate $\eta_{e}$
and $\eta_{c}$. As shown on Figure 4b and c the calculated fluorescence curves
(blue lines) along the fibre length agree very well with our experimental
data. Figure 4d and e show the fluorescence spectra at the beginning and the
end of the fibre at a given time $t$. As the diffusion process takes place, we
record a series of these maps which are processed the same way. Figure 5a
shows the temporal evolution of the atomic density at the end ($\eta_{e}$) and
in the center ($\eta_{c}$) of the fibre. It is clear that the atomic density
at the edge rapidly decreases (green curve) following the change in the cell.
The change in the center (blue curve) is much slower, because the rubidium
atoms need to diffuse out of the fibre. Like the transmission measurement
(Figure 3a), the fluorescence measurement shows that the atomic density
rapidly changes as soon as the reservoir temperature is modified (Figure 5a).
After this abrupt drop the evolution of the atomic density is much slower,
with timescales of $\tau_{c}=$36\text{\,}\mathrm{h}$$ and
$\tau_{e}=$45\text{\,}\mathrm{h}$$, when fitting with a double exponential
decay of the form

$\eta(t)=ae^{-t/\tau_{1}}+be^{-t/\tau_{2}}+c.$ (4)

Here $\tau_{c}$ respectively $\tau_{e}$ is the larger of the two lifetimes
$\tau_{1}/\tau_{2}$, corresponding to the slower diffusion timescale of the
system.

Figure 5: Retrieved atomic densities at the center (blue) and edges (green) of
a $60\text{\,}\mathrm{\SIUnitSymbolMicro m}$ Kagomé fibre when decreasing the
reservoir temperature from 65 to $45\text{\,}\mathrm{\SIUnitSymbolCelsius}$
while keeping a constant cell temperature of
$80\text{\,}\mathrm{\SIUnitSymbolCelsius}$. The grey curve is the relative
density change
$2\,\nicefrac{{\left(\eta_{c}-\eta_{e}\right)}}{{\left(\eta_{c}+\eta_{e}\right)}}$.

To conclude, we presented here an all-glass assembly integrating alkali
vapours with a hollow-core photonic crystal fibre. Its small dimensions and
homogeneous temperature permit a rapid modification of the atomic density in a
controlled way. Additionally, the all-glass design allows to perform easily
accessible measurements from the side. Measuring the fluorescence of the
rubidium allows observing in real-time and spatially resolved the diffusion of
the atoms out of the fibre when changing the atomic density inside the vapour
cell. Both transmission and fluorescence measurements confirm that the atomic
density inside the fibre has reached an equilibrium only after a few days.
This drastically contrasts with previous experiments, where hollow-core fibres
were filled with hot alkali metal in bulky metal based vacuum chambers [12].
We believe that such a cell makes an attractive tool for atomic spectroscopy,
Rydberg physics and non-linear optics. Its small size makes this type of cells
more transportable and integration in an spectroscopy setup much easier. In a
next step it would be interesting to have an all fibre based setup to avoid
free beam coupling to the fibre [17].

## Acknowledgements

This project is supported by the DFG (JO 1090/4-1, LO 1657/6-1 & RU 1426/1-1)
as part of the SPP 1929 ”Giant Interactions in Rydberg Systems (GiRyd)”. We
also thank SAOT for financial support.

## References

  * [1] Ioannis Caltzidis et al.  “Atomic Faraday beam splitter for light generated from pump-degenerate four-wave mixing in a hollow-core photonic crystal fiber” Publisher: American Physical Society  In _Physical Review A_ 103.4, 2021, pp. 043501  DOI: 10.1103/PhysRevA.103.043501
  * [2] C.. McCormick, V. Boyer, E. Arimondo and P.. Lett  “Strong relative intensity squeezing by four-wave mixing in rubidium vapor” Publisher: Optical Society of America  In _Optics Letters_ 32.2, 2007, pp. 178–180  DOI: 10.1364/OL.32.000178
  * [3] Ran Finkelstein et al.  “Fast, noise-free memory for photon synchronization at room temperature” Publisher: American Association for the Advancement of Science  In _Science Advances_ , 2018  DOI: 10.1126/sciadv.aap8598
  * [4] K.. Kaczmarek et al.  “High-speed noise-free optical quantum memory” Publisher: American Physical Society  In _Physical Review A_ 97.4, 2018, pp. 042316  DOI: 10.1103/PhysRevA.97.042316
  * [5] Dmitry Budker and Michael Romalis  “Optical magnetometry” Bandiera_abtest: a Cg_type: Nature Research Journals Number: 4 Primary_atype: Reviews Publisher: Nature Publishing Group  In _Nature Physics_ 3.4, 2007, pp. 227–234  DOI: 10.1038/nphys566
  * [6] David H. Meyer, Paul D. Kunz and Kevin C. Cox  “Waveguide-Coupled Rydberg Spectrum Analyzer from 0 to 20 GHz” Publisher: American Physical Society  In _Physical Review Applied_ 15.1, 2021, pp. 014053  DOI: 10.1103/PhysRevApplied.15.014053
  * [7] John C. Travers et al.  “Ultrafast nonlinear optics in gas-filled hollow-core photonic crystal fibers [Invited]” Publisher: Optical Society of America  In _JOSA B_ 28.12, 2011, pp. A11–A26  DOI: 10.1364/JOSAB.28.000A11
  * [8] Ana M. Cubillas et al.  “Photonic crystal fibres for chemical sensing and photochemistry” Publisher: The Royal Society of Chemistry  In _Chemical Society Reviews_ 42.22, 2013, pp. 8629–8648  DOI: 10.1039/C3CS60128E
  * [9] G. Epple et al.  “Rydberg atoms in hollow-core photonic crystal fibres” Number: 1 Publisher: Nature Publishing Group  In _Nature Communications_ 5.1, 2014, pp. 1–5  DOI: 10.1038/ncomms5132
  * [10] Saikat Ghosh et al.  “Low-Light-Level Optical Interactions with Rubidium Vapor in a Photonic Band-Gap Fiber” Publisher: American Physical Society  In _Physical Review Letters_ 97.2, 2006, pp. 023603  DOI: 10.1103/PhysRevLett.97.023603
  * [11] P.. Light et al.  “Electromagnetically induced transparency in Rb-filled coated hollow-core photonic crystal fiber” Publisher: Optical Society of America  In _Optics Letters_ 32.10, 2007, pp. 1323–1325  DOI: 10.1364/OL.32.001323
  * [12] G. Epple et al.  “Effect of stray fields on Rydberg states in hollow-core PCF probed by higher-order modes” Publisher: Optical Society of America  In _Optics Letters_ 42.17, 2017, pp. 3271–3274  DOI: 10.1364/OL.42.003271
  * [13] A Urvoy et al.  “Optical coherences and wavelength mismatch in ladder systems”  In _Journal of Physics B: Atomic, Molecular and Optical Physics_ 46.24, 2013, pp. 245001  DOI: 10.1088/0953-4075/46/24/245001
  * [14] Daniel Adam Steck  “Quantum and Atom Optics” Google-Books-ID: bc9TMwEACAAJ, 2007
  * [15] Mark A. Zentile et al.  “ElecSus: A program to calculate the electric susceptibility of an atomic ensemble”  In _Computer Physics Communications_ 189, 2015, pp. 162–174  DOI: 10.1016/j.cpc.2014.11.023
  * [16] Georg Epple  “Spectroscopy of Rydberg Atoms in Hollow-Core Photonic Crystal Fibers”, 2017
  * [17] Josephine Gutekunst et al.  “Fiber-integrated spectroscopy device for hot alkali vapor”  In _Applied Optics_ 56.21, 2017, pp. 5898  DOI: 10.1364/AO.56.005898


# Impact of Electric Vehicles Botnets on the Power Grid

Omniyah Gul M Khan1, Ehab El-Saadany21, Amr Youssef3 and Mostafa Shaaban4
1Electrical and Computer Engineering, University of Waterloo, Canada
2Advanced Power and Energy Center, EECS Department, Khalifa University, UAE
and Adjunct Professor, ECE Department, University of Waterloo, Canada
3Concordia Institue for Information Systems Engineering, Concordia University,
Canada  4Department of Electrical Engineering, American University of Sharjah,
U.A.E

###### Abstract

The increased penetration of Electric Vehicles (EVs) in the transportation
sector has increased the requirement of Fast Charging Direct Current (FCDC)
stations to meet customer’s speedy charging requirements. However, both
charging stations and EVs connection to the communication infrastructure as
well as the power grid makes it vulnerable to cyber attacks. In this paper,
the vulnerability of the EV charging process is initially studied. We then
show how a botnet of compromised EVs and FCDC stations can be utilized to
launch cyber attacks on the power grid resulting in an increase in the load at
a specific time. The effect of such attacks on the distribution network in
terms of line congestion and voltage limit violations is investigated.
Moreover, the effect of the botnet of the transmission network is also
studied. Simulation results demonstrate the possibility of line failures, and
power outage; and hence, the system’s vulnerability to cyber attacks is
established.

##  I Introduction

The increased deployment of Electric Vehicles (EVs) has increased the need for
public rapid charging stations. Currently, based on the SAE J1772 standard
[1], the following three main charging methods are used either residentially,
commercially or in public environments:

  * •

Slow Charging-Level 1: It is mainly a residential charger that supplies AC
energy to the EV on-board charger at a maximum rate of 16 amps at 120V.
Depending on the EV’s battery technology, level 1 charging takes 8 to 12 hours
to fully charge an empty battery.

  * •

Standard Charging-Level 2: It is typically found residentially or commercially
supplying AC energy to the EV on-board charger at a maximum rate of 80 amps at
240V. Depending on the EV’s battery technology, level 2 charging takes 4 to 6
hours to fully charge an empty battery.

  * •

Fast Charging-Level 3: It is typically found commercially or on highways
supplying DC energy from an off-board charger to the EV. Most common forms of
level 3 chargers currently provide 50kW charge per hour. Tesla level 3
chargers, on the other hand, provides 120kW of charge per hour.

As EVs are becoming more popular day by day, they are becoming more affordable
to middle and lower class households. However, not all EVs consumers own their
own residential property to charge their cars. Instead, many of them may live
in more dense environments that would require the installation of public level
2 or level 3 charging stations to satisfy their charging needs. Moreover, to
attract more people into driving EVs, the future would see the deployment of a
large scale fast charging Direct Current (FCDC) stations that provides speedy
charging. However, both EVs and charging stations are considered as Internet
of Things (IoT) devices due to their interconnection via the Internet. Hence,
cyber security of EVs and charging stations play an important role in their
design and integration in the electric grid. Moreover, relevant EV charging
communication standards are in its infancy and not open enough forcing
security by obscurity.

Cyber attackers can create a network of infected EVs and charging stations,
referred to as a botnet, to cause a larger effect on the power grid (A botnet
is a network of controllable devices that are remotely controlled by the
botmaster to commit cyber attacks [2]). Charging of EVs already is of concern
in terms of unconsidered increase in the load especially during peak time. An
EV botnet can cause a large effect on the grid if the attacker decides to
create a scenario utilizing the botnet to further increase drastically the
load on the grid, most probably leading to outages. The effect of such an EV
botnet attack on the distribution and transmission networks will be studied in
this paper.

Studies on cyber security of modern cars essentially started around 2010. In
[3], the external attack surface of cars was studied. Different attacks were
implemented compromising the Electronic Control Unit (ECU) of a car. Attackers
were able to utilize the vulnerable car’s external I/O interfaces to remotely
control various vehicular functions. In [4], a wireless attack was implemented
using a connected vehicle where the driver’s phone was connected to the
vehicle’s Controller Area Network (CAN). Initially, the attacker was assumed
to have acquired access to the Car Area Network (CAN) data frame to control
the vehicle. Moreover, the attacker was assumed to have published a malicious
app via which the attacker could control the driver’s phone when the driver
downloads it. Moreover, based on [5], charging station infrastructures are
open for potential cyber attacks as they are unmanned and located in remote
areas occasionally. On the other hand, [6] and [7] demonstrated that an IoT
botnet of high wattage devices provides attackers the ability to manipulate
the demand on the grid resulting in power outages and/or blackouts. Soltan et
al. [8] provide methods to prevent line failures due to such demand
manipulation. However, to the best of our knowledge, no previous work has been
done in studying the effect of a botnet made up of EVs and charging stations
on the transmission and distribution networks.

The main contributions of this paper are two-fold: $(1)$ assess the
vulnerability of the EV charging process and $(2)$ simulate a cyber attack on
the distribution network and transmission network and study its corresponding
effect on congestion, line failures, and outage.

##  II Vulnerability Analysis of EV charging

The basic characteristics that should be present in any cyber security system
are represented by the CIA triad [9] which symbolizes confidentiality,
integrity, and availability. Confidentiality is needed to ensure restricted
and authorized access to sensitive information. Integrity, on the other hand,
refers to the assurance that information is authentic and not corrupted.
Finally, availability is the guarantee that authorized users have access to
required information and services at all times. The success of EV charging,
hence, depends on the CIA of the cyber-network which can pose significant
threats to the smart grid if security vulnerabilities are not addressed.

Utilizing an EV’s physical and communication connection to a charging station,
previous studies have revealed several sources of vulnerabilities that a cyber
attacker can utilize in the process of charging an EV [10, 5, 11]. These
vulnerabilities have been summarized as follows (also see Figure 1):

  * •

When needing to refuel, an EV user communicates with the Mobility Service
Provider (MSP), that he/she has a contract with, through a mobile application
to determine the closest charging station that is available. Attacking the MSP
can allow access to personal EV driver and vehicle information, direct EV
drivers to busy charging stations, show the unavailability of desired charging
stations forcing EV owners to drive to more expensive charging options (FCDC),
or affect the billing and payment for consumed energy.

  * •

Information is exchanged between the EV and the charging station through the
connector ports of the charging cable. The State of Charge (SOC) is initially
communicated by the EV to the charging station along with the EV battery’s
voltage ranges and ampacity [12]. If the EV is infected with a malware, the
charging station could also get infected and the malware can spread to other
EVs that refuel using the same charging station in the future. Mobility of EVs
can also be used to spread the malware to other charging stations.

  * •

Attacking the charging station itself can cause the EV not to get charged to
the desired SOC, i.e. the attacker can stop charging the EV. The attacker can
also cause the EV to get charged free of charge. Also, the attacker can ignore
the ampacity limit of the EV battery and charge at a rate beyond the capacity
of the EV causing intentional damage to the EV.

  * •

The Charge Point Operator (CPO) is responsible for operating and maintaining a
collection of charging stations. When a user arrives at the charging station,
the MSP verifies the user and the CPO is then capable of starting the charging
process. Attacking the CPO can result in providing the wrong amount of charge
to the EV or even stopping the charging process without achieving the user’s
requirement,

  * •

A cyber attacker can implement a Man in the Middle attack and intercept or
even change information exchanged between the EV and the charging station, EV
and the MSP, MSP and the CPO, or the CPO and the charging station [13].

Figure 1: Cyber vulnerabilities in the EV charging process

Utilizing any of these vulnerabilities can have devastating effects. In this
paper, charging stations and EVs are compromised to form a botnet that
increases the grid load. However, attacks such as Denial of service attacks,
electricity theft and privacy infringement are not considered in this paper
[14] [15].

##  III Effect of EV Botnets on Distribution Network

To study the effect of an EV botnet on the distribution network, the IEEE 33
bus network [16], illustrated in Figure 2, is utilized. The non-flexible base
load profiles of the 32 load buses are designed to be equivalent to the
default loading of the IEEE 33 bus system [16]. All the load buses, except bus
24, are assumed to be residential, consisting of EV and Heat Pumps (HP) [17]
as flexible loads. Having the largest base load, bus 24 represents a
commercial bus consisting of an EV fast charging parking lot that could
accommodate fifty EVs. Data obtained from Toronto parking authority [18]
representing arrival and departure times of EVs in a commercial area parking
lot was used to simulate the availability of EVs in the parking lot on bus 24.
The initial state of charge (SOC) of each EV is randomized to be uniformly
distributed between $20\%$ and $30\%$ and it is assumed that the owners would
want the car to be completely charged on departure. The various parameters
needed for simulating the operation of the HP and the EV, and performing power
flow analysis are listed in Table I.

Figure 2: IEEE 33 bus system used as a distribution network case study [16] Table I: Case Study parameters Variable | Value
---|---
Coefficient of Performance (COP) | 2.2
Residential House Areas | 1500-2500 sq.ft
House Temperature Range | 18-24°C
EV battery size | 36 kWh
Maximum Residential EV charging power | 11 kW
Maximum Fast Charging EV power | 50 kW
Initial SOC of EV | 0.2-0.3
Power flow limit of branch 1-2 | 7.91 MW
Upper voltage limit | 1.05 pu
Lower voltage limit | 0.95 pu

Moreover, fast charging stations are assumed to be placed on some of the
residential buses for consumers living in multi-dwelling units lacking the
space required for home charging stations. MATPOWER [19] was used to simulate
the base case of the IEEE 33 bus network to determine the weakest point in the
network in terms of the largest voltage drop. Bus 17 followed by bus 16 were
observed to have the lowest bus voltages after performing load flow analysis.
Hence, assuming that the attacker has knowledge of the network she is
attacking, fifty FCDC stations are assumed to be located at these two buses.
Figure 3(a) illustrates the amount of power flow with and without the flexible
load in branch 0-1 of the IEEE 33 system. As observed, there is no congestion
as not all the FCDC are operational at the same instant of time. However, when
the attacker utilizes the EV botnet and directs all EVs requiring to be
charged to the FCDC charging stations at time 07:00, the demand for
electricity increases as all the FCDC stations are operational. Such an attack
is a type of load altering attack [20] which results in congestion in the grid
as observed in Figure 3(b) where the power flowing in branch 0-1 is exceeding
its capacity. Distribution Network Operator (DNO) would require to immediately
load shed or line failure of branch 0-1 occurs causing a power outage.

Figure 3: Impact of an EV botnet attack on the loading of line 0-1 where (a)
represents the normal operation power flow and (b) represents the power flow
as a result of the cyber attack

Since the FCDC points were located at the two weakest voltage points at bus 16
and 17, the effect on voltage can be clearly seen in Figure 4. Due to the
cyber attack at time 07:00, the minimum voltages at that time decreases to
below the NEC 5% voltage drop recommendation. Hence, either voltage support
methods need to be implemented in the network which would incur cost or the
voltage that is supplied to the consumer would not be at the rated voltage
causing improper, erratic or no operation of the loads and eventually damage
to the equipment. Moreover, an excess voltage drop is a concern due to the
potential fire hazard at high resistance connections.

Figure 4: Impact of an EV botnet cyber attack on the minimum bus voltages

##  IV Effect of EV Botnets on Transmission Network

A load altering attack is generally observed at the distribution network
level. However, the impact of such an attack can be comprehended by studying
the system at the transmission level. Hence, to study the effect of an EV
botnet on the transmission network, the IEEE 39 [21] bus system was used which
consists of 39 buses, 10 generators and 46 branches as illustrated in Figure
5.

Figure 5: IEEE 39 bus system used as a transmission network case study [21]

The IEEE 39 bus system has 19 load buses. To simulate a cyber attack that
utilizes an EV botnet, the load on these buses is increased by 5%. For
example, bus 20 has a 680MW base load. Hence, an increase of 5% corresponds to
an increase of 34MW which represents 680 FCDC points being operational at the
same time to charge EVs. In order not to trigger the protection system in the
distribution network, this increase in the load buses should not cause any
congestion in their respective networks. Bus 9 of the IEEE 39 bus system has a
load of 6.5MW which corresponds to the load at time 19:00 of the IEEE 33 bus
system explained in the previous section. Hence, the IEEE 33 bus system with
loads at time 19:00 were used to simulate the load on bus 9 of the IEEE 39 bus
system. Initially, in the absence of an attack, 5 FCDC points on each of the
buses 24, 17 and 16 were simulated in which EVs were being charged based on
the data obtained from the Toronto Parking Authority [18]. Load flow analysis
was performed to obtain the power flow in branch 0-1 and observe if there will
be any congestion. A cyber attack is then triggered to cause all the 5 FCDC
points to be charging at time 19:00. Figure 6 depicts the absence of
congestion even when the attack happens at 19:00 causing an increase in the
load by 6.5%. Hence, no protection system at the distribution level is
triggered.

Figure 6: Impact of an EV botnet consisting of 5 FCDC stations on bus 16, 17
and 24 on the loading of line 0-1

When the load at 19:00 is then used to study the effect of such attacks on the
transmission system, on performing load flow analysis, branch 24 linking bus
15 and 14 is observed to be overloaded. This branch is deactivated, and the
network is still complete and no isolated islands are observed to exist. Load
flow analysis is re-performed resulting in overloading branch 7 linking bus 18
and 3. This branch is also deactivated, and the network is still complete and
no isolated islands are observed to exist. Load flow analysis is repeated
resulting in overloading branches 31 linking bus 27 and 17, and 40 linking
buses 26 and 25. This causes the creation of three islands: island 1 which
consists of 23 branches, 5 generators and 20 buses. Island 2 consists of 14
buses, 4 generators and 14 buses. Island 3 consists of 5 branches, 1 generator
and 5 buses. Moreover, even though we have lines being deactivated due to
being overloaded, the system is still capable of providing all the loads.
Hence, no outage is observed.

However, increasing the load on all the load buses by 10% causes an outage.
Initially, it was ensured that this increase in the load buses does not cause
any congestion in the distribution system triggering its protection system.
The same process explained earlier was repeated utilizing the IEEE 33 bus
system as a load for bus 9 of the IEEE 39 bus system having a load of 6.5MW.
In the absence of an attack, 10 FCDC points on each of the buses 24, 17 and 16
were simulated in which EVs were being charged based on the data obtained from
the Toronto Parking Authority. Load flow analysis was performed to determine
the power flowing in branch 0-1 and observe if there will be any congestion. A
cyber attack is then triggered to cause all the 10 FCDC points to be charging
at time 19:00. Figure 7 illustrates the absence of congestion in the
distribution network even when the attack happens at 19:00 causing an increase
in the load by 11.8%. Hence, no protection system at the distribution level is
triggered.

Figure 7: Impact of an EV botnet consisting of 10 FCDC stations on bus 16, 17
and 24 on the loading of line 0-1

When the load at 19:00 is then used to study the effect of such attacks on the
transmission system, 4 branches are observed to be overloaded. These branches
are deactivated, and the network is still complete and no isolated islands are
observed to exist. Load flow analysis is re-performed resulting in the
overload of 9 branches which when deactivated causes the creation of nine
islands. This process continues resulting in 16 branches being overloaded and
an outage of 559.4MW (represents 8% of the original load) since load bus 13
and 4 are isolated and has no way to be provided. Figure 8 illustrates the
deactivated branches using dotted lines and the red boxes represents the
isolated load buses.

Figure 8: Two dead load buses formed as a result of overload in lines due to
the botnet cyber attack increasing the load in all buses by 10%

##  V Conclusions

The effect of a cyber attacker exploiting a botnet comprising of EVs and FCDC
stations to attack the grid was investigated in this paper. The vulnerability
of the EV charging process was studied initially and the IEEE 33 bus system
was used to demonstrate the effect of increasing the load using the EV botnet
on the distribution network causing line congestion and voltage limit
violations. Moreover, the attacker could increase the load such that the
distribution network’s protection system is not triggered yet the effect of
the attack is observed in the transmission network in terms of outages. The
security of EVs and FCDC stations is hence, vital for the safe and secure
operation of the power grid. Further studies investigating the effect of a
coordinated attack on multiple charging stations on the system’s voltage and
frequency needs to be performed. Moreover, preventive methods that detect and
protects the system against such attacks need to be developed.

## References

  * [1] Y. Kongjeen, W. Junlakan, K. Bhumkittipich, and M. Nadarajah, “Estimation of the quick charging station for electric vehicles based on location and population density data,” _International Journal of Intelligent Engineering and Systems_ , vol. 11, pp. 233–241, 06 2018.
  * [2] L. Cao and X. Qiu, “Defence against botnets: A formal definition and a general framework,” in _2013 IEEE Eighth International Conference on Networking, Architecture and Storage_ , July 2013, pp. 237–241.
  * [3] S. Checkoway, D. McCoy, B. Kantor, D. Anderson, H. Shacham, S. Savage, K. Koscher, A. Czeskis, F. Roesner, and T. Kohno, “Comprehensive experimental analyses of automotive attack surfaces,” in _Proceedings of the 20th USENIX Conference on Security_ , ser. SEC’11. Berkeley, CA, USA: USENIX Association, 2011.
  * [4] S. Woo, H. J. Jo, and D. H. Lee, “A practical wireless attack on the connected car and security protocol for in-vehicle can,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 16, no. 2, pp. 993–1006, April 2015.
  * [5] C. Hille and M. Allhoff, “Ev charging: Mapping out the cyber security threats and solutions for grids and charging infrastructure,” _UtiliNet Europe_ , May 2018.
  * [6] S. Soltan, P. Mittal, and H. V. Poor, “Blackiot: Iot botnet of high wattage devices can disrupt the power grid,” in _USENIX Security Symposium_ , 2018\.
  * [7] A. Dabrowski, J. Ullrich, and E. Weippl, “Grid shock: Coordinated load-changing attacks on power grids: The non-smart power grid is vulnerable to cyber attacks as well,” 12 2017, pp. 303–314.
  * [8] S. Soltan, P. Mittal, and H. V. Poor, “Protecting the grid against iot botnets of high-wattage devices,” _Computing Research Repository_ , 2018.
  * [9] M. Vai, B. Nahill, J. Kramer, M. Geis, D. Utin, D. Whelihan, and R. Khazan, “Secure architecture for embedded systems,” in _2015 IEEE High Performance Extreme Computing Conference (HPEC)_ , Sep. 2015, pp. 1–5.
  * [10] Y. Sun, L. Wu, S. Wu, S. Li, T. Zhang, L. Zhang, J. Xu, Y. Xiong, and X. Cui, “Attacks and countermeasures in the internet of vehicles,” _Annals of Telecommunications_ , vol. 72, 11 2016.
  * [11] S. Fries and R. Falk, “Electric vehicle charging infrastructure – security considerations and approaches,” in _International Conference on Evolving Internet_ , June 2012.
  * [12] Hydro-Quebec, “Electric vehicle charging stations technical installation guide,” Tech. Rep., 2015.
  * [13] K. Harnett, B. Harris, D. Chin, and G. Watso, “Doe/dhs/dot volpe technical meeting on electric vehicle and charging station cybersecurity report,” Tech. Rep., 2017.
  * [14] M. A. Mustafa, N. Zhang, G. Kalogridis, and Z. Fan, “Smart electric vehicle charging: Security analysis,” in _2013 IEEE PES Innovative Smart Grid Technologies Conference (ISGT)_ , Feb 2013, pp. 1–6.
  * [15] R. M. Pratt and T. E. Carroll, “Vehicle charging infrastructure security,” in _2019 IEEE International Conference on Consumer Electronics (ICCE)_ , Jan 2019, pp. 1–5.
  * [16] M. E. Baran and F. F. Wu, “Network reconfiguration in distribution systems for loss reduction and load balancing,” _IEEE Transactions on Power Delivery_ , vol. 4, no. 2, pp. 1401–1407, April 1989.
  * [17] W. Liu, Q. Wu, F. Wen, and J. Ostergaard, “Day-ahead congestion management in distribution systems through household demand response and distribution congestion prices,” _IEEE Transactions on Smart Grid_ , vol. 5, no. 6, Nov 2014.
  * [18] E. A. Rezai, M. Shaaban, E. El-Saadany, and F. Karray, “Demand response through interactive incorporation of plug-in electric vehicles,” in _2015 IEEE Power Energy Society General Meeting_ , 2015, pp. 1–5.
  * [19] R. D. Zimmerman, C. E. Murillo-Sánchez, and R. J. Thomas, “Matpower: Steady-state operations, planning and analysis tools for power systems research and education,” _IEEE Transactions on Power Systems_ , vol. 26, no. 1, pp. 12–19, Feb 2011.
  * [20] O. G. M. Khan, E. El-Saadany, K. Saleh, M. Shaaban, and A. Youssef, “Cyber attacks on distributed congestion management methods,” in _Accepted in 2019 IEEE PES General Meeting_ , 2019.
  * [21] T. Athay, R. Podmore, and S. Virmani, “A practical method for the direct analysis of transient stability,” _IEEE Transactions on Power Apparatus and Systems_ , vol. PAS-98, no. 2, pp. 573–584, March/April 1979.


* Levine et al. [2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  _Journal of Machine Learning Research_ , 17(1):1334–1373, 2016.
  * Lütkepohl [2005] Helmut Lütkepohl.  _New Introduction to Multiple Time Series Analysis_.  Springer Science & Business Media, 2005.
  * Meinshausen and Ridgeway [2006] Nicolai Meinshausen and Greg Ridgeway.  Quantile regression forests.  _Journal of Machine Learning Research_ , 7(6), 2006.
  * Milgrom and Segal [2002] Paul Milgrom and Ilya Segal.  Envelope theorems for arbitrary choice sets.  _Econometrica_ , 70(2):583–601, 2002.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Nass et al. [2019] David Nass, Boris Belousov, and Jan Peters.  Entropic risk measure in policy search.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1101–1106. IEEE, 2019.
  * Osband [1985] Kent Osband.  _Providing incentives for better cost forecasting_.  PhD thesis, University of California, Berkeley, 1985.
  * Peng [1997] Shige Peng.  Backward SDE and related G-expectation.  _Pitman Research Notes in Mathematics Series_ , pages 141–160, 1997\.
  * Richman [2021] Ronald Richman.  Mind the gap - Safely incorporating deep learning models into the actuarial toolkit.  _Available at SSRN 3857693_ , 2021.
  * Riedel [2004] Frank Riedel.  Dynamic coherent risk measures.  _Stochastic Processes and their Applications_ , 112(2):185–200, 2004.
  * Rockafellar et al. [2000] R Tyrrell Rockafellar, Stanislav Uryasev, et al.  Optimization of conditional value-at-risk.  _Journal of Risk_ , 2:21–42, 2000.
  * Rodrigues and Pereira [2020] Filipe Rodrigues and Francisco C Pereira.  Beyond expectation: Deep joint mean and quantile regression for spatiotemporal problems.  _IEEE Transactions on Neural Networks and Learning Systems_ , 31(12):5377–5389, 2020.
  * Ruszczyński [2010] Andrzej Ruszczyński.  Risk-averse dynamic programming for Markov decision processes.  _Mathematical Programming_ , 125(2):235–261, 2010\.
  * Saerens [2000] Marco Saerens.  Building cost functions minimizing to some summary statistics.  _IEEE Transactions on Neural Networks_ , 11(6):1263–1271, 2000.
  * Savage [1971] Leonard J Savage.  Elicitation of personal probabilities and expectations.  _Journal of the American Statistical Association_ , 66(336):783–801, 1971.
  * Shapiro et al. [2014] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyński.  _Lectures on Stochastic Programming: Modeling and Theory_.  SIAM, 2014.
  * Silver et al. [2016] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 2016.
  * Smith [2017] Leslie N Smith.  Cyclical learning rates for training neural networks.  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 464–472. IEEE, 2017.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  _Reinforcement Learning: An Introduction_.  MIT press, 2018.
  * Sutton et al. [2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In _Advances in Neural Information Processing Systems_ , pages 1057–1063, 2000.
  * Takeuchi et al. [2006] Ichiro Takeuchi, Quoc Le, Timothy Sears, Alexander Smola, et al.  Nonparametric quantile estimation.  2006\.
  * Tamar et al. [2016] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor.  Sequential decision making with coherent risk.  _IEEE Transactions on Automatic Control_ , 62(7):3323–3338, 2016.
  * Thomson [1978] William Thomson.  Eliciting production possibilities from a well-informed manager.  1978\.
  * Van Hasselt et al. [2016] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double Q-learning.  In _Proceedings of the 30th AAAI Conference on Artificial Intelligence_ , 2016.
  * Weber [2006] Stefan Weber.  Distribution-invariant risk measures, information, and dynamic consistency.  _Mathematical Finance: An International Journal of Mathematics, Statistics and Financial Economics_ , 16(2):419–441, 2006\.
  * Zhang et al. [2019] Zihao Zhang, Stefan Zohren, and Stephen Roberts.  Extending deep learning models for limit order books to quantile regression.  _arXiv preprint arXiv:1906.04404_ , 2019.
  * Ziegel [2016] Johanna F Ziegel.  Coherence and elicitability.  _Mathematical Finance_ , 26(4):901–918, 2016\.

##  A Hyperparameters

In this section, we expand on the hyperparameters of our RL algorithm used in
the different examples of Section 7. We emphasise that the choices of
hyperparameters, optimisation rules and learning rate schedulers depend on the
specific experiment, and one may tune them to accelerate the learning process.
We also refer the reader to our Github repository RL-ElicitableDynamicRisk for
further information on the code implementation and architecture.

###  A.1 Statistical Arbitrage Example

For each CVaR with threshold $\alpha$, our actor-critic uses the following
hyperparameters during the training phase. All ANNs – i.e. the policy
$\pi^{\theta}$, the VaR $H_{1}^{\psi_{1}}$, and the difference between CVaR
and VaR $H_{2}^{\psi_{2}}$ – consist of five layers of 16 hidden nodes each
with SiLU activation functions. $\pi^{\theta}$ has a linear transformation of
a sigmoid activation function that maps to
$(-a_{\operatorname*{\mathrm{min}\vphantom{\mathrm{infsup}}}},a_{\operatorname*{\mathrm{max}\vphantom{\mathrm{infsup}}}})$
for its output layer, $H_{1}^{\psi_{1}}$ has no activation function, and
$H_{2}^{\psi_{2}}$ has a softplus activation function to ensure the outputs
are non-negative. The learning rate for $\pi^{\theta}$ initially starts at
$4\times 10^{-3}$, and decays by 0.95 every 50 epochs until it reaches
$5\times 10^{-4}$. The learning rates for $H_{1}^{\psi_{1}}$ and
$H_{2}^{\psi_{2}}$ are of the order of $5\times 10^{-3}$ and decay by 0.95
every 100 epochs.

The critic procedure – i.e. estimation of the value function – is performed
for 1,000 epochs, where the target networks are replaced every 400 epochs, and
it uses mini-batches of 750 full episodes. The actor procedure – i.e. update
of the policy – is performed for 30 epochs with mini-batches of
$500/(1-\alpha)$ full episodes. We execute the whole algorithm for 1,500
iterations, where a single iteration corresponds to both the critic and actor
procedures.

We train the models on the Niagara servers, managed by the Digital Research
Alliance of Canada. Every Python script runs on a single CPU node composed of
40 cores, i.e. two sockets with 20 Intel Skylake cores at 2.4GHz. The training
times for the elicitable and nested simulation approaches are approximately 5
and 9 hours, respectively.

###  A.2 Portfolio Allocation Example

The ANN $\pi^{\theta}$ giving the mean of the Gaussian distribution consists
of five layers of 16 hidden nodes each with SiLU activation functions, a
softplus output activation function, and a learning rate that starts at
$5\times 10^{-3}$ and decays by 0.97 every 100 epochs until it reaches
$3\times 10^{-4}$. The ANNs $H_{1}^{\psi_{1}},H_{2}^{\psi_{2}}$ characterising
the value function are both composed of five layers of 16 hidden nodes each
with SiLU activation functions – $H_{1}^{\psi_{1}}$ has no activation
function, while $H_{2}^{\psi_{2}}$ has a softplus activation function. Their
learning rates initially start at $5\times 10^{-3}$ and decay by 0.95 every 50
epochs.

For the experiments where price dynamics are given by either one of the SDEs
in Eqs. 36a and 36b, the actor-critic algorithm runs for 2,000 iterations,
alternating between the critic procedure – i.e. estimation of the value
function for 1,000 epochs with mini-batches of 1,000 full episodes, where the
target networks are replaced every 300 epochs – and the actor procedure – i.e.
update of the policy for 10 epochs with mini-batches of 1,000$/(1-\alpha)$
full episodes. For the experiments where price dynamics are simulated from the
VECM, the actor-critic algorithm instead runs for 4,000 iterations, and we
increase the number of epochs for the critic procedure to 2,000 epochs. The
results shown in Fig. 7 are obtained after approximately 24 hours of training
on the Niagara servers.

##  B Parameter Estimates of the VECM

In this section, we provide a brief overview of vector error correction models
(VECMs), and show the parameter estimates obtained with our dataset for
replication purposes. For more information on VECMs, we refer the reader to
Chapter 7 of Lütkepohl [2005] and the code from the Python library named
statsmodels.tsa.vector_ar.vecm.

In our case, the VECM has the form

$\Delta Y_{\tau}=\Pi Y_{\tau-1}+\Gamma_{1}\Delta
Y_{\tau-1}+\cdots+\Gamma_{k_{ar}-1}\Delta Y_{\tau-
k_{ar}+1}+CD_{\tau}+u_{\tau},$

where $Y_{\tau}$ is the $d$-dimensional process of interest, $\Pi$ a $(d\times
d)$ matrix of rank $r$, $k_{ar}$ the number of lagged differences, $C$ the
parameter estimates of the deterministic terms outside the co-integration
relation $D_{\tau}$, and $u_{\tau}$ a $d$-dimensional white noise with
covariance matrix $\Sigma_{u}$. Without loss of generality, we order
alphabetically the asset prices in the random vector $Y_{\tau}$, i.e. the
$d=8$ dimensions correspond respectively to assets AAL, AMZN, CCL, FB, IBM,
INTC, LYFT and OXY.

When fitting this model on a normalised version of the stock tickers with the
transformation $Y_{\tau}^{(i)}=\log(S_{\tau}^{(i)}/S_{0}^{(i)})$, we estimate
$\hat{k}_{ar}=0$ (which implies $\hat{\Gamma}=\mathbf{0}$) and $\hat{r}=2$
using the Bayesian information criterion (BIC). We also have the following
parameter estimates:

$\displaystyle\hat{\Pi}$
$\displaystyle=\begin{bmatrix}-9.1&-4.0&5.7&6.6&-6.6&-0.2&2.7&0.2\\\
1.4&0.7&-0.6&-2.0&1.4&-0.4&-0.3&-0.3\\\ -3.7&-2.3&0.2&8.7&-4.6&2.5&0.1&2.0\\\
2.1&1.1&-0.6&-3.6&2.2&-0.9&-0.3&-0.7\\\
-1.4&-0.5&1.3&0.1&-0.7&-0.5&0.6&-0.3\\\ -0.6&0.2&2.0&-4.2&1.1&-2.0&0.9&-1.5\\\
-1.7&-0.9&0.8&2.2&-1.5&0.4&0.4&0.3\\\ -1.6&-0.3&2.4&-3.0&0.2&-1.8&1.2&-1.3\\\
\end{bmatrix}\times 10^{-2},$ $\displaystyle\hat{\Sigma}_{u}$
$\displaystyle=\begin{bmatrix}18.4&1.1&15.7&2.0&3.5&3.0&7.7&9.4\\\
1.1&3.4&1.8&2.6&1.2&2.1&1.8&1.6\\\ 15.7&1.8&26.2&3.5&4.5&4.8&12.2&13.9\\\
2.0&2.6&3.5&4.9&1.6&2.7&2.9&2.9\\\ 3.5&1.2&4.5&1.6&3.7&2.6&2.8&4.5\\\
3.0&2.1&4.8&2.7&2.6&6.3&3.4&4.4\\\ 7.7&1.8&12.2&2.9&2.8&3.47&16.8&8.3\\\
9.4&1.6&13.9&2.9&4.5&4.4&8.3&26.8\\\ \end{bmatrix}\times 10^{-4},$
$\displaystyle\hat{C}$
$\displaystyle=\begin{bmatrix}-1.3&1.5&-7.6&2.9&1.0&5.2&-1.3&4.6\\\
\end{bmatrix}\times 10^{-5}.$