:
LEAVE UNSET 2023 LEAVE UNSET LEAVE UNSET Machine Learning for Health (ML4H)
2023

# Autoregressive Language Models For Estimating the Entropy of Epic EHR Audit
Logs

Benjamin C. Warner<EMAIL_ADDRESS>
Thomas Kannampallil<EMAIL_ADDRESS>
Seunghwan Kim<EMAIL_ADDRESS>
Washington University in St. Louis

###### Abstract

EHR audit logs are a highly granular stream of events that capture clinician
activities, and is a significant area of interest for research in
characterizing clinician workflow on the electronic health record (EHR).
Existing techniques to measure the complexity of workflow through EHR audit
logs (audit logs) involve time- or frequency-based cross-sectional
aggregations that are unable to capture the full complexity of a EHR session.
We briefly evaluate the usage of transformer-based tabular language model
(tabular LM) in measuring the entropy or disorderedness of action sequences
within workflow and release the evaluated models publicly.

###### keywords:

Epic EHR audit logs, clinical workflow, cross-entropy, tabular transformers

##  1 Introduction

Modern day clinical work relies heavily on documentation on EHRs (Jha, 2010).
Raw EHR audit logs are trails of clinician activity on the EHR and capture a
fine-grained representation of electronic interaction within the clinical
workflow. However, the high dimensionality and lack of context in EHR audit
logs makes it challenging to use it to effectively evaluate the complexity of
clinical workflow and work activities. Current methods for quantifying
clinical workflow complexities using audit logs tend to be limited to time-
based or frequency-based measure of action events (Kannampallil and Adler-
Milstein, 2023), which does not take into account for the underlying temporal
relationships between EHR actions and the sequential nature of EHR-based
interactions.

Being able to quantify the disorderedness in audit log action sequences could
be useful in accurately characterizing the complexity of clinical workflow. To
the best of our knowledge, there is no standardized technique currently
available for doing so. To measure the disorder or entropy of actions captured
in EHR audit logs we trained several tabular LMs and use them to estimate the
cross-entropy of each action event as a proxy of action entropy in an audit
log action sequence.

###  1.1 Contributions

Our contributions are the following:

  * •

Publicly-available autoregressive language models for evaluating the entropy
of user interaction sequences using EHR audit logs.

  * •

A comparison and discussion of different language models architectures for
assessing audit logs.

  * •

An evaluation of the cross-entropy of an audit log sequence as a proxy for
measuring workflow complexity.

We make our code available on GitHub here: https://github.com/bcwarner/audit-
log-lm.

Model weights are available on Hugging Face and reachable through our GitHub.

##  2 Related Work

###  2.1 EHR audit logs

As mandated by the Health Insurance Portability and Accountability Act
(HIPAA), all EHR-based activities are recorded to monitor access to protected
patient health information. Raw EHR audit logs are user-centric time-series
stream of action events resulting from click-level user interaction on the
EHR, and captures a broad range of actions and their data (Adler-Milstein et
al., 2020). Each row of audit logs represents an action event. Such unique
characteristics of EHR audit logs allows for studying clinical workflow
processes at scale. Currently, there are no standard metrics for assessing
workflow and work activities from audit logs and considerable effort is often
expended on designing, computing, and validating a myriad of different
measures (Kannampallil and Adler-Milstein, 2023). However, most currently
proposed metrics regarding EHR use based on cross-sectional aggregation of
individual action events within a sequence and limited work has been done to
incorporate inter-event transitions and relationships to more accurately
quantify the workflow complexity.

Previous audit log analyses that attempted to incorporate such sequential
characteristics of EHR-based interactions have utilized language models (LMs).
Such studies leverage training word2vec (Mikolov et al., 2013) on action
events, and then using the embeddings for various downstream tasks such as
clustering (Lou et al., 2023; Jones et al., 2020) or in supervised learning of
physician states such as burnout (Liu et al., 2022).

###  2.2 Entropy

Shannon (1948) introduced the measure of entropy for measuring the number of
bits required to represent a given state. The measure of entropy is dependent
on the probability function $P$, whose true behavior is unknown. For this
reason, we generally evaluate cross-entropy, which evaluates a model $\hat{P}$
of the probability function. Cross-entropy has the useful property that it is
an upper-bound for true entropy, and decreases monotonically as $\hat{P}$
improves. Entropy has generally been used to represent the amount of surprise
or disorder, as lower probability states will have higher entropy. In natural
language processing tasks, cross-entropy is often reported through perplexity,
which is the exponentiation of cross-entropy, as it represents the geometric
average of the number of possible outcomes of $\hat{P}$ (Jurafsky and Martin,
2023).

Using raw EHR audit log data, cross-entropy calculated for each EHR-based
action can be used as proxy of representing complexity of clinical workflow.
We expect higher entropy states to have higher action sequence complexity, as
action sequences that deviate from commonly-used sequences represent greater
disorder within the workflow. Conversely, more common action sequences will
have lower complexity and hence, less entropy.

###  2.3 Language model

Language models (LMs)represent a model $P$ of words or characters in a given
language, or at a more fundamental level, a model of a sequence. Language
models can be divided into two broad classes: autoregressive and
bidirectional. In the autoregressive case, LMs attempt to learn
$P(x_{i}|x_{i-k}...x_{i-1})$, the next word in a sequence given a context of
up to $k$ words (Radford et al., 2018), and use cross-entropy as a training
objective. A bidirectional model attempts to predict words from both preceding
and succeeding tokens (Devlin et al., 2018). We focus primarily on
autoregresssive language models here since our principal interest is to use
these models for calculating cross-entropy directly.

Most recent LMs are based upon the transformer architecture introduced in
Vaswani et al. (2017), and many of these variations attempt to address
different architectural issues. We will examine three different architectures:
GPT-2 (Radford et al., 2018), RWKV (Peng et al., 2023), and LLaMA (Touvron et
al., 2023). We choose to evaluate GPT-2 for its generality and simplicity,
RWKV for its linearized attention mechanism, and LLaMA for its recency in
architectural advancements.

###  2.4 Tabular language models

Tabular language models are LMs that are trained to predict tabular time-
series data. This is done by converting each entry $x_{i,j}$ of a table into a
sequence of tokens $t_{1,1},...,t_{1,n},t_{2,1},...,t_{m,n}$, and then
training or inferring from the tokenized data. Each field in a tabular dataset
has its own field vocab, which can be concatenated together to form a global
vocab. To evaluate the loss for a tabular LM, we take the cross-entropy over
the logits for each field vocab, and then reduce all field losses together
(Padhi et al., 2021).

This vocabulary scheme mandates that all fields have a finite set of tokens
from which to choose from, and requires that fields that are not finite,
namely continuous fields, to be processed. The approach Padhi et al. (2021)
propose is to quantize the values, where bins are computed for a given field
and continuous values are then mapped to these bins. An alternative approach
that Solatorio and Dupriez (2023) propose is to serialize and partition the
formatted string value into finite tokens, which may require multiple columns.

##  3 Methodology

###  3.1 Preprocessing

The dataset consisted of retrospectively collected all raw audit log events of
clinicians in 2019, for those who worked across four surgical intensive care
units (ICUs) at least once at Barnes-Jewish Hospital, a large academic medical
center in St. Louis, MO, USA. This study was approved by the institutional
review board of Washington University (IRB# 202009032) with a waiver of
informed consent.

Audit logs contain many features, and we restrict the LMs discussed to predict
only three fields: METRIC_NAME, ACCESS_TIME, PAT_ID. METRIC_NAME is one of
$4,037$ English-level descriptions of the action performed (e.g. “Radiology
Front Desk Orders Report accessed”). PAT_ID is a unique identifier for the
patient that was interacted with at a given action if present. ACCESS_TIME
represents the date and time of an event with second-level precision.

To prepare the audit log data for tokenization, several preprocessing steps
are performed. There are instances where a single EHR-based action triggers
multiple action events to be recorded in the audit log with an identical
ACCESS_TIME timestamp. To account for those, we further sort by the
ACCESS_INSTANT field, which has sub-second precision and lower super-second
accuracy. Upon sorting, we convert the ACCESS_TIME feature to time-deltas.
Audit logs are then subdivided into demarcated shifts of work, with
ACCESS_TIME zeroed after a predetermined gap. We use a gap of 6 hours of
inactivity to split audit logs into shifts.

We then convert PAT_ID into identifiers indicating their appearance in a
shift. To ensure that the field vocab for the PAT_ID is finite, we limit the
patient count to 128 per shift.

After splitting the data into shifts, we then split the audit log data into
sessions, which Ouyang et al. (2016) define as periods of activity with gaps
no greater than 5 minutes. Each of these sessions represents an EHR-based
interaction sequence of related clinician work behaviors in a clinical
setting.

Time-deltas are then quantized into a series of logarithmically-spaced bins of
5 intervals ranging from 0 to 240 seconds. We use logarithmic spacing as this
will capture the difference between shorter and longer actions, while
minimizing the number of bins the LM must learn. After this, we tokenize each
field. Sessions longer than the sequence length supported by the model are
divided into chunks of the appropriate length.

###  3.2 Training

Our training strategy is to randomly stratify by individual clinicians and
then shuffle examples in each set, with 114 clinicians appearing in the
training set and 24 clinicians appearing in both the test and validation set.
With the aforementioned preprocessing strategy, there are 398,412 training
examples. We train with a batch size of 2 and use 4 gradient accumulation
steps for 5 epochs on all models, and where available we use TensorFloat-32
operations. We optimize the LMs discussed here using Sophia, which has been
shown experimentally to be significantly faster than Adam (Liu et al., 2023).

Figure 1: Training loss, with exponentially weighted-smoothing
($\alpha=0.01$).

The training loss for our tested models (with exponentially weighted-smoothing
for visibility) can be seen above in fig:train_loss. We train for 5 epochs,
although it appears that 1-2 epochs may be enough to reach convergence.

##  4 Results

###  4.1 Audit Log Generation Performance

As there is no literature evaluating generative model performance for audit
logs, we suggest the usage of two categories of metrics: next-action
prediction accuracy and ROUGE scores (Lin, 2004). We evaluate both of these
tasks with contrastive search using the top 5 candidates, except for RWKV for
which we use sample search.

Table 1: Accuracy by model for predicting the next action on the test set. (M = METRIC_NAME, P = PAT_ID, A = ACCESS_TIME) Model | M | P | A | All
---|---|---|---|---
gpt2-25.3M | 0.274 | 0.130 | 0.527 | 0.016
gpt2-46.5M | 0.266 | 0.114 | 0.357 | 0.018
gpt2-89.0M | 0.225 | 0.075 | 0.372 | 0.007
gpt2-131.6M | 0.293 | 0.122 | 0.489 | 0.014
rwkv-65.7M | 0.136 | 0.050 | 0.200 | 0.005
rwkv-127.2M | 0.011 | 0.023 | 0.031 | 0.001
llama-58.1M | 0.177 | 0.085 | 0.337 | 0.012
llama-112.0M | 0.156 | 0.039 | 0.407 | 0.012
llama-219.8M | 0.174 | 0.038 | 0.381 | 0.001

We evaluate next-action accuracy with a randomly sampled subset of sessions
from the test set, which total around 82,685 audit log events. For each audit
log event in the session beyond the first event, we use all preceding context
to generate per-event predictions. We evaluate the accuracy of each feature,
as well as the ability to generate correct predictions for all features, in
tab:next_action_results.

We believe that ROUGE-1 is an appropriate metric since we wish to evaluate the
recall performance of specific events. As with accuracy, we evaluate ROUGE-1
performance per-field and over the entire sequence, seen in tab:rouge_scores.
For evaluation, we use 50% of the input to generate predictions and use the
remaining 50% as a reference. We tested ROUGE-1 and ROUGE-L, and find that
they are identical, possibly because the generated and reference sequences
have no common order.

Table 2: ROUGE-1 scores with 5,000 randomly sampled test sessions. (M = METRIC_NAME, P = PAT_ID, A = ACCESS_TIME) Model | M | P | A | All
---|---|---|---|---
gpt2-25.3M | 0.168 | 0.159 | 0.463 | 0.264
gpt2-46.5M | 0.249 | 0.088 | 0.319 | 0.219
gpt2-89.0M | 0.254 | 0.072 | 0.333 | 0.220
gpt2-131.6M | 0.213 | 0.095 | 0.337 | 0.215
rwkv-65.7M | 0.042 | 0.051 | 0.254 | 0.116
rwkv-127.2M | 0.003 | 0.075 | 0.009 | 0.029
llama-58.1M | 0.045 | 0.064 | 0.169 | 0.093
llama-112.0M | 0.059 | 0.019 | 0.448 | 0.175
llama-219.8M | 0.165 | 0.042 | 0.357 | 0.188

###  4.2 Per-Feature Perplexity

We evaluate the per-feature cross-entropy performance on the validation and
test sets. The results from each field are shown in fig:ppl_features, and
reported as perplexity for viewability. As expected, we find that METRIC_NAME
has the highest perplexity, and the LLaMA models appear to perform the best
overall.

Figure 2: Perplexity of features by model.

This suggests that an upper bound for the average number of possible
successive METRIC_NAME is approximately $4.3230$, which is remarkable given
the $4,037$ distinct possibilities in the METRIC_NAME vocabulary.

###  4.3 Cross-entropy as a Measure of Workflow Complexity

We include a few brief examples of the primary motivation behind the
development of our audit log LMs, which is to measure the complexity in
clinician workflow. We include an example of short sequence with moderate
cross-entropy and pairs of repeating elements, a sequence with many repeating
elements, and high entropy sequence in apd:examples. It should be noted that
the ACCESS_TIME listed is the upper-bound of the quantized bin, and the PAT_ID
is its unique identifier within its parent shift.

In the example shown in tab:average_ent, we have two repeating sets of
actions, “Notes viewed” and ”Inpatient Patient Lists list loaded.” In both
cases, there is decreasing entropy in the latter element of the pairs. This is
particularly true with the second instance of “Notes viewed,” which we believe
is reflective of the fact that clinicians will often look through multiple
clinical notes for a patient before they move onto the next action step in the
care sequence.

In tab:repeat_ent, we present an example with not only a higher number of
repeating elements, but also several system-generated automated events and
unexpected events. For example, when “Inpatient Patient Lists list loaded”
occurs it is always followed by a “Inpatient system list accessed” event,
which is reflected in the low cross-entropy of the third row in
tab:repeat_ent. We also see many repeated elements of “Inpatient system list
accessed” and “In Basket message viewed,” and when their cycles are broken
with “In Basket folder loaded,” “Cosign clinical note,” and “View FastNote
activity,” we see a significant increase in the cross-entropy, indicating that
they are essentially unexpected events.

The example in tab:high_ent is a case with high row cross-entropy. We believe
that the entropy is high here because of the fact the session is taking place
through Epic Haiku, which is a mobile client with its own subset of
METRIC_NAME, and so the actions performed have a much lower likelihood overall
and more unstructured workflow compared to typical EHR usage on a desktop
workstation.

##  5 Discussion

We find that audit logs can be modeled with considerable effectiveness using
LMs and that these LMs can be used to measure the cross-entropy of audit logs,
particularly in relative comparisons. Although there are many limitations to
the tabular LM approach used, these models provide an initial insight into
measuring the disorderedness in audit logs, which may be applied to quantify
the complexity of clinical workflow. While the generative performance is
considerably weaker than in natural language, we believe this is due to the
inherent structure of audit logs, as they do not have all context of a
clinical workflow, contain much more repetition, and have a more rigid
structure than natural language. This may be part of the reason we find that
higher parameter counts do not always translate into higher performance, as
has generally been seen in other LMs (Kaplan et al., 2020).

###  5.1 Limitations

A major design limitation of these models is the use of quantization, as the
number of bins is effectively limited by the imbalanced number of examples,
and requires significant manual tuning to determine the appropriate number of
bins.

Another limitation of the LMs presented here is that we train on a private
dataset of ICU clinicians, which limits the transferability of the model to
other types of clinicians with different specialties and roles, as some types
of workflows may not appear at all in the training set. Future models should
include a more diverse set of clinical specialties. Furthermore, we treat each
session of EHR-based work independently, under the assumption that clinicians
who practice in similar specialty and settings have similar workflow, and
potential individual differences in EHR interaction styles are not considered
as a result.

Finally, the audit logs used in this study captures all EHR-based work during
a fixed period of time from a set of clinicians who are assumed to perform
similar workflow. However, there may be inherent heterogeneity of workflow
being introduced from different clinician job levels or work locations, which
were not explicitly addressed during this study.

###  5.2 Future Work

The validation of cross-entropy as a measure of workflow complexity, either
against existing proxies of characterizing workflow or through an expert
manual review on a sample of action event sequences, will be a significant
next step in this area.

Furthermore, training separate models after identifying different subgroups
within the audit logs may allow us to capture the non-routineness of
clinicians’ EHR-based actions with better accuracy, which in turn can be
better reflective description of the clinician workflow.

Although our original intent was to measure the cross-entropy of audit logs,
we expect these tabular LMs can be effectively fine-tuned for a variety of
different supervised tasks involving clinical workflow. Future work should
also focus on improvements in the architectural design, especially with
respect to the quantization of the PAT_ID and ACCESS_TIME field vocabs; and on
a clinician-centric training approach for a customized assessment of the
effect of divergent behavioral patterns in evaluating work efficiencies at the
individual clinician level.

This work was made possible with the support of grants 1R25LM014224-01 and
R01LM013778 from the National Library of Medicine.

## References

  * Adler-Milstein et al. (2020) Julia Adler-Milstein, Jason S Adelman, Ming Tai-Seale, Vimla L Patel, and Chris Dymek.  Ehr audit logs: a new goldmine for health services research?  _Journal of biomedical informatics_ , 101:103343, 2020.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Jha (2010) Ashish K Jha.  Meaningful use of electronic health records: the road ahead.  _JAMA_ , 304(15):1709–1710, 2010.
  * Jones et al. (2020) Barrett Jones, Xinmeng Zhang, Bradley A Malin, and You Chen.  Learning tasks of pediatric providers from electronic health record audit logs.  In _AMIA Annual Symposium Proceedings_ , volume 2020, page 612. American Medical Informatics Association, 2020.
  * Jurafsky and Martin (2023) Daniel Jurafsky and James H Martin.  Speech and language processing.  2023\.
  * Kannampallil and Adler-Milstein (2023) Thomas Kannampallil and Julia Adler-Milstein.  Using electronic health record audit log data for research: insights from early efforts.  _Journal of the American Medical Informatics Association_ , 30(1):167–171, 2023.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Lin (2004) Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W04-1013.
  * Liu et al. (2022) Hanyang Liu, Sunny S Lou, Benjamin C Warner, Derek R Harford, Thomas Kannampallil, and Chenyang Lu.  Hipal: A deep framework for physician burnout prediction using activity logs in electronic health records.  _arXiv preprint arXiv:2205.11680_ , 2022.
  * Liu et al. (2023) Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma.  Sophia: A scalable stochastic second-order optimizer for language model pre-training.  _arXiv preprint arXiv:2305.14342_ , 2023.
  * Lou et al. (2023) Sunny S Lou, Hanyang Liu, Derek Harford, Chenyang Lu, and Thomas Kannampallil.  Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach.  _Journal of the American Medical Informatics Association_ , 30(3):539–544, 2023.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space, 2013.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.  Model cards for model reporting.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pages 220–229, 2019.
  * Ouyang et al. (2016) David Ouyang, Jonathan H Chen, Jason Hom, and Jeffrey Chi.  Internal medicine resident computer usage: an electronic audit of an inpatient service.  _JAMA internal medicine_ , 176(2):252–254, 2016\.
  * Padhi et al. (2021) Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman.  Tabular transformers for modeling multivariate time series.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 3565–3569. IEEE, 2021.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al.  Rwkv: Reinventing rnns for the transformer era.  _arXiv preprint arXiv:2305.13048_ , 2023.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Shannon (1948) Claude Elwood Shannon.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423, 1948.
  * Solatorio and Dupriez (2023) Aivin V Solatorio and Olivier Dupriez.  Realtabformer: Generating realistic relational and tabular data using transformers.  _arXiv preprint arXiv:2302.02041_ , 2023.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.

##  Appendix A Examples

We highlight three different examples of a session of audit log, as well as
their per-row cross-entropies calculated using llama-112.0M using all
preceding context. The first example in tab:average_ent is a randomly selected
example with average cross-entropy near 1 and several repeating elements. The
next example in tab:repeat_ent is a session with many repeating and automated
events. Finally, we include a session with high cross-entropy in tab:high_ent.

METRIC_NAME | PAT_ID | ACCESS_TIME | Row Entropy
---|---|---|---
Inpatient Patient Lists list loaded | -1 |  $\leq$ 1 | -
Results Review accessed | 2 | 3.936 | 1.766
Notes viewed | 2 | 15.492 | 1.247
Notes viewed | 2 | 3.936 | 0.266
Notes viewed | 2 | 3.936 | 0.222
Results Review exited | 2 | 15.492 | 1.249
Results Review exited | 1 |  $\leq$ 1 | 1.249
Table 3: An example of a session with cross-entropy near 1 and several repeating elements. METRIC_NAME | PAT_ID | ACCESS_TIME | Row Entropy
---|---|---|---
Inpatient Patient Lists list loaded | -1 |  $\leq$ 1 | -
Inpatient system list accessed | -1 |  $\leq$ 1 | 0.111
Inpatient system list accessed | -1 |  $\leq$ 1 | 0.289
Inpatient system list accessed | -1 |  $\leq$ 1 | 0.179
In Basket folder loaded | -1 | 15.492 | 1.332
In Basket folder loaded | -1 |  $\leq$ 1 | 0.886
In Basket message viewed | 3 | 3.936 | 0.068
View FastNote activity | 3 | 3.936 | 1.844
Cosign clinical note with attestation | 3 | 15.492 | 1.544
Problem List accessed | 3 |  $\leq$ 1 | 1.353
In Basket message viewed | 3 | 3.936 | 0.782
In Basket message viewed | 3 | 3.936 | 0.269
In Basket message viewed | 15 | 3.936 | 0.121
Cosign clinical note | 15 | 3.936 | 0.785
In Basket message viewed | 15 |  $\leq$ 1 | 0.052
In Basket message viewed | 0 |  $\leq$ 1 | 0.220
Cosign clinical note | 0 | 3.936 | 0.512
In Basket message viewed | 0 |  $\leq$ 1 | 0.051
In Basket message viewed | 3 |  $\leq$ 1 | 0.535
Table 4: An example of a session with highly cyclical events, as well as interrupting events. METRIC_NAME | PAT_ID | ACCESS_TIME | Row Entropy
---|---|---|---
Haiku login | -1 |  $\leq$ 1 | -
Inpatient system list accessed | -1 | 3.936 | 1.232
Report viewed | 116 | 3.936 | 4.703
Results List viewed | 116 | 3.936 | 5.167
Report viewed | 116 | 3.936 | 5.041
Report viewed | 116 | 3.936 | 5.377
Results List viewed | 116 | 3.936 | 4.921
Report viewed | 116 | 3.936 | 4.033
Report viewed | 116 | 3.936 | 5.724
Results table viewed | 116 | 3.936 | 5.831
Results table viewed | 116 | 3.936 | 5.121
Haiku login | -1 | 15.492 | 2.508
Table 5: An example of a session with some high cross-entropy elements.

##  Appendix B Model Card

tab:model_card contains a model card (Mitchell et al., 2019; Touvron et al.,
2023; Anil et al., 2023) with information for users who wish to utilize one of
our models.

Model Details
---
Model Developers |  Washington University in St. Louis
Variations |  GPT-2 (25.3M, 46.5M, 89.0M, 131.6M), RWKV (65.7M, 113.5M), LLaMA (58.1M, 112.0M, 219.8M)
Input |  Epic EHR audit log data, up to 1024 tokens (341 complete rows)
Output |  Tokens representing generated Epic EHR audit log events in the METRIC_NAME, PAT_ID, and ACCESS_TIME columns.
Model Architecture |  All GPT-2: 6 attention heads; gpt2-25.3M: 3 layers; gpt2-46.5M: 6 layers; gpt2-89.0M: 12 layers; gpt2-131.6M: 18 layers; All RWKV: 512 attention hidden size, 2048 intermediate size; rwkv-65.7M: 18 layers; rwkv-127.2M: 36 layers; All LLaMA: 32 attention heads, 512 hidden size; llama-58.1M: 3 layers; llama-112.0M: 6 layers; llama-219.8M: 12 layers
Model Dates |  These models were trained in September and October 2023.
Status |  This model was trained with a private dataset of ICU clinician audit logs. We do not anticipate releasing updates to our model, however further studies involving different clinician types may result in new models.
License |  Software: Apache 2.0, Model Weights: Apache 2.0
Where to send comments |  Up-to-date instructions for leaving comments can will be in the README of our GitHub repo at https://github.com/bcwarner/audit-log-lm
Intended Use
Intended Use Cases |  Our models are intended for research use with Epic EHR audit logs, either for generative modeling or for cross-entropy measurement.
Out-of-Scope Usage |  Because of the experimental nature of these models, we do not recommend the usage of our models in a non-research context.
Hardware and Software
Training |  We used Hugging Face transformers (Wolf et al., 2019) for the model architectures atop PyTorch (Paszke et al., 2019) and PyTorch Lightning, and we use Sophia to optimize (Liu et al., 2023). We trained on a mixture of NVIDIA A40s, 80GB A100s, and GeForce RTX 2080s in a research compute cluster maintained at Washington University in St. Louis.
Training Data
Overview |  These models were trained on a set of audit log obtained from ICU physicians performing work in the BJC HealthCare/Washington University system as a part of the ASPIRE study (IRB# 202009032, NLM grant 5R01LM013778-03). The original training data is protected health information (PHI) and cannot be released.
Data Freshness |  The training data was taken over a period of March 2019 to February 2020 (pre-COVID).
Evaluation Results
See sec:results for details on the performance of each of our models.
Ethical Considerations and Limitations
Because of their experimental nature, we do not recommend the usage of these
models outside of a research context, due to the limitations discussed in
sec:limitations. In addition, no generated data will contain PHI, as the the
two PHI columns (PAT_ID and ACCESS_TIME) have been quantized, and clinician
identifiers are not used as a feature.
Table 6: Model card for the models discussed.