# Fine-grained Identity-Preserving Landmark Synthesis for Face Reenactment

Haichao Zhang1,2,3, Youcheng Ben3, Weixi Zhang1, Tao Chen1, Gang Yu3, Bin
Fu1,3

###### Abstract

Recent face reenactment works are limited by the coarse reference landmarks,
leading to unsatisfactory identity preserving performance due to the
distribution gap between the manipulated landmarks and those sampled from a
real person. To address this issue, we propose a fine-grained identity-
preserving landmark-guided face reenactment approach. The proposed method has
two novelties. First, a landmark synthesis network which is designed to
generate fine-grained landmark faces with more details. The network refines
the manipulated landmarks and generates a smooth and gradually changing face
landmark sequence with good identity preserving ability. Second, several novel
loss functions including synthesized face identity preserving loss,
foreground/background mask loss as well as boundary loss are designed, which
aims at synthesizing clear and sharp high-quality faces. Experiments are
conducted on our self-collected BeautySelfie and the public VoxCeleb1
datasets. The presented qualitative and quantitative results show that our
method can reenact fine-grained higher quality faces with good ID-preserved
appearance details, fewer artifacts and clearer boundaries than state-of-the-
art works. Code will be released for reproduction.

Figure 1: Illustration of our method on identity-preserving landmark synthesis
for face reenactment. The top row on the right is synthesized landmarks with
gradually changing poses from beginning to end using different approaches. The
bottom is reenacted face images guided by the above landmarks. As we can see
in the red circle, the results guided by interpolated landmarks greatly change
the face shape and other facial organs like the unnatural eyes, which leads to
an identity preserving problem. In contrast to that, our synthesised landmarks
can preserve identity related facial attribute information, and help to
reenact better ID-preserved faces.

Figure 2: nose width (narrow $\to$ wide) Figure 3: Our experiments on
manipulating different facial attributes show the existence of the identity
gap. The identity gap in the manipulated landmarks causes face attribute
manipulation to clearly change the identities of reenacted faces.

## Introduction

Face reenactment is aiming to transfer a reference face landmark including the
expression, pose or other details to the target while keeping the appearance
and identity of the target unchanged. Take faithful photo-realistic facial
reenactment as an example, this technique can find a wide range of
applications from video conferencing, film production to augmented reality.
Landmark synthesis is widely used as a guided signal of this technique, which
has a significant impact on the face reenactment performance. However, due to
its geometry structure, the guided face landmark which is supposed to provide
emotion information only, often carries the identity information from its
provider. This brings difficulty for preserving the identity information of
the original face in the face reenactment process. Therefore, an identity-
preserving landmark synthesis network for face reenactment is urgently needed.

A number of works related to landmark synthesis and face reenactment have
emerged recently. We divide them into three classes. The first class use more
diversified inputs than landmark as guided signal for face reenactment,
ignoring the identity gap problem. DeepFake (dee 2017) uses an autoencoder
network to swap faces of two targets. Tripathy et al. (Tripathy, Kannala, and
Rahtu 2019) use facial attributes as input to guide the source and then
synthesize face images with corresponding poses and expressions. Similarly,
GANimation (Pumarola et al. 2019) also uses action units(AU) (Friesen and
Ekman 1978) to synthesize magnitude-controllable expressions. FSGAN (Nirkin,
Keller, and Hassner 2019) uses segmentation, inpainting and landmark to
produce more realistic results. The second class use few-shot training or
landmark adaption to convey the emotion, and remove identity information from
the target in an explicit or implicit way. Few-Shot (Zakharov et al.
2019),Burkov (Burkov et al. 2020), FaceSwapNet (Zhang et al. 2019) and
FReeNet(Zhang et al. 2020) leverage on landmarks as a tool to restrain the
expression of the target. Compared with Few shot-based method (Zakharov et al.
2019), FaceSwapNet and FReeNet make further progress by optimizing the
reference landmarks to make the generated face shape look more like the
source. The third class Face2Face (Thies et al. 2016; Kim et al. 2018)
generate output faces using 3D models, which can help to suppress the subtle
changes of expressions on the synthesized faces.

Although these recent studies have pushed the research of face reenactment
forward a lot, they still suffer from several challenges:  (1) Very coarse
exemplar landmarks that lose much face content (part) details such as eyeball
positions, lip shape or depths, more fine expression and pose types are used,
which leads to limited capability to preserve the face appearance and identity
information. For example, due to the coarse landmarks used in Few-Shot,
FaceSwapNet and FSGAN, these works are difficult to reenact faces with
arbitrary and flexible poses and expressions. Even though FSGAN combines
landmark, segmentation and inpainting as well as blending proposed in (Nirkin,
Keller, and Hassner 2019) together, the appearance and ID preservation
capability of reenacted faces is still not satisfactory.  (2) Some of the
above works, e.g., Few-Shot (Zakharov et al. 2019),Burkov (Burkov et al.
2020),FaceSwapNet (Zhang et al. 2019) focus on generating better cross-
identity results, but still ignore scenarios that require to synthesize
landmarks of the same face identity or maintain identity of the source, which
is especially important to application scenarios such as talking heads
synthesis (Zakharov et al. 2019), face animation (Sanyal et al. 2019) and
video synthesis driven by voice (Kumar et al. 2017; Zhou et al. 2020) or text
(Fried et al. 2019). Fig.3 illustrates the identity gap between synthesized
landmarks and real landmarks, which makes it difficult to preserve identity
information in synthesized faces.

To address these intractable problems, we propose a fine-grained identity-
preserving landmark synthesis approach, which can efficiently generate fine-
grained id-preserving landmark sequence for face reenactment according to
given two face landmarks. Several novel loss functions including synthesized
face identity preserving loss, foreground and background mask loss as well as
boundary loss are designed, which aim at synthesizing clear and sharp high-
quality faces. Specially, the designed face identity preserving loss
consisting of an adversarial training loss and support task, provides
supervision for both real/fake generation and the landmark identity
prediction, and shows ability to alleviate the identity gap in both explicit
and implicit space between two input landmarks.

Further, the designed foreground and background mask loss makes the generated
faces to be more clean and sharp with high quality. Boundary loss improves the
results of generated side faces (faces observed from side view) with large
poses. Note that the side-face reenactment problem due to large pose changes
is not well solved in the above-mentioned works. The large-pose data
augmentation and face alignment is leveraged in this work to further improve
the large-pose face reenactment results.

To sum up, our major contributions are listed as follows:

  * •

A fine-grained identity-preserved landmark sequence generator is created for
alleviating the identity gap problem for face reenactment. As far as we know,
this is the fist work focusing on bridging the identity gap in synthesizing
landmarks. In addition to landmarks, our model also shows good potential for
reducing the identity gap for manipulating faces to a certain degree.

  * •

Several novel strategies including the designed loss functions and large pose
data augmentation, aiming at fine-grained exemplar landmark synthesis and face
reenactment, are developed, which can help to preserve the landmark and face
identity and renders the synthesized results look clean, sharp and high-
fidelity.

  * •

A largest Asian facial expression dataset, selfie dataset, that contains high-
quality video clips of 23,000 different identities involving most ever seen
expressions and poses is created for comprehensive face reenactment and
manipulation evaluation.

## Related Work

Our work is closely related to conditional face reenactment, identity-
preserving synthesis and landmark-guided sequence generation. We discuss main
related publications on these research topics as follows.

### Conditional Face Reenactment

Landmark based face reenactment have been widely exploited recently. Many
works synthesized faces by introducing landmarks as conditional input in their
GAN networks (Goodfellow et al. 2014), and the results are really impressive.
FSGAN (Nirkin, Keller, and Hassner 2019) presents a better solution with the
combination of landmarks, segmentation and inpainting. While it shares certain
similarity with our work, it still differs from our work in following aspects.
First, landmarks in FSGAN are mainly used for generating facial expressions of
different angles, while our work also leverages landmarks to capture more
facial details, such as eyeball movement and lip shapes. Second, the aim of
segmentation in FSGAN is to get more concordant blending results of
reenactment, while our work uses segmentation mask as restrictions to help
generate cleaner and sharper results.

Other works such as GANimation (Pumarola et al. 2019) uses AU (Friesen and
Ekman 1978) to synthesize magnitude-controllable expressions. Although AU can
be easily used to control expressions, it also limits the range of expressions
generated in reenactment task. Besides, artifact is another problem of this
method.

### Identity Preserving Landmark Synthesis

To preserve identity information of source faces during generation, previous
works such as Few-Shot (Zakharov et al. 2019), FaceSwapNet (Zhang et al. 2019)
guarantees identity by imposing restrictions to synthesized landmarks. In
terms of many-to-many face reenactment, recent works (Huang, Yang, and Wang
2020; Zhang et al. 2020; Burkov et al. 2020) have explored effects of
landmarks on the identity preserving of synthesized faces, and the goal is to
achieve identity-invariant face reenactment.

However, most recent works focus on generating better cross-identity results,
with less attention paid to the quality of manipulated landmarks for the same
face identity, which is actually crucial to application scenarios such as
talking heads synthesis (Zakharov et al. 2019), face animation (Sanyal et al.
2019) and video synthesis driven by voice (Kumar et al. 2017; Zhou et al.
2020) or text (Fried et al. 2019). The gap between manipulated and real
landmarks makes it difficult to preserve source identity information in
reenacted faces, and in this work we aim to reduce this gap.

Figure 4: Framework of our proposed fine-grained landmark guided face
reenactment network. The left bottom upper branch is Framework of our proposed
identity-preserved landmark sequence generator. Our Bi-LSTM based generator G1
takes two facial landmarks to predict six medium landmark states of the
inputs. The D1, D2 are two discriminators providing adversarial losses, while
S1 and S2 are two networks providing the support task losses. The D1 and D2
predict the realness score of the output landmarks and the probability that
the input and output belongs to the same id, respectively. The S1, S2 try to
predict ID label from the hidden state feature and output landmarks of G1,
respectively. The upper branch is the training stage and the lower branch is
the inference stage. In the training stage, an input face goes through face
alignment and forwarded to the generator $G$ together with an exemplar
reference landmark created by a pretrained network. Several newly designed
losses: arc loss, mask loss and boundary loss coupled with perceptual and
adversarial loss are used as training objectives.

## The Proposed Approach

In this section, we will present the details of the proposed landmark
synthesis guided face reenactment approach. It can be divided into two stages.
In the first stage, our main objective is to predict a facial landmark
sequence which keeps the identity related information unchanged and approaches
the real temporal sequence distribution. In the second stage, our main
objective is to transfer pose and expression from one face to another. Given a
source face image $F_{s}$ which contains the identity information and a target
face image $F_{t}$ which provides the pose and expression guidance, our model
is supposed to synthesize a new face which retains the appearance of $F_{s}$
and the pose and expression of $F_{t}$.

The overall architecture of our approach is described in Section Fine-grained
landmark-guided face reenactment. Then we describe the identity-preserved
landmark sequence generator in section Identity-preserved Landmark Sequence
Generator.. Training objectives and data augmentation strategy are detailed in
Section Fine-grained landmark-guided face reenactment and Section Pose-Aware
Data Augmentation respectively.

### Identity-preserved Landmark Sequence Generator.

Figure 4 illustrates the framework of our proposed identity-preserved landmark
sequence generator (LSG) . We adopted a Bi-LSTM auto-encoder $G_{l}$ as the
backbone network for our identity-preserved landmark generation. The landmark
generator takes two sets of face keypoints $p_{s1},p_{sk}$ from two face
images by HRNet(Sun et al. 2019) as input. We use $[p_{s1},p_{sk}]$ to denote
the input, and the $[p_{1},p_{2},...,p_{K}],(K\in N)$ to denote the
output,where $k$ is number of generated landmark frames. During training
process, we use realness discriminator $D_{1}$ and identity consistence
discriminator $D_{2}$ to provide adversarial loss, and use two fully-connected
nets to predict the id label from each output landmark $p_{k},k\in K$ and
hidden state features of $G_{1}$, respectively.

Forward Process.  Firstly, the $[p_{s1},p_{sk}]$ are upsampled to $K$ frames,
then $G_{l}$ takes them as the input and predicts shifts for each key point.
The forward process can be denoted as:

$\begin{split}[p_{1},...,p_{K}]=&\Delta shift+f_{up}[p_{s1},p_{sk}]\\\
=&G_{l}(p_{s1},p_{s1},..,p_{sk})+[p_{s1},,..,p_{sk}]\end{split}$ (1)

in which $f_{up}$ denotes the upsampling process. The overall loss function
can be denoted as:

$\mathbf{L}_{LSG}=\lambda_{1}\mathcal{L}_{D_{1}}+\lambda_{2}\mathcal{L}_{D_{2}}+\lambda_{3}\mathcal{L}_{S_{1}}+\lambda_{4}\mathcal{L}_{S_{2}}$
(2)

where the $\lambda_{i},(i=1,2,3,4)$ denotes the weight of each loss function.

Support task Loss Huang (Huang, Yang, and Wang 2020) uses an ID classifier to
remove the identity feature in landmark latent space. However, we find that
such way of using ID classifier could encourage our hidden state to take the
identity information from the input landmark. We employ $S_{2}$ to predict the
ID label of input face key points $p_{s1},p_{sk}$ , and use the cross entropy
loss of $S_{2}$ to force our LSG to learn identity information as much as
possible. So the $\mathcal{L}_{S_{2}}$ can be denoted as:

$\mathcal{L}_{S_{2}}=\mathcal{L}_{CE}(c_{p},c_{S_{2}})$ (3)

here, $c_{p}$ and $c_{S_{2}}$ represents identity class label of input
$p_{s1},p_{sk}$ . In the explicit way, $S_{1}$ predicts the ID label and
provide cross-entropy loss from generated landmarks.

$\mathcal{L}_{S_{2}}=\mathcal{L}_{CE}(c_{p},c_{S_{1}}(p_{k})),k\in[1,K]$ (4)

Adversarial Loss We employee two discriminator $D_{1}$, $D_{2}$ to provide
adversarial losses. The $D_{1}$ takes one landmark each time, to predict
whether the landmark is synthesized by $G_{1}$ or sampled from real world. The
$D_{2}$ takes two landmarks and predict the possibility that they belongs to
the same identity.

### Fine-grained landmark-guided face reenactment

Figure 4 illustrates the pipeline of our proposed approach for fine-grained
landmark guided face reenactment. We adopt Pix2pixHD (Wang et al. 2018) as the
backbone network for face image generation. Besides the source input image
$F_{s}$, our model also takes as input the sketch map $p_{t}$ of a target face
image $F_{t}$, which is obtained based on $98$ facial landmarks predicted by
HRNet (Sun et al. 2019). Therefore, the input to our model is a concatenated
volume $x=[F_{s},p_{t}],x\in\mathcal{R}^{H\times W\times(3+3)}$. The output of
our model is a volume $y\in\mathcal{R}^{H\times W\times 3}$ which refers to a
synthesized face image. During the inference stage, the ID-preserving landmark
generator is used to produce more fine-grained landmarks with different
expressions and poses, which are used to reenact more faces with fine-grained
expression and pose details.

Training Objectives For training the face reenactment network, the adversarial
loss (Goodfellow et al. 2014; Wang et al. 2018) and perceptual loss (Johnson,
Alahi, and Fei-Fei 2016) are adopted to generate faces with high fidelity.
Besides, we also propose novel boundary loss and mask loss to ensure fewer
artifacts and sharper details on the boundary and surface of synthesized faces
respectively. Furthermore, identity loss is explored which effectively retains
the appearance information of the source face $F_{s}$.

#### Adversarial Loss

To make our synthesized faces more realistic, we employ an adversarial
objective to push the distribution of generated images as close as possible to
that of real images. Similar to Pix2pixHD (Wang et al. 2018), we adopt a
multi-scale discriminator consisting of multiple discriminators,
$D_{1},D_{2},\dots,D_{n}$, with each operating on a different image
resolution. For a generator $G$ and a multi-scale discriminator $D$, our
adversarial loss is defined as:

$\mathcal{L}_{adv}(G,D)=\min_{G}\max_{D_{1},\dots,D_{n}}\sum_{i=1}^{n}\mathcal{L}_{GAN}(G,D_{i}),$
(5)

#### Identity Loss

To preserve the identity information of source face $F_{s}$, we propose an
identity consistency loss which regularizes the synthesized face to be close
to $F_{s}$ in the ArcFace (Deng et al. 2019) feature space, which enables the
generator to preserve the face identity effectively. The identity consistency
loss is defined as:

$\mathcal{L}_{ID}(G)=\mathbb{E}_{x}[d_{cos}(\theta(G(x)),\theta(x))],$ (6)

Here, the $\theta$ is a pre-trained ArcFace (Deng et al. 2019) network.

#### Mask Loss

Generally, the synthesis of foreground facial details is more important than
that of background. Therefore, we attempt to segment the facial area of the
ground truth to obtain a foreground mask. Leveraging on that mask, we
introduce pixel-level $L1$ loss between the generated images and real images
in order to reduce artifacts and make the synthesized faces more clear. The
mask loss is defined as:

$\mathcal{L}_{MASK}(G)=\mathbb{E}_{x,y}[||G(x)\odot M,y\odot M||_{1}],$ (7)

where $M$ is a foreground attention mask parsered by a pre-trained BiSeNet (Yu
et al. 2018).

#### Boundary Loss

As described before, we adopt HRNet (Sun et al. 2019) to predict facial
landmarks of both synthesized and real images. By aligning landmarks on the
boundary of real and synthesized faces as close as possible, we expect our
model to generate images with fewer artifacts on the boundary of face area.
The boundary loss is defined as:

$\mathcal{L}_{BOUND}(G)=\mathbb{E}_{x,y}[d_{smoothL1}(\eta(G(x)),\eta(y))],$
(8)

where $\eta$ is a pre-trained HRNet (Sun et al. 2019) on WFLW (Wu et al.
2018), and smooth L1 distance is implemented as (Girshick 2015).

Figure 5: Qualitative comparison of different approaches on the VoxCeleb1
dataset. One-shot inference on an identity not seen during training is shown
in the source column. Next columns show ground truth image, taken from the
rest part of the video sequence, and the synthesized results of different
approaches. Note that our result is slightly different from others due to
customed face alignment applied to the whole dataset.

### Pose-Aware Data Augmentation

To deal with large pose transfer cases or side-view face generation problem,
we develop a novel pose-aware data augmentation strategy, which gradually
increases the difficulty of pose transfer during training. In particular, for
video frames of each identity in the dataset, we first gather statistics of
maximum pose variation range in terms of Euler angles(i.e., yaw and pitch).
Then we set an initial threshold based on the computed pose statistics to
determine whether to sample from specific identity or not. If the identity
pose angle is above the threshold, it will be sampled for training. After
every tens of training epochs, we gradually increase the threshold to allow
for sampling more difficult pose transfer cases.

There are two advantages by adopting this augmentation policy. One is that our
model is capable of learning pose transfer more smoothly along with gradually
increased pose sampling threshold. The other is that under this sampling
policy, our model is likely to observe more images with side-view faces, which
can synthesize better face results with fewer artifacts.

## Experiments

To showcase the effectiveness of our proposed approach, we perform
comprehensive studies on two video datasets involving facial movements and
compare our model with several state-of-the-art approaches.

### Datasets

We evaluate our model on both public VoxCeleb1 (Nagrani, Chung, and Zisserman
2017) dataset and our self-collected BeautySelfie dataset. BeautySelfie
contains $37k$ face videos of beautified selfies, each corresponding to a
particular identity, of which we use $10k$ for training which are of good
quality. For fair and fast comparison, we only conduct ablation studies on
this private dataset. VoxCeleb1 (Nagrani, Chung, and Zisserman 2017) (256p
videos at 1fps) is a famous dataset consisting of talking head videos, which
contains over $100,000$ utterances for $1,251$ celebrities. We follow (Wiles,
Sophia Koepke, and Zisserman 2018; Tripathy, Kannala, and Rahtu 2019) to
perform train/test splits on this dataset. Since VoxCeleb1 is a public
benchmark for face reenactment, we perform both qualitative and quantitative
comparisons with previous works on this dataset.

### Implementation details

For both datasets, we preprocess raw video frames by first detecting faces
using a pre-trained face detector. We filter out cropped faces which are
blurry or have lower resolution than $128\times 128$ pixels. Then we align
faces based on facial landmarks predicted by HRNet (Sun et al. 2019) under
least square constraints. We fill missing pixels with zeros which may result
from rotation operations during the alignment. Finally, faces are rescaled to
$256\times 256$ pixels in all experimental settings.

Figure 6:  Ablation study of the effectiveness of different loss functions on
the BeautySelfie dataset. One-shot inference on an identity not seen during
training is shown in the source column. Sketch map corresponds to facial
landmarks predicted by HRNet on a target face from the same identity. Result
of the model trained with only adversarial and perceptual loss is shown in the
vanilla column. Results of models trained with additional identity loss, mask
loss and boundary loss are shown in the next three columns respectively.

In each training iteration, we sample source faces $F_{s}$ and target faces
$F_{t}$ from the same video sequences to construct training pairs for face
reenactment. We sample $10$ pairs of training examples iteratively for each
identity in the dataset during this process. The proposed model is trained
from scratch, where the weights are initialized randomly using a normal
distribution. We use Adam (Kingma and Ba 2014) optimization
($\beta_{1}=0.5,\beta_{2}=0.999$) for both generator and discriminator with an
initial learning rate of $0.0002$. We train our network for a total of $45$
epochs and reduce the learning rate linearly to zero after $30$ epochs.

Evaluation metrics. For quantitative comparison, we evaluate two indexes,
i.e.the fidelity and the semantics of the synthesized results. For the
evaluation of reality and diversity, we adopt the widely used Fréchet
Inception Distance (Heusel et al. 2017) metric. In terms of the semantics, we
use SSIM (Wang et al. 2004) to measure the low-level similary between
synthesized faces and expected real faces, and CSIM (Zakharov et al. 2019) to
calculate cosine similarity between embedding vectors of the state-of-the-art
face recognition network (Deng et al. 2019) for measuring identity mismatch.
These two metrics to some extent reveal how successfully we tranfer pose and
expression from a source face $F_{s}$ to a target face $F_{t}$ while retaining
the appearance information of $F_{s}$.

#### Qualitative Landmark Synthesis Performance

Fig.7 shows the refinement phenomenon in our landmark synthesis model. The two
landmark generation approaches take two noisy input landmarks, and predict six
landmarks. The synthesised landmarks was then send them into a pix2pixHD
generator to reenact the synthesised face. Our landmarks generator show the
ability to refine and generate natural landmarks with the noisy input, in the
meanwhile, the linear interpolation has nothing to do with it. This experiment
also shows the robustness and potential refinement ability of our approaches.

Figure 7:  The refinement phenomenon appeared in our landmark generator. With
two input noisy landmarks, the landmark sequences was generated by linear
interpolation and our approaches. Due to the linear interpolation calculation,
the left and right landmark in the Linear Interpolation Landmarks are the same
as the two landmarks. Our Landmarks generator show the ability to refine the
noise in the input landmarks, which prevents the face reenactment results with
less artifacts, and in this situation, better preserving the identities.

#### Qualitative Face Reenactment Performance

Figure 5 compares synthesized one-shot results of several approaches on the
VoxCeleb1 (Nagrani, Chung, and Zisserman 2017) dataset, where we quote results
of X2Face, Pix2pixHD and Few-Shot directly from (Zakharov et al. 2019).
According to (Zakharov et al. 2019), X2Face is used with pre-trained weights
provided by the original authors, while both Pix2pixHD and Few-Shot are pre-
trained from scratch on the whole dataset. Note that these approaches are also
fine-tined additionally on the few-shot set. In comparison, our model is
learned only on the training set.

Compared to previous approaches, our model is capable of generating more
realistic human faces with fewer artifacts. Further, the reenacted faces of
our approach meets the target better from different aspects, i.e. eyeball
positions, lip or mouth shapes and face poses. More importantly, we observe
more fine-grained details in our synthesized faces that are consistent with
the source faces. This shows that our model has better capability to retain
appearance information of the source faces and transfer target expression or
pose information for reenacting results.

Figure 8:  Histogram of CSIM distribution quantitative comparison of our
Landmark Synthesis Performance approaches in Table1 , where horizontal axis
and vertical axis denote the CSIM score and relative numbers of samples
perform this score.

### Quantitative Results

#### Quantitative Results on Landmark Synthesis Performance

Table 1 and Fig.8 compare the face reenactment result with different
approaches of synthesized landmark, with the metric cosine similarity
(CSIM)(Zakharov et al. 2019) to mearsure the identity similarity. Due to few
works focus on the identity-preserving landmark synthesis, we use the linear
interpolation (LI) result to simulate a smooth and gradually changing between
two input landmarks as our baseline.

Our synthesized landmark has better performance on CSIM score, which indicates
our synthesized landmark preserve better identities than LI baseline. The
histogram shows the distribution of our method has more ares in the score
range 0.80-0.97, while the different area of LI nearly with the same size is
in lower score range. To sum up, our ID-preserving model has better identity-
preserving ability than the baseline.

Method | CSIM$\uparrow$
---|---
VoxCeleb1
LI | $0.70$
Ours | $0.72$
Table 1:  Quantitative comparison of our Landmark Synthesis Performance
approaches with LI baseline.

#### Quantitative Results on Face Reenactment Performance.

Method | FID$\downarrow$ | SSIM$\uparrow$ | CSIM$\uparrow$
---|---|---|---
VoxCeleb1
X2Face | $45.8$ | $0.68$ | $0.16$
Pix2pixHD | $\mathbf{42.7}$ | $0.56$ | $0.09$
Few-Shot | $43.0$ | $0.67$ | $0.15$
Ours | $43.9$ | $\mathbf{0.79}$ | $\mathbf{0.18}$
Table 2:  Quantitative comparison of several approaches under one-shot
experimental setting with three different metrics on the VoxCeleb1 dataset.
The baselines here are X2Face (Wiles, Sophia Koepke, and Zisserman 2018),
Pix2pixHD (Wang et al. 2018) and Few-Shot (Zakharov et al. 2019).

Table 2 compares synthesized one-shot results of several approaches on the
VoxCeleb1 dataset with three different metrics, where we quote results of
X2Face, Pix2pixHD and Few-Shot from (Zakharov et al. 2019) likewise in section
Quantitative Results on Landmark Synthesis Performance.

Our synthesized results have similar fidelity compared to other approaches in
terms of the FID metric. For the other two metrics, SSIM and CSIM, significant
improvements are observed. In particular, our model increases the SSIM score
of X2Face by 11%, and CSIM score of X2Face by as high as 2%, which reveals
that our model is superior to other approaches in transferring the semantics
of pose and expression while retaining the identity information.

### Ablation Study

To systematically evaluate the proposed method, we perform comprehensive
ablation studies using different combinations of the proposed loss functions
(i.e.identity loss, mask loss and boundary loss) and pose-aware data
augmentation policy.

Effect of the proposed loss functions. Figure 6 demonstrates synthesized
results of our model under different groups of loss functions on the
BeautySelfie dataset. Results in the vanilla column provide a strong baseline
for comparison. We sequentially compare the last three columns with the
vanilla column to figure out the effect of each proposed loss function.

Compared to synthesized faces in column $3$, we observe better synthesis
results in column $4$ with nose, mouth, eyes and other fine-grained details
more consistent with source, which reveals that the identity preserving loss
effectively retains appearance and part information of source images. As a
comparison between synthesized faces in column $4$ and $5$, it can be seen
that the mask loss helps eliminate artifacts on the surface of synthesized
faces. As for results displayed in column $6$, we find boundary loss enables
our model to generate faces with better boundary lines and fewer artifacts
around.

Effect of pose-aware data augmentation. As shown in Figure 9, we compare our
model under different settings of pose-aware data augmentation on the
BeautySelfie dataset. We observe fewer artifacts and clearer boundary lines on
the synthesized side-view faces using the model trained with developed pose-
aware data augmentation strategy. This well validates the effectiveness of the
proposed data augmentation strategy to improve the capability of our model in
transferring poses with large variations.

(a) without pose-aware data argumentation

(b) with pose-aware data argumentation

Figure 9:  Ablation study of the effectiveness of pose-aware data
argumentation on the BeautySelfie dataset. Synthesized results from models
with the different setting of pose-aware data argumentation are shown in (a)
and (b) respectively.

## Conclusion

In this work, we propose a fine-grained ID-preserving landmark synthesis
approach for deep face reenactment, addressing the problem of used coarse
landmark lacking details and landmark ID alternation for the first time. Also,
we design several novel losses as training objectives to deal with result blur
and artifact which are ubiquitous in the previous studies. Finally, we develop
large pose data augmentation strategy coupled with face alignment, to solve
the side-view face reenactment problem due to large pose changes. The
qualitative and quantitative experimental results demonstrate the superiority
of our approach over the state-of-the-art works.

> ## References
>
>   * dee (2017) 2017\.  Github of Deepfake.
> https://github.com/deepfakes/faceswap/.
>   * Burkov et al. (2020) Burkov, E.; Pasechnik, I.; Grigorev, A.; and
> Lempitsky, V. 2020.  Neural Head Reenactment with Latent Pose Descriptors.
> In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
> Recognition_ , 13786–13795.
>   * Deng et al. (2019) Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2019.
> Arcface: Additive angular margin loss for deep face recognition.  In
> _Proceedings of the IEEE Conference on Computer Vision and Pattern
> Recognition_ , 4690–4699.
>   * Fried et al. (2019) Fried, O.; Tewari, A.; Zollhöfer, M.; Finkelstein,
> A.; Shechtman, E.; Goldman, D. B.; Genova, K.; Jin, Z.; Theobalt, C.; and
> Agrawala, M. 2019.  Text-based editing of talking-head video.  _ACM
> Transactions on Graphics (TOG)_ 38(4): 1–14.
>   * Friesen and Ekman (1978) Friesen, E.; and Ekman, P. 1978.  Facial action
> coding system: a technique for the measurement of facial movement.  _Palo
> Alto_ 3\.
>   * Girshick (2015) Girshick, R. 2015.  Fast r-cnn.  In _Proceedings of the
> IEEE international conference on computer vision_ , 1440–1448.
>   * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.;
> Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.
> Generative adversarial nets.  In _Advances in neural information processing
> systems_ , 2672–2680.
>   * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler,
> B.; and Hochreiter, S. 2017\.  Gans trained by a two time-scale update rule
> converge to a local nash equilibrium.  In _Advances in Neural Information
> Processing Systems_ , 6626–6637.
>   * Huang, Yang, and Wang (2020) Huang, P.-H.; Yang, F.-E.; and Wang, Y.-C.
> F. 2020.  Learning Identity-Invariant Motion Representations for Cross-ID
> Face Reenactment.  In _Proceedings of the IEEE/CVF Conference on Computer
> Vision and Pattern Recognition_ , 7084–7092.
>   * Johnson, Alahi, and Fei-Fei (2016) Johnson, J.; Alahi, A.; and Fei-Fei,
> L. 2016.  Perceptual losses for real-time style transfer and super-
> resolution.  In _European conference on computer vision_ , 694–711.
> Springer.
>   * Kim et al. (2018) Kim, H.; Carrido, P.; Tewari, A.; Xu, W.; Thies, J.;
> Niessner, M.; Pérez, P.; Richardt, C.; Zollhöfer, M.; and Theobalt, C. 2018.
> Deep video portraits.  _ACM Transactions on Graphics (TOG)_ 37(4): 163.
>   * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for
> stochastic optimization.  _arXiv preprint arXiv:1412.6980_ .
>   * Kumar et al. (2017) Kumar, R.; Sotelo, J.; Kumar, K.; de Brébisson, A.;
> and Bengio, Y. 2017.  Obamanet: Photo-realistic lip-sync from text.  _arXiv
> preprint arXiv:1801.01442_ .
>   * Nagrani, Chung, and Zisserman (2017) Nagrani, A.; Chung, J. S.; and
> Zisserman, A. 2017.  VoxCeleb: a large-scale speaker identification dataset.
> In _INTERSPEECH_.
>   * Nirkin, Keller, and Hassner (2019) Nirkin, Y.; Keller, Y.; and Hassner,
> T. 2019.  FSGAN: Subject Agnostic Face Swapping and Reenactment.  In
> _Proceedings of the IEEE International Conference on Computer Vision_ ,
> 7184–7193.
>   * Pumarola et al. (2019) Pumarola, A.; Agudo, A.; Martinez, A. M.;
> Sanfeliu, A.; and Moreno-Noguer, F. 2019\.  GANimation: One-shot
> anatomically consistent facial animation.  _International Journal of
> Computer Vision_ 1–16.
>   * Sanyal et al. (2019) Sanyal, S.; Bolkart, T.; Feng, H.; and Black, M. J.
> 2019.  Learning to regress 3d face shape and expression from an image
> without 3d supervision.  In _Proceedings of the IEEE Conference on Computer
> Vision and Pattern Recognition_ , 7763–7772.
>   * Sun et al. (2019) Sun, K.; Xiao, B.; Liu, D.; and Wang, J. 2019.  Deep
> High-Resolution Representation Learning for Human Pose Estimation.  In
> _CVPR_.
>   * Thies et al. (2016) Thies, J.; Zollhofer, M.; Stamminger, M.; Theobalt,
> C.; and Nießner, M. 2016\.  Face2face: Real-time face capture and
> reenactment of rgb videos.  In _Proceedings of the IEEE Conference on
> Computer Vision and Pattern Recognition_ , 2387–2395.
>   * Tripathy, Kannala, and Rahtu (2019) Tripathy, S.; Kannala, J.; and
> Rahtu, E. 2019.  ICface: Interpretable and Controllable Face Reenactment
> Using GANs.  _arXiv preprint arXiv:1904.01909_ .
>   * Wang et al. (2018) Wang, T.-C.; Liu, M.-Y.; Zhu, J.-Y.; Tao, A.; Kautz,
> J.; and Catanzaro, B. 2018\.  High-Resolution Image Synthesis and Semantic
> Manipulation with Conditional GANs.  In _Proceedings of the IEEE Conference
> on Computer Vision and Pattern Recognition_.
>   * Wang et al. (2004) Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E.
> P.; et al. 2004.  Image quality assessment: from error visibility to
> structural similarity.  _IEEE transactions on image processing_ 13(4):
> 600–612.
>   * Wiles, Sophia Koepke, and Zisserman (2018) Wiles, O.; Sophia Koepke, A.;
> and Zisserman, A. 2018.  X2face: A network for controlling face generation
> using images, audio, and pose codes.  In _Proceedings of the European
> Conference on Computer Vision (ECCV)_ , 670–686.
>   * Wu et al. (2018) Wu, W.; Qian, C.; Yang, S.; Wang, Q.; Cai, Y.; and
> Zhou, Q. 2018.  Look at boundary: A boundary-aware face alignment algorithm.
> In _Proceedings of the IEEE Conference on Computer Vision and Pattern
> Recognition_ , 2129–2138.
>   * Yu et al. (2018) Yu, C.; Wang, J.; Peng, C.; Gao, C.; Yu, G.; and Sang,
> N. 2018.  Bisenet: Bilateral segmentation network for real-time semantic
> segmentation.  In _Proceedings of the European Conference on Computer Vision
> (ECCV)_ , 325–341.
>   * Zakharov et al. (2019) Zakharov, E.; Shysheya, A.; Burkov, E.; and
> Lempitsky, V. 2019.  Few-Shot Adversarial Learning of Realistic Neural
> Talking Head Models.  _arXiv preprint arXiv:1905.08233_ .
>   * Zhang et al. (2019) Zhang, J.; Zeng, X.; Pan, Y.; Liu, Y.; Ding, Y.; and
> Fan, C. 2019.  FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment.
> _arXiv preprint arXiv:1905.11805_ .
>   * Zhang et al. (2020) Zhang, J.; Zeng, X.; Wang, M.; Pan, Y.; Liu, L.;
> Liu, Y.; Ding, Y.; and Fan, C. 2020\.  FReeNet: Multi-Identity Face
> Reenactment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision
> and Pattern Recognition_ , 5326–5335.
>   * Zhou et al. (2020) Zhou, Y.; Li, D.; Han, X.; Kalogerakis, E.;
> Shechtman, E.; and Echevarria, J. 2020\.  MakeItTalk: Speaker-Aware Talking
> Head Animation.  _arXiv preprint arXiv:2004.12992_ .
>


# Enhancing Antibiotic Stewardship using a Natural Language Approach for
Better Feature Representation

Simon A. Lee  Trevor Brokowski  Jeffrey N. Chiang

###### Abstract

The rapid emergence of antibiotic-resistant bacteria is recognized as a global
healthcare crisis, undermining the efficacy of life-saving antibiotics. This
crisis is driven by the improper and overuse of antibiotics, which escalates
bacterial resistance. In response, this study explores the use of clinical
decision support systems, enhanced through the integration of electronic
health records (EHRs), to improve antibiotic stewardship. However, EHR systems
present numerous data-level challenges, complicating the effective synthesis
and utilization of data. In this work, we transform EHR data into a serialized
textual representation and employ pretrained foundation models to demonstrate
how this enhanced feature representation can aid in antibiotic susceptibility
predictions. Our results suggest that this text representation, combined with
foundation models, provides a valuable tool to increase interpretability and
support antibiotic stewardship efforts.

Machine Learning, ICML

##  1 Introduction

The Centers for Disease Control and Prevention (CDC) has declared the rapid
emergence of resistant bacteria a global healthcare crisis, threatening the
efficacy of antibiotics that have saved millions of lives (Ventola, 2015;
Golkar et al., 2014; Gould & Bal, 2013; Sengupta et al., 2013; Nature, 2013;
Lushniak, 2014). This crisis is primarily driven by the mishandling and
overuse of these antibiotics, which leads to bacteria developing resistance
through repetitive exposure (Viswanathan, 2014; Read & Woods, 2014). These
resistant bacteria impose significant clinical and financial burdens on
healthcare systems, as well as on patients and their families worldwide
(Bartlett et al., 2013).

Clinical decision support systems hold substantial potential to assist
healthcare providers in adhering to antibiotic stewardship practices. This
potential is largely facilitated by electronic health record (EHR) software,
which allows for the seamless integration of patient health histories in
digital form (Evans, 2016; Cowie et al., 2017; Hoerbst & Ammenwerth, 2010).
The integration with EHRs enables the use of continuously updated and deployed
machine learning models for clinical decision-making. However, EHR data in its
raw form presents numerous challenges, such as data ingestion and feature
representation (Wu et al., 2010).

In this work, we present a methodology that converts EHR data into a
serialized text form called pseudo-notes to predict antibiotic susceptibility.
This conversion from tabular to text format facilitates the creation of
interpretable data inputs for pretrained foundation models, known for their
rich feature representation. Our primary objective is to develop a predictive
model that incorporates this representation strategy in conjunction with
foundation models. This approach aims to enhance decision support systems and
accurately identify the most suitable antibiotics for patients, thereby
offering a data-driven solution to combat antibiotic resistance.

##  2 Related Works

#### Medical Representation Learning

Medical representation learning on Electronic Health Records (EHRs) has
emerged as a critical area in healthcare research, focusing on transforming
complex medical data for enhanced clinical decision-making. Initially, this
involved extensive feature engineering to convert raw EHR data into formats
suitable for traditional machine learning models (Tang et al., 2020; Ferrao et
al., 2016). However, this approach can be labor-intensive and varies
significantly across research groups due to the lack of a standard protocol.

Recently, the focus has shifted to advanced foundation models that learn to
represent medical data by analyzing extensive text corpora, including clinical
notes, medical literature, and records. These models, primarily based on the
BERT architecture, generate rich, contextual latent representations of patient
histories, significantly reducing the manual effort in feature engineering
(Rasmy et al., 2021; Liu et al., 2021; Alsentzer et al., 2019; Lee et al.,
2020). Moreover, new techniques have been developed to incorporate the
longitudinal nature of EHR data, leveraging a patients progression to predict
outcomes (Steinberg et al., 2023; Wornow et al., 2024; Pang et al., 2021; Li
et al., 2022).

#### Clinical Decision Support

Machine learning enhances clinical decision support systems by analyzing vast
datasets to provide evidence-based recommendations that improve patient care
outcomes (Sutton et al., 2020). Since the accessibility to EHR systems, there
have been numerous such use cases in predicting diseases (Liu et al., 2018;
Cheng et al., 2016), and various other patient outcomes (Lee et al., 2024;
Suter et al., 1994; Churpek et al., 2014) across all institutions within the
healthcare system. Furthermore, the integration of predictive models into
clinical workflows enables researchers to preemptively manage chronic
conditions and mitigate potential health crises before they escalate (Li et
al., 2020; Goldstein et al., 2017; Hohman et al., 2023). By continuously
learning from new data, these systems evolve to provide more accurate
assessments and recommendations, which support ongoing improvements in medical
practices and patient management strategies. However, much work remains to be
done on addressing the generalization and biases prevalent in many of these
predictive algorithms (Goetz et al., 2024; Agniel et al., 2018).

##  3 Methods

#### EHR

Electronic Health Records are digital versions of a patient’s medical history,
maintained over time by healthcare providers. These records are valuable but
consist of heterogeneous tabular datasets organized into separate tables such
as diagnostics, demographics, and medication, presenting numerous challenges
for researchers.

The primary challenge with EHRs is the heterogeneous nature of the data, which
includes numerical, categorical, and free-text formats that are difficult to
integrate and convert into machine-readable formats. Furthermore, categorical
data fields often contain a large number of classes, potentially distorting
their original representation. For instance, employing feature engineering
techniques such as dummy coding for categorical variables can introduce
collinearity, increase dimensionality, and result in sparse data
representations. These modifications can complicate simple table readouts and
require more memory capacity for statistical models to function effectively.

#### Pseudo-notes: Clinical Notes Generation from Tabular Data

Recent research has introduced a methodology for serializing tabular data into
text using text templates (Hegselmann et al., 2023). This approach
significantly enhances our work by enabling uniform representation of all data
in the EHR as human-readable and interpretable text, rather than as a
collection of merged tables. Moreover, it creates an interface to use
foundation models pre-trained on large text corpora, facilitating rich feature
representation. In our work, we use a mapping function $f:T\rightarrow S$ that
turns individual tables into serialized text, where $T$ stands for individual
tables and $S$ for serialized text. For each patient, we convert each of their
$N$ tables—which cover different aspects like diagnostics, medications, and
vitals—into text segments. These segments are then joined to create a single,
detailed paragraph per patient. This method combines all pertinent patient
information from various sources into one unified narrative, effectively
transforming the data structure into $S=\bigcup_{i=1}^{N}f(T_{i})$, where
$T_{i}$ is the ith table row concerning the patient.

#### Data Source and Inclusion Criteria

Figure 1: Area under the Receiver Operating Characteristic (AUROC) curves for
each antibiotic classification shows that, despite the nuances, BioMegatron
performs the best.

We sourced data from the Medical Information Mart for Intensive Care IV
(MIMIC-IV) and MIMIC-IV Emergency Department (ED) databases (Johnson et al.,
2020, 2023). This study focused on ED patients presumed to have staph
infections, selected based on specific inclusion criteria. Eligible
participants included those with any microbiological culture testing positive
for a staph-related organism, sourced from bodily fluids such as blood, urine,
cerebral spinal fluid, pleural cavity, or joint fluid, accompanied by a
prescribed antibiotic whose susceptibility was subsequently tested (Tong et
al., 2015; Kwiecinski & Horswill, 2020). From these criteria, we identified
5976 unique prescriptions in our database. Additionally, patients with
multiple ED admissions that met the criteria were analyzed separately but were
grouped within the same train/test divisions to prevent test set
contamination. This cohort included 10 unique antibiotics, whose prevalences
are shown in Table 1. A demographic overview of our cohort is presented in
Appendix Section B.

Table 1: Antibiotic Prevalence in MIMIC IV Cohort

Antibiotic Train Test Total Prevalence (%) Clindamycin 2645 624 54.69%
Daptomycin 1815 425 37.51% Erythromycin 2626 639 54.59% Gentamicin 4549 1127
94.89% Levofloxacin 2866 715 60.00% Oxacillin 2702 667 56.32% Rifampin 1929
459 39.96% Tetracycline 3747 909 76.57% Trimethoprim/sul 3671 908 71.66%
Vancomycin 2529 611 52.53%

To motivate our experimental setup, we examine the information available about
a patient at their time of arrival in the emergency department. To predict
antibiotic use, we utilize six clinical modalities from the MIMIC ED Database.
These EHR modalities include arrival and triage information, medication
reconciliation (medrecon), diagnostic codes (ICD-9/10), vital signs, and Pyxis
data. All these data points are linked to antibiotic labels from the MIMIC
database using a patient ID, visit, and Hospital Admission ID (Hadm_id),
allowing us to accurately identify the patients and their tests in which
certain antibiotics were effective.

#### Experiments

In this work, we benchmark different representation strategies of EHR to
identify the most effective method for predicting antibiotic susceptibility.
We approach this problem as a multilabel binary classification, where we train
the same base model (Light Gradient Boosted Machines) using various
representation startegies of the input. These representations include: raw
tabular data; EHR-shot (Wornow et al., 2024), a foundation model for tabular
EHR; and three text-based representations: word2vec, a generic language model,
and a medical language model, BioMegatron (Shin et al., 2020). Additionally,
we conduct a clustering of our pseudonotes using the BERTopic algorithm
(Grootendorst, 2022) to determine if these embeddings can naturally cluster
patients. Identifying these clusters can provide insights into their potential
performance in settings like zero-shot learning and provide insights into the
decision making process.

##  4 Results

Figure 2: Area under the Precision Recall Curve (AUPRC) curves for each
antibiotic classification shows that, despite the nuances, BioMegatron
performs the best.

#### Antibiotic Susceptibility Prediction

In our analysis of antibiotic prediction, we measure the Area Under the
Receiver Operating Characteristic curve (AUROC) and Area Under the Precision-
Recall Curve (AUPRC). Additionally, we bootstrap 1,000 times to generate 95%
confidence intervals. Our AUROC and AUPRC results are displayed in Figures 1
and 2. We also measure additional F1 scores and Matthews correlation
coefficients, with a whole table readout which are included in the appendix.

#### Clustering Experiment

In our clustering experiments, we aim to identify clusters using the BERTopic
algorithm. By identifying clusters based on embeddings, we believe this
approach can form the basis for zero-shot applications across various clinical
tasks. Additionally, finding similar embeddings could provide insights into
decision-making processes in these black-box models. We showcase the
similarity matrix of our patient clusters in Figure 3.

##  5 Discussion

#### Clinical Notes with Foundation Models Provide the best representation and
interpretability

Figure 3: Identifying clusters from our patient embeddings is indicated by the
squares forming along the diagonal of our similarity matrix.

From Figures 1 and 2, we observe that the foundation models operating on our
pseudo-notes method provide the best overall performance across most of the
antibiotics. While a generic foundation model and EHR-shot excel with some
antibiotics, the clinical foundation model consistently shows superior
performance across both AUROC and AUPRC metrics.

Beyond enhanced predictive abilities, an advantage of our pseudo-notes method
over EHR foundation models and tabular representations is its
interpretability. Compared to the specific structuring required by EHR-shot,
our method offers a simpler and more effective interface to understand the
data that is being modeled which can improve the trust between AI and
healthcare professionals.

#### Interoperability

Another advantage of pseudo-notes over EHR foundation models is their
interoperability with proprietary healthcare systems. Our method offers a
straightforward interface for converting any EHR tabular data from tables to
text. Our conversion also facilitates the use of off-the-shelf open-source
foundation models. As improved models are continually developed, this data
format provides an easy interface to adapt and swap the backbone for improved
representation of our clinical text. Additionally, EHR foundation models do
not operate on non-OMOP vocabularies, which limits its effectiveness on
datasets like MIMIC-IV that utilize these specialized vocabularies.

#### Patient Similarity

One final advantage of our pseudo-notes is illustrated in Figure 3, which
demonstrates the capability to perform a similarity search on our patient
embeddings. From this analysis, we identified clusters related to sepsis,
diabetes, stomach acid issues, anxiety, painkillers, respiratory conditions,
and antidepressants. This opens up potential use cases for this data
representation strategy to be used in zero-shot learning studies and offers
insights into decision-making processes based on the embeddings. Further
research is needed to explore both of these areas.

##  6 Conclusion

In this work, we introduced a methodology called pseudo-notes, which converts
EHR tabular data into text to achieve an optimal representation strategy. We
discovered that pseudo-notes outperformed various representation strategies
and remains a highly flexible framework, compatible with the ongoing
development of foundation model backbones, which could further enhance its
performance. Additionally, we found that pseudo-notes can identify patient
clusters within the EHR, opening up promising avenues for future studies in
zero-shot learning and model interpretation.

From an application perspective, we demonstrated how a straightforward data
transformation has emerged as an easy interface for making EHR data synergized
before integrating it into machine learning models. We think that this
strategy could be a better way to work with EHR data for future research and
help build trust due to its interpretability. Particularly in this study, we
illustrated its potential by identifying suitable antibiotics for patients
arriving at the ED, where timely and accurate decisions are critical. We as a
group have highlighted the importance of improving antibiotic stewardship and
showcase the impact of data-driven strategies in addressing pressing
healthcare challenges.

### Impact Statement

The goal of this work is to advance the field of Machine Learning in
Healthcare, and thus presents a novel potential interface between modern NLP
(Foundation models) and clinical data.

## References

  * Agniel et al. (2018) Agniel, D., Kohane, I. S., and Weber, G. M.  Biases in electronic health record data due to processes within the healthcare system: retrospective observational study.  _Bmj_ , 361, 2018.
  * Alsentzer et al. (2019) Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., and McDermott, M.  Publicly available clinical bert embeddings.  _arXiv preprint arXiv:1904.03323_ , 2019.
  * Bartlett et al. (2013) Bartlett, J. G., Gilbert, D. N., and Spellberg, B.  Seven ways to preserve the miracle of antibiotics.  _Clinical infectious diseases_ , 56(10):1445–1450, 2013.
  * Cheng et al. (2016) Cheng, Y., Wang, F., Zhang, P., and Hu, J.  Risk prediction with electronic health records: A deep learning approach.  In _Proceedings of the 2016 SIAM international conference on data mining_ , pp. 432–440. SIAM, 2016.
  * Churpek et al. (2014) Churpek, M. M., Yuen, T. C., Park, S. Y., Gibbons, R., and Edelson, D. P.  Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards.  _Critical care medicine_ , 42(4):841–848, 2014.
  * Cowie et al. (2017) Cowie, M. R., Blomster, J. I., Curtis, L. H., Duclaux, S., Ford, I., Fritz, F., Goldman, S., Janmohamed, S., Kreuzer, J., Leenay, M., et al.  Electronic health records to facilitate clinical research.  _Clinical Research in Cardiology_ , 106:1–9, 2017.
  * Evans (2016) Evans, R. S.  Electronic health records: then, now, and in the future.  _Yearbook of medical informatics_ , 25(S 01):S48–S61, 2016.
  * Ferrao et al. (2016) Ferrao, J. C., Oliveira, M. D., Janela, F., and Martins, H. M.  Preprocessing structured clinical data for predictive modeling and decision support.  _Applied clinical informatics_ , 7(04):1135–1153, 2016.
  * Goetz et al. (2024) Goetz, L., Seedat, N., Vandersluis, R., and van der Schaar, M.  Generalization—a key challenge for responsible ai in patient-facing clinical applications.  _npj Digital Medicine_ , 7(1):1–4, 2024.
  * Goldstein et al. (2017) Goldstein, B. A., Navar, A. M., Pencina, M. J., and Ioannidis, J. P.  Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review.  _Journal of the American Medical Informatics Association: JAMIA_ , 24(1):198, 2017.
  * Golkar et al. (2014) Golkar, Z., Bagasra, O., and Pace, D. G.  Bacteriophage therapy: a potential solution for the antibiotic resistance crisis.  _The Journal of Infection in Developing Countries_ , 8(02):129–136, 2014.
  * Gould & Bal (2013) Gould, I. M. and Bal, A. M.  New antibiotic agents in the pipeline and how they can help overcome microbial resistance.  _Virulence_ , 4(2):185–191, 2013.
  * Grootendorst (2022) Grootendorst, M.  Bertopic: Neural topic modeling with a class-based tf-idf procedure.  _arXiv preprint arXiv:2203.05794_ , 2022.
  * Hegselmann et al. (2023) Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D.  Tabllm: Few-shot classification of tabular data with large language models.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 5549–5581. PMLR, 2023.
  * Hoerbst & Ammenwerth (2010) Hoerbst, A. and Ammenwerth, E.  Electronic health records.  _Methods of information in medicine_ , 49(04):320–336, 2010.
  * Hohman et al. (2023) Hohman, K. H., Martinez, A. K., Klompas, M., Kraus, E. M., Li, W., Carton, T. W., Cocoros, N. M., Jackson, S. L., Karras, B. T., Wiltz, J. L., et al.  Leveraging electronic health record data for timely chronic disease surveillance: the multi-state ehr-based network for disease surveillance.  _Journal of Public Health Management and Practice_ , 29(2):162–173, 2023.
  * Johnson et al. (2020) Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., and Mark, R.  Mimic-iv.  _PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021)_ , pp. 49–55, 2020.
  * Johnson et al. (2023) Johnson, A. E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al.  Mimic-iv, a freely accessible electronic health record dataset.  _Scientific data_ , 10(1):1, 2023.
  * Kwiecinski & Horswill (2020) Kwiecinski, J. M. and Horswill, A. R.  Staphylococcus aureus bloodstream infections: pathogenesis and regulatory mechanisms.  _Current opinion in microbiology_ , 53:51–60, 2020.
  * Lee et al. (2020) Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240, 2020.
  * Lee et al. (2024) Lee, S. A., Jain, S., Chen, A., Ono, K., Fang, J., Rudas, A., and Chiang, J. N.  Emergency department decision support using clinical pseudo-notes, 2024.
  * Li et al. (2020) Li, R., Chen, Y., Ritchie, M. D., and Moore, J. H.  Electronic health records and polygenic risk scores for predicting disease risk.  _Nature Reviews Genetics_ , 21(8):493–502, 2020.
  * Li et al. (2022) Li, Y., Mamouei, M., Salimi-Khorshidi, G., Rao, S., Hassaine, A., Canoy, D., Lukasiewicz, T., and Rahimi, K.  Hi-behrt: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records.  _IEEE journal of biomedical and health informatics_ , 27(2):1106–1117, 2022.
  * Liu et al. (2018) Liu, J., Zhang, Z., and Razavian, N.  Deep ehr: Chronic disease prediction using medical notes.  In _Machine Learning for Healthcare Conference_ , pp. 440–464. PMLR, 2018.
  * Liu et al. (2021) Liu, N., Hu, Q., Xu, H., Xu, X., and Chen, M.  Med-bert: A pretraining framework for medical records named entity recognition.  _IEEE Transactions on Industrial Informatics_ , 18(8):5600–5608, 2021.
  * Lushniak (2014) Lushniak, B. D.  Antibiotic resistance: a public health crisis.  _Public Health Reports_ , 129(4):314–316, 2014.
  * Nature (2013) Nature, E.  The antibiotic alarm.  _Nature_ , 495(7440):141, 2013.
  * Pang et al. (2021) Pang, C., Jiang, X., Kalluri, K. S., Spotnitz, M., Chen, R., Perotte, A., and Natarajan, K.  Cehr-bert: Incorporating temporal information from structured ehr data to improve prediction tasks.  In _Machine Learning for Health_ , pp. 239–260. PMLR, 2021.
  * Rasmy et al. (2021) Rasmy, L., Xiang, Y., Xie, Z., Tao, C., and Zhi, D.  Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.  _NPJ digital medicine_ , 4(1):86, 2021.
  * Read & Woods (2014) Read, A. F. and Woods, R. J.  Antibiotic resistance management.  _Evolution, medicine, and public health_ , 2014(1):147, 2014.
  * Sanh et al. (2019) Sanh, V., Debut, L., Chaumond, J., and Wolf, T.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ , 2019.
  * Sengupta et al. (2013) Sengupta, S., Chattopadhyay, M. K., and Grossart, H.-P.  The multifaceted roles of antibiotics and antibiotic resistance in nature.  _Frontiers in microbiology_ , 4:47, 2013.
  * Shin et al. (2020) Shin, H.-C., Zhang, Y., Bakhturina, E., Puri, R., Patwary, M., Shoeybi, M., and Mani, R.  Biomegatron: Larger biomedical domain language model, 2020.
  * Steinberg et al. (2023) Steinberg, E., Fries, J., Xu, Y., and Shah, N.  Motor: A time-to-event foundation model for structured medical records.  _arXiv preprint arXiv:2301.03150_ , 2023.
  * Suter et al. (1994) Suter, P., Armaganidis, A., Beaufils, F., Bonfill, X., Burchardi, H., Cook, D., Fagot-Largeault, A., Thijs, L., Vesconi, S., Williams, A., et al.  Predicting outcome in icu patients.  _Intensive Care Medicine_ , 20:390–397, 1994.
  * Sutton et al. (2020) Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., and Kroeker, K. I.  An overview of clinical decision support systems: benefits, risks, and strategies for success.  _NPJ digital medicine_ , 3(1):17, 2020.
  * Tang et al. (2020) Tang, S., Davarmanesh, P., Song, Y., Koutra, D., Sjoding, M. W., and Wiens, J.  Democratizing ehr analyses with fiddle: a flexible data-driven preprocessing pipeline for structured clinical data.  _Journal of the American Medical Informatics Association_ , 27(12):1921–1934, 2020.
  * Tong et al. (2015) Tong, S. Y., Davis, J. S., Eichenberger, E., Holland, T. L., and Fowler Jr, V. G.  Staphylococcus aureus infections: epidemiology, pathophysiology, clinical manifestations, and management.  _Clinical microbiology reviews_ , 28(3):603–661, 2015.
  * Ventola (2015) Ventola, C. L.  The antibiotic resistance crisis: part 1: causes and threats.  _Pharmacy and therapeutics_ , 40(4):277, 2015.
  * Viswanathan (2014) Viswanathan, V.  Off-label abuse of antibiotics by bacteria.  _Gut microbes_ , 5(1):3–4, 2014.
  * Wolf et al. (2019) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Wornow et al. (2024) Wornow, M., Thapa, R., Steinberg, E., Fries, J., and Shah, N.  Ehrshot: An ehr benchmark for few-shot evaluation of foundation models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Wu et al. (2010) Wu, J., Roy, J., and Stewart, W. F.  Prediction modeling using ehr data: challenges, strategies, and a comparison of machine learning approaches.  _Medical care_ , 48(6):S106–S113, 2010.

##  Appendix A Appendix

###  A.1 Additional Commentary

#### Limitations

Some limitations of this work include the variability in patients’ histories
and the 512 sequence length limitation imposed by the DistilBERT (Sanh et al.,
2019) and BioMegatron models (Shin et al., 2020). Consequently, portions of a
patient’s medical history may be truncated depending on the length of that
history. Tokenization strategies (e.g., sub-word tokenization) can
significantly influence how we handle the analysis.

#### Future Work

Future work in our group, from a methodological perspective, aims to explore
how these notes can enhance studies in model interpretability and zero-shot or
few-shot frameworks. From an application standpoint, we are interested in
applying this methodology across various departments and applications. We plan
to collaborate with clinicians throughout our institution to determine the
types of clinical decision support models that are most needed and to assess
how AI can benefit these healthcare facilities.

Additionally, future work will include benchmarking the plethora of foundation
models available on the Huggingface Platform (Wolf et al., 2019). This will
help us identify the best foundation model for specific tasks and determine
whether these embeddings are task-agnostic.

##  Appendix B Dataset Characteristics

###  B.1 Patient Demographics

Table 2: MIMIC IV Cohort Data Overview
Description | Category | Train | Test | Totals
---|---|---|---|---
Prescription, n | Total | 4803 | 1173 | 5976
Unique ID, n | Total | 3283 | 878 | 4161
Age Mean (SD) |  | 59 (17) | 58 (17) |
Sex % | Female | 1341 | 351 | 1692
| Male | 1942 | 527 | 2469
Race/Ethnicity % | White | 2212 | 583 | 2795
| Black | 416 | 119 | 535
| Other | 401 | 96 | 497
| Hispanic/Latino | 150 | 55 | 205
| Asian | 88 | 20 | 108
| Unable | 12 | 3 | 15
| Native Hawaiian | 4 | 2 | 6

###  B.2 Clinical Modalities

Table 3: Overview of Clinical Modalities in Emergency Department Visits Modality Name |  Description
---|---
Arrival Information |  Records patient demographics, time of arrival, and mode of arrival (e.g., ambulance, walk-in).
Triage Information |  Documents vital signs, severity of condition using scales like ESI, and initial chief complaints upon arrival.
Medication Reconciliation |  Details previous and current medications the patient is taking, including dosages and frequency.
Patient Vitals |  Ongoing measurements throughout the ED visit including heart rate, blood pressure, temperature, etc.
Diagnosis Codes |  ICD-9/10 codes used to classify and record diagnoses during the visit.
Pyxis Information |  Information on medications administered during the ED stay via the Pyxis system, including timing and dosage.

##  Appendix C Results

Table 4: Performance Metrics for Clindamycin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.7179 $\pm$ 0.032
0.7719 $\pm$ 0.019 0.7737 $\pm$ 0.031 0.7786 $\pm$ 0.015 0.7689 $\pm$ 0.029
MCC 0.0914 $\pm$ 0.011 0.4162 $\pm$ 0.0624 0.3561 $\pm$ 0.026 0.3772 $\pm$
0.028 0.3379 $\pm$ 0.022 ROC-AUC 0.6029 $\pm$ 0.044 0.7664 $\pm$ 0.020 0.7263
$\pm$ 0.023 0.7443 $\pm$ 0.030 0.7244 $\pm$ 0.034 PRC-AUC 0.6427 $\pm$ 0.010
0.7859 $\pm$ 0.026 0.7660 $\pm$ 0.013 0.7684 $\pm$ 0.015 0.7653 $\pm$ 0.019

Table 5: Performance Metrics for Daptomycin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.2667 $\pm$ 0.032
0.3704 $\pm$ 0.069 0.3333 $\pm$ 0.050 0.3529 $\pm$ 0.012 0.3529 $\pm$ 0.035
MCC 0.2867 $\pm$ 0.065 0.3584 $\pm$ 0.022 0.3943 $\pm$ 0.041 0.4586 $\pm$
0.058 0.4587 $\pm$ 0.034 ROC-AUC 0.6107 $\pm$ 0.063 0.6651 $\pm$ 0.065 0.60223
$\pm$ 0.070 0.5211 $\pm$ 0.060 0.6708 $\pm$ 0.062 PRC-AUC 0.1107 $\pm$ 0.006
0.1679 $\pm$ 0.015 0.2323 $\pm$ 0.004 0.2319 $\pm$ 0.005 0.2474 $\pm$ 0.006

Table 6: Performance Metrics for Erythromycin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.5495 $\pm$ 0.030
0.6575 $\pm$ 0.023 0.6394 $\pm$ 0.038 0.6592 $\pm$ 0.020 0.6473 $\pm$ 0.025
MCC 0.1306 $\pm$ 0.021 0.3807 $\pm$ 0.029 0.3209 $\pm$ 0.042 0.3702 $\pm$
0.037 0.3406 $\pm$ 0.028 ROC-AUC 0.5879 $\pm$ 0.044 0.7590 $\pm$ 0.022 0.7320
$\pm$ 0.025 0.7597 $\pm$ 0.023 0.7600 $\pm$ 0.025 PRC-AUC 0.4530 $\pm$ 0.017
0.6718 $\pm$ 0.024 0.6754 $\pm$ 0.016 0.6872 $\pm$ 0.012 0.6964 $\pm$ 0.014

Table 7: Performance Metrics for Gentamicin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.9762 $\pm$ 0.030
0.9775 $\pm$ 0.065 0.9776 $\pm$ 0.040 0.9766 $\pm$ 0.045 0.9776 $\pm$ 0.032
MCC 0.2521 $\pm$ 0.055 0.3634 $\pm$ 0.021 0.3953 $\pm$ 0.030 0.3969 $\pm$
0.065 0.3667 $\pm$ 0.035 ROC-AUC 0.6158 $\pm$ 0.089 0.6310 $\pm$ 0.047 0.6727
$\pm$ 0.047 0.6777 $\pm$ 0.042 0.6523 $\pm$ 0.039 PRC-AUC 0.9706 $\pm$ 0.036
0.9672 $\pm$ 0.004 0.9675 $\pm$ 0.002 0.9713 $\pm$ 0.002 0.9678 $\pm$ 0.011

Table 8: Performance Metrics for Levofloxacin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.7641 $\pm$ 0.028
0.8386 $\pm$ 0.017 0.8088 $\pm$ 0.012 0.8034 $\pm$ 0.013 0.8066 $\pm$ 0.013
MCC 0.1766 $\pm$ 0.025 0.5094 $\pm$ 0.015 0.4302 $\pm$ 0.025 0.4260 $\pm$
0.017 0.4261 $\pm$ 0.017 ROC-AUC 0.6326 $\pm$ 0.034 0.7972 $\pm$ 0.017 0.7787
$\pm$ 0.021 0.7974 $\pm$ 0.018 0.7937 $\pm$ 0.021 PRC-AUC 0.7324 $\pm$ 0.013
0.8290 $\pm$ 0.014 0.8157 $\pm$ 0.011 0.8459 $\pm$ 0.012 0.8449 $\pm$ 0.014

Table 9: Performance Metrics for Oxacillin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.7264 $\pm$ 0.027
0.8229 $\pm$ 0.024 0.7899 $\pm$ 0.018 0.7790 $\pm$ 0.023 0.7975 $\pm$ 0.014
MCC 0.2012 $\pm$ 0.015 0.4955 $\pm$ 0.017 0.4456 $\pm$ 0.021 0.4028 $\pm$
0.020 0.4674 $\pm$ 0.018 ROC-AUC 0.5607 $\pm$ 0.027 0.7996 $\pm$ 0.016 0.7688
$\pm$ 0.017 0.7692 $\pm$ 0.013 0.7723 $\pm$ 0.015 PRC-AUC 0.6069 $\pm$ 0.011
0.8408 $\pm$ 0.018 0.7847 $\pm$ 0.019 0.7807 $\pm$ 0.018 0.7960 $\pm$ 0.017

Table 10: Performance Metrics for Rifampin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.5619 $\pm$ 0.026
0.6250 $\pm$ 0.024 0.6582 $\pm$ 0.012 0.6455 $\pm$ 0.017 0.6434 $\pm$ 0.018
MCC $\geq$0.0000 $\pm$ 0.000 0.4136 $\pm$ 0.027 0.3907 $\pm$ 0.012 0.3599
$\pm$ 0.017 0.3583 $\pm$ 0.021 ROC-AUC 0.5083 $\pm$ 0.026 0.7634 $\pm$ 0.015
0.7691 $\pm$ 0.015 0.7553 $\pm$ 0.016 0.7644 $\pm$ 0.015 PRC-AUC 0.4082 $\pm$
0.002 0.6927 $\pm$ 0.011 0.7017 $\pm$ 0.013 0.6940 $\pm$ 0.011 0.6999 $\pm$
0.012

Table 11: Performance Metrics for Tetracycline

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.8950 $\pm$ 0.025
0.9009 $\pm$ 0.024 0.9028 $\pm$ 0.027 0.9035 $\pm$ 0.023 0.9049 $\pm$ 0.021
MCC 0.1657 $\pm$ 0.025 0.3805 $\pm$ 0.012 0.3696 $\pm$ 0.015 0.3795 $\pm$
0.017 0.3865 $\pm$ 0.021 ROC-AUC 0.5822 $\pm$ 0.035 0.6908 $\pm$ 0.018 0.6843
$\pm$ 0.023 0.6843 $\pm$ 0.025 0.6933 $\pm$ 0.023 PRC-AUC 0.8467 $\pm$ 0.004
0.8571 $\pm$ 0.005 0.8717 $\pm$ 0.005 0.8760 $\pm$ 0.003 0.8822 $\pm$ 0.002

Table 12: Performance Metrics for Trimethoprim/sulfa

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.8835 $\pm$ 0.018
0.8856 $\pm$ 0.027 0.9080 $\pm$ 0.032 0.9080 $\pm$ 0.024 0.9100 $\pm$ 0.025
MCC $\geq$ 0.0000 $\pm$ 0.000 0.3785 $\pm$ 0.023 0.4070 $\pm$ 0.031 0.4162
$\pm$ 0.023 0.4321 $\pm$ 0.032 ROC-AUC 0.5393 $\pm$ 0.031 0.7026 $\pm$ 0.016
0.7025 $\pm$ 0.018 0.6946 $\pm$ 0.027 0.7122 $\pm$ 0.026 PRC-AUC 0.8159 $\pm$
0.017 0.8707 $\pm$ 0.015 0.8748 $\pm$ 0.004 0.8742 $\pm$ 0.004 0.8815 $\pm$
0.008

Table 13: Performance Metrics for Vancomycin

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron F1 0.6786 $\pm$ 0.021
0.7201 $\pm$ 0.016 0.7227 $\pm$ 0.015 0.7244 $\pm$ 0.014 0.7287 $\pm$ 0.023
MCC 0.1433 $\pm$ 0.026 0.3342 $\pm$ 0.023 0.3382 $\pm$ 0.026 0.3370 $\pm$
0.025 0.3555 $\pm$ 0.024 ROC-AUC 0.5431 $\pm$ 0.020 0.7566 $\pm$ 0.014 0.7449
$\pm$ 0.011 0.7542 $\pm$ 0.012 0.7610 $\pm$ 0.013 PRC-AUC 0.5537 $\pm$ 0.005
0.7781 $\pm$ 0.018 0.7676 $\pm$ 0.015 0.7754 $\pm$ 0.019 0.7708 $\pm$ 0.006

Table 14: Number of Winning Metrics

Metric Tabular EHR-shot Word2Vec DistilBERT BioMegatron Total Number 0 13 3 7
17 40