# Toward Pioneering Sensors and Features
Using Large Language Models in Human Activity Recognition

HARU KANEKO  0000-0002-1881-5182 Kyushu Institute of Technology2-4,
Hibikino,Wakamatsu wardKitakyushuFukuokaJapan<EMAIL_ADDRESS>and  SOZO INOUE  0000-0003-1109-8130 Kyushu Institute of Technology2-4,
Hibikino,Wakamatsu wardKitakyushuFukuokaJapan<EMAIL_ADDRESS>
(2018)

###### Abstract.

In this paper, we propose a feature pioneering method using Large Language
Models (LLMs). In the proposed method, we use ChatGPT
111https://chat.openai.com/chatm to find new sensor locations and new
features. Then we evaluate the machine learning model which uses the found
features using Opportunity Dataset (Roggen et al., 2010; Chavarriaga et al.,
2013). In current machine learning, humans make features, for this engineers
visit real sites and have discussions with experts and veteran workers.
However, this method has the problem that the quality of the features depends
on the engineer. In order to solve this problem, we propose a way to make new
features using LLMs. As a result, we obtain almost the same level of accuracy
as the proposed model which used fewer sensors and the model uses all sensors
in the dataset. This indicates that the proposed method is able to extract
important features efficiently.

Human Activity Recognition, ChatGPT, Feature Engineering, Machine Learning

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation email; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Human Activity Recognition (HAR) using wearables refers to the process of
automatically identifying and classifying human activities based on data
collected from wearable devices. These wearables, which include smartwatches,
fitness trackers, and motion sensors, are outfitted with sensors such as
accelerometers, gyroscopes, and heart rate monitors (Lara and Labrador, 2012;
Das Antar et al., 2019). These sensors record information about the user’s
movements, body positions, and physiological responses. HAR algorithms analyze
this data to recognize and categorize activities such as walking, running,
cycling, sleeping, and others. HAR systems utilize machine learning techniques
to learn patterns, create accurate real-time models, and differentiate between
various activities by selecting informative and discriminative features.

However, developing features from raw data requires both domain knowledge and
human efforts. Later, choosing features for the best fit can take time as
well. Despite these efforts, contemporary machine learning methods
predominantly rely on features manually crafted by humans to determine which
values are pertinent for estimation. In particular, to reflect the
field/domain-specific knowledge in the features, the people visited the field
where the data was from and had discussions with specialists/veteran workers
in that field, when they made features. However, those reflections of domain
knowledge by such methods depend on the experience and knowledge of the data
scientists who create features.

In this paper, we propose a feature pioneering method using Large Language
Models (LLMs). We input ChatGPT about the machine learning task which is we
want to do and the current sensor location and ask ChatGPT where we should
install new sensors in the human body. Then based on the output of ChatGPT, we
add new sensors and new features.

In the experiment, we assume that only elbow sensor data are used for activity
recognition, then ask the location of sensors that should be added to ChatGPT.
As a result, we got 7 new sensor locations and 14 new features suggested by
ChatGPT. Then we evaluate the human activity recognition model using
Opportunity Dataset (Roggen et al., 2010; Chavarriaga et al., 2013) which mode
of locomotion. In the result of HAR models, we obtain almost the same level of
accuracy as the proposed model which used fewer sensors and the model uses all
sensors in the dataset. This indicates that the proposed method is able to
extract important features efficiently.

##  2\. Background

Basically, machine learning uses features created by humans to figure out what
values are useful for estimation. Additionally, to reflect the field/domain-
specific knowledge in the features, people visited the field where the data
was from, and had discussions with specialists/veteran workers in that field,
when they make features. However, those reflections of domain knowledge by
such methods rely on the experience and knowledge of the data scientists who
create features.

In the machine learning area, also in the human activity recognition area,
there are methods to evaluate the effectiveness of sensors in prediction
models, there are methodologies available in the HAR field. Examining feature
importance is one method of doing this (Ahmed et al., 2020; Thakur and Biswas,
2022) or SHAP value(SHapley Additive exPlanation) (Bragança et al., 2022;
Dubey et al., 2022). Utilizing a technique called ”feature importance
evaluation”, researchers can determine the importance and relevance of
specific features or sensors in terms of how much they contribute to a model’s
accuracy and prognostication. This evaluation process helps determine how well
sensors capture pertinent data for activity recognition by quantifying the
influence and impact of various features. It enables the creation of robust
prediction models by enabling researchers to prioritize and pick the most
important features or sensors. The machine learning community can choose
sensors and create features with greater knowledge by using feature importance
evaluation, which will ultimately improve performance. However, these methods
can only use after data collection.

Recently, Large language models (LLMs) like GPT-4, which are incredibly
powerful, are now being used in a variety of applications, including
productivity suites, code generation tools, and search engines. These
generative models are anticipated to have an impact across industries,
altering how we approach tasks like writing, programming, design, and more
(Agrawal et al., 2022; Felten et al., 2023). Such, prompt engineering has been
attracting a lot of attention. This is a technique that optimizes the input
text and improves the quality of the output when using text input as the UI
for a language model called LLM. Due to the increasing performance of LLMs,
many prompt patterns have been developed (White et al., 2023a, b).

For this reason, in this paper, we aim to solve the problem that the features
making are depending on the skill of engineers, and we propose prompts for
pioneering features. For this, we make a prompt to find new sensor locations
and new features for human activity recognition using ChatGPT. Then, assuming
2 cases the case before data collection and the case after data collection, we
propose a feature pioneering method for both cases. In the case of before data
collection, the proposed prompts use only information from the task of
activity recognition, and in the case of after data collection, the proposed
prompt also uses additional information from the result of the human activity
recognition model.

##  3\. Method

In this section, we propose a feature pioneering prompt for LLMs and evaluate
the machine learning model which uses features funded by the proposed method.
For this evaluation, we use the Opportunity Dataset which is human activities
of daily living (ADL) dataset including activity labels and IMU/accelerometer
sensor data.

###  3.1. Problem Setup

The problem setup is the case where data collection and machine learning have
already been done, then want to further data collection to improve the
accuracy. In this case, we propose the sensor pioneering method (in section
3.2) and the feature augmentation method (in section 3.3).

In the sensor pioneering method, we propose to find new sensor locations that
are the locations of human surfaces to be useful for recognizing human
activity. In the feature augmentation method, we propose to find new feature
calculations such as mean, variance, etc. for recognizing human activity.

###  3.2. Sensor Pioneering

Our prompt is made up of the following 5 structures, (1) “Your role”, (2) “The
problem you need to solve”, (3) “Output activity labels”, (4) “Current
features”, (5) “Current result”, and (6) “Your task”.

#### (1) “Your role” :

Here defines the position of the skilled person from whom you wish to obtain
domain knowledge.

#### (2) “The problem you need to solve” :

Here explain what kind of machine learning task. Explain what you want to do,
what the task is, and what data is available. At this point, use specialized
words such as IMU and acc.

#### (3) “Output activity labels” :

Here we show the prediction labels. In this case, the labels are shown as
words rather than IDs, etc., so that the meaning of the labels can be
understood.

#### (4) “Current features” :

Here we explain the current location of the sensor. At this point, explain
using the words of position in the body.

#### (5) “Current result” :

Here we describe the results of the current prediction model. In particular,
in order to obtain features which is effective for ignoring misclassification,
we describe the case of often misses classification based on the confusion
matrix.

#### (6) “Your task” :

Here we indicate what you want ChatGPT to output. Since the purpose of this
paper is to extend features, we specify that new features should be output.

###  3.3. Feature Augmentation

This prompt is made up of 4 structures, and the following 2 structures (1)
“The problem you need to solve”, and (2) “Output activity labels” are the same
with section 3.2. Then, (3) “Current features” and (4) “Your task” are changed
as below.

#### (3) “Current features” :

Here we explain the current computed features. At this point, explain using
the words of position in the body.

#### (4) “Your task” :

Here we indicate what you want ChatGPT to output. Since the purpose of this
paper is to extend features, we specify that new features should be output.

##  4\. Experiment

In this section, in order to evaluate the proposed prompts, we make a machine
learning model to perform human activity recognition (HAR) and evaluate the
accuracy and F1-score.

###  4.1. Experiment setting

In the experiment, we use the Opportunity dataset (Roggen et al., 2010;
Chavarriaga et al., 2013). This dataset includes data from 5 subjects, from
accelerometers and IMU sensors attached to the full body. Additionally, this
dataset has 5 classes of activity labels, which is “Stand”, “Sit”, “Walk”,
“Lie”, and “Others”. The machine learning model identifies these 5 activity
classes from sensor data.

The experiment will be conducted using GPT-4 (May 24 Version), a feature of
ChatGPT Plus. The input language is English. In the sensor pioneering
experiment, we use a new chat session which not have any chat history, with no
history of the previous prompt. In the feature augmentation experiment, we use
chat to continue the sensor pioneering experiment. In those experiments, we
assume that the data collection and modeling were done using only elbow sensor
data before inputting the prompt into ChatGPT.

In the pre-processing, missing values were complemented with the average of
the before and after values. For segmentation, we use the sliding window
method with a window size of 5 seconds and an overlap rate of 30%.

###  4.2. Result

In this section, we evaluate the results of the experiment and the accuracy of
the activity recognition model created based on the experimental results.
Figure 1 \- 4 shows the input prompt which is described in section 3, and the
output result from ChatGPT.

####  4.2.1. Sensor Pioneering :

In the experiment, we enter the 2 types of prompts, A) without (5) “Current
results” and B) with (5) “Current results”. Both of those promote entered into
the initialized prompts. Figure 1 and 2 is the result of input without (5)
“Current results”, figure 3 and 4 is the result of input with (5) “Current
results”. In the baseline, we use the following 4 sensors [”RUA”̂, ”LUA”̂,
”RUA_”, ”LUA_”].

As shown in figure 1 and 2, when we use prompt A) which does not include (5)
“Current results”, 6 new sensor locations are suggested. Based on this, we
decided to add the following 9 sensors [”R-SHOE”, ”L-SHOE”, ”RWR”, ”LWR”,
”RUA_”, ”RUA”̂, ”LUA_”, ”LUA”̂, ”HIP”].

As shown in figure 3 and 4, when we use prompt B) which includes (5) “Current
results”, 7 new sensor locations are suggested. Based on this, we decided to
add those following 10 sensors [”R-SHOE”, ”L-SHOE”, ”RLA”, ”LLA”, ”BACK”,
”RUA”, ”LUA”, ”RWR”, ”LWR”, ”HIP”]. Comparing these figures, we can know that
we can get more additional sensor suggestions when we input with (5) “Current
results”.

Figure 1. Input prompt for sensor pioneering in human activity recognition (
A: without (5) “Current results” ). Figure 2. Output from ChatGPT for sensor
pioneering in human activity recognition ( A: without (5) “Current results” ).
Figure 3. Input prompt for sensor pioneering in human activity recognition (
B: with (5) “Current results” ). Figure 4. Output from ChatGPT for sensor
pioneering in human activity recognition ( B: with (5) “Current results” ).

####  4.2.2. Feature Augmentation :

Figure 5 \- 6 shows the input prompt for feature augmentations proposed in
section 3, and the output result of ChatGPT. This result is the output when we
input the feature augmentation prompt after the B) sensor pioneering prompt
with (5) “Current results”. In the baseline, we use the following 5 features
“mean”, “standard deviation”, “variance”, “min”, and “max”.

As shown in figure 6, we got 14 types of features. Among the ChatGPT output,
we added the following 10 features that are easy to calculate, “(1) Signal
Magnitude Area (SMA)”, “(2) Energy”, “(3) Entropy”, “(4) Zero Crossing Rate”,
“(5) Mean Crossing Rate”, “(7) Fast Fourier Transform (FFT) Coefficients”,
“(8) Correlation between axes”, “(10) Pitch and Roll”, “(11) Jerk”, and “(12)
Peak Frequency” .

Figure 5. Input prompt for feature augmentation in human activity recognition.
Figure 6. Output from ChatGPT for feature augmentation in human activity
recognition.

####  4.2.3. Estimation Model :

In this section, we evaluate the model using new sensors and features made by
the proposed prompt. Table 1 are shown the Accuracy and F1-score. “Sensor
Pioneering A” means the prompts without (5) “Current results”, and “Sensor
Pioneering B” means the prompts with (5) “Current results”.

In “(a) Baseline”, “(b) All sensors”, and “(d) Sensor Pioneering B” the
following features are used, “mean”, “standard deviation”, “variance”, “min”,
and “max” which are the same when we describe the current features in the
input prompt.

First, compared to “(a) Baseline”, “(d) Sensor Pioneering A + Feature
Augmentation”, and “(f) Sensor Pioneering B + Feature Augmentation” has higher
accuracy. This is indicating that the proposed method is able to improve the
accuracy.

Next, if we focus on evaluating the sensor pioneering method, “(e) Sensor
Pioneering B” has higher accuracy than “(a) Baseline”. On the other hand, “(c)
All sensors + Feature Augmentation” has higher accuracy than “(f) Sensor
Pioneering B + Feature Augmentation”. However, the difference is not very
large.

Finlay if we focus on evaluating the feature augmentation method, “(f) Sensor
Pioneering B + Feature Augmentation” has higher accuracy than “(e) Sensor
Pioneering B”. This is the same in the comparison between “(b) All sensors”
and “(c) All sensors + Feature Augmentation”. This indicates that accuracy can
be improved by adding features using the proposed method.

Table 1. Accuracy% / F1-score% comparison between the model using features calculated by the proposed method and previous studies. Model | Average
---|---
(a) Baseline | 74.3% / 75.4%
(b) All sensors | 81.2% / 82.9%
(c) All sensors + Feature Augmentation | 83.3% / 84.8%
(d) Sensor Pioneering A + Featrue Augmentation | 78.5% / 80.6%
(e) Sensor Pioneering B | 80.5% / 82.0%
(f) Sensor Pioneering B + Featrue Augmentation | 81.3% / 82.8%

##  5\. Discussion

We obtain the accuracy and F1-score of the All sensors model and the model
made by the proposed prompt are not significantly different in table 1. This
is also the same in both of the comparisons by subject or by average. In
addition, while the “(b) All sensors” mode uses 18 sensors, the “(e) Sensor
Pioneering B” only uses 11 sensors. This means that the proposed prompt is
possible to efficiently find important sensor locations and features.

In the feature augmentation, “(f) Sensor Pioneering B + Feature Augmentation”
has higher accuracy than “(e) Sensor Pioneering B”. In other words, the
overall accuracy is better when features are added using the proposed method.
However, the improvement is very small, and hardly any changes at all. In the
experiment, we use initialized prompts with no previous conversation history
when we input the prompt. However, as the experiment progressed, it became
changed that ChatGPT was showing more and more suggestions which is new sensor
locations and new feature augmentations. During this, all experiments were
conducted using ChatGPT May 24 Version, and we do not know the reason why the
number of suggestions increased. However, one possibility is that my other
chat history was used as background information for input in ChatGPT.

##  6\. Conclusion

In this paper, we propose a feature engineering method using Large Language
Models (LLMs) in order to efficient sensing and feature selection. As a
result, we got new sensor locations from ChatGPT. Then, we make a machine
learning model by using the proposed method. Although the number of sensors in
the model made by the proposed method is small, the model shows almost the
same accuracy as the mode by the all of sensors in the Opportunity dataset.
This indicates that the proposed method is able to extract important features
efficiently. In addition, in the augmentation of the features, we obtained new
feature calculations, and show that the prediction accuracy could be improved
over the existing method.

Our proposed method can be used to know the critical sensing location before
sensing. However, when looking at the output of ChatGPT, the output describes
features that are important in general human activity recognition. Therefore,
in order to support special types of behavior, it may be necessary to devise a
way to input activity labels name with more detailed descriptions instead of
label names.

## References

  * (1)
  * Agrawal et al. (2022) Ajay Agrawal, Joshua Gans, and Avi Goldfarb. 2022\.  ChatGPT and how AI disrupts Industries.  _Harvard Business Review_ (2022), 1–6.
  * Ahmed et al. (2020) Nadeem Ahmed, Jahir Ibna Rafiq, and Md Rashedul Islam. 2020\.  Enhanced Human Activity Recognition Based on Smartphone Sensor Data Using Hybrid Feature Selection Model.  _Sensors_ 20, 1 (2020).  https://doi.org/10.3390/s20010317
  * Bragança et al. (2022) Hendrio Bragança, Juan G. Colonna, Horácio A. B. F. Oliveira, and Eduardo Souto. 2022.  How Validation Methodology Influences Human Activity Recognition Mobile Systems.  _Sensors_ 22, 6 (2022).  https://doi.org/10.3390/s22062360
  * Chavarriaga et al. (2013) Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R. Millán, and Daniel Roggen. 2013\.  The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition.  _Pattern Recognition Letters_ 34, 15 (2013), 2033–2042.  https://doi.org/10.1016/j.patrec.2012.12.014 Smart Approaches for Human Action Recognition.
  * Das Antar et al. (2019) Anindya Das Antar, Masud Ahmed, and Md Atiqur Rahman Ahad. 2019\.  Challenges in Sensor-based Human Activity Recognition and a Comparative Analysis of Benchmark Datasets: A Review. In _2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR)_. 134–139.  https://doi.org/10.1109/ICIEV.2019.8858508
  * Dubey et al. (2022) Anand Dubey, Niall Lyons, Avik Santra, and Ashutosh Pandey. 2022\.  XAI-BayesHAR: A novel Framework for Human Activity Recognition with Integrated Uncertainty and Shapely Values. In _2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA)_. 1281–1288.  https://doi.org/10.1109/ICMLA55696.2022.00205
  * Felten et al. (2023) Ed Felten, Manav Raj, and Robert Seamans. 2023.  How will Language Modelers like ChatGPT Affect Occupations and Industries?  _arXiv preprint arXiv:2303.01157_ (2023).
  * Lara and Labrador (2012) Oscar D Lara and Miguel A Labrador. 2012.  A survey on human activity recognition using wearable sensors.  _IEEE communications surveys & tutorials_ 15, 3 (2012), 1192–1209.
  * Roggen et al. (2010) Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, Jakob Doppler, Clemens Holzmann, Marc Kurz, Gerald Holl, Ricardo Chavarriaga, Hesam Sagha, Hamidreza Bayati, Marco Creatura, and José del R. Millàn. 2010.  Collecting complex activity datasets in highly rich networked sensor environments. In _2010 Seventh International Conference on Networked Sensing Systems (INSS)_. 233–240.  https://doi.org/10.1109/INSS.2010.5573462
  * Thakur and Biswas (2022) Dipanwita Thakur and Suparna Biswas. 2022.  An Integration of feature extraction and Guided Regularized Random Forest feature selection for Smartphone based Human Activity Recognition.  _Journal of Network and Computer Applications_ 204 (2022), 103417\.  https://doi.org/10.1016/j.jnca.2022.103417
  * White et al. (2023a) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023a.  A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.  arXiv:2302.11382 [cs.SE]
  * White et al. (2023b) Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023b.  ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design.  arXiv:2303.07839 [cs.SE]


Institut für Optik und Atomare Physik, Technische Universität Berlin, Strasse
des 17. Juni 135, 10623 Berlin, Germany Laboratory for Thin Films and
Photovoltaics, Swiss Federal Laboratories for Materials Science and
Technology, Überlandstrasse 129, CH-8600 Dübendorf, Switzerland Institut für
Optik und Atomare Physik, Technische Universität Berlin, Strasse des 17. Juni
135, 10623 Berlin, Germany

# Direct observation of ultrafast lattice distortions during exciton-polaron
formation in lead-halide perovskite nanocrystals

Hélène Seiler  Fritz Haber Institute of the Max Planck Society, Faradayweg
4-6, 14195 Berlin, Germany<EMAIL_ADDRESS>Daniela Zahn  Fritz Haber
Institute of the Max Planck Society, Faradayweg 4-6, 14195 Berlin, Germany
Victoria C. A. Taylor  Fritz Haber Institute of the Max Planck Society,
Faradayweg 4-6, 14195 Berlin, Germany  Maryna I. Bodnarchnuk  Laboratory for
Thin Films and Photovoltaics, Swiss Federal Laboratories for Materials Science
and Technology, Überlandstrasse 129, CH-8600 Dübendorf, Switzerland  Yoav W.
Windsor  Fritz Haber Institute of the Max Planck Society, Faradayweg 4-6,
14195 Berlin, Germany  Maxsym V. Kovalenko  Institute of Inorganic Chemistry,
Department of Chemistry and Applied Biosciences, ETH Zürich, CH-8093 Zürich,
Switzerland.  Ralph Ernstorfer  Fritz Haber Institute of the Max Planck
Society, Faradayweg 4-6, 14195 Berlin, Germany<EMAIL_ADDRESS>
###### Abstract

The microscopic origin of slow carrier cooling in lead-halide perovskites
remains debated, and has direct implications for applications. Slow carrier
cooling has been attributed to either polaron formation or a hot-phonon
bottleneck effect at high excited carrier densities (> 1018 cm-3). These
effects cannot be unambiguously disentangled from optical experiments alone.
However, they can be distinguished by direct observations of ultrafast lattice
dynamics, as these effects are expected to create qualitatively distinct
fingerprints. To this end, we employ femtosecond electron diffraction and
directly measure the sub-picosecond lattice dynamics of weakly confined
CsPbBr3 nanocrystals following above-gap photo-excitation. The data reveal a
light-induced structural distortion appearing on a time scale varying between
380 fs to 1200 fs depending on the excitation fluence. We attribute these
dynamics to the effect of exciton-polarons on the lattice, and the slower
dynamics at high fluences to slower hot carrier cooling, which slows down the
establishment of the exciton-polaron population. Further analysis and
simulations show that the distortion is consistent with motions of the
[PbBr3]- octahedral ionic cage, and closest agreement with the data is
obtained for Pb-Br bond lengthening. Our work demonstrates how direct studies
of lattice dynamics on the sub-picosecond timescale can discriminate between
competing scenarios, thereby shedding light on the origin of slow carrier
cooling in lead-halide perovskites.

Lead halide perovskites (LHPs) have attracted significant attention for their
remarkable opto-electronic properties, in particular their unusual
photovoltaic performance 1, 2, 3, 4. There is ongoing debate over the origin
of long carrier lifetimes observed in LHPs, which is of direct relevance to
applications. One explanation is screening by large polaron formation, which
may protect carriers from scattering by phonons and defects 5, 6, 7, 8. At
high excitation densities (> 1018 cm-3), a hot-phonon bottleneck effect has
also been considered to explain the observed slower carrier cooling rates. In
such a scenario, a strongly nonthermal population of LO phonons generated by
electron-phonon coupling remains out-of-equilibrium with other phonons for
several picoseconds 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21. These
two scenarios are expected to give rise to qualitatively different lattice
dynamics, and can therefore be distinguished by such observations. Hence
having direct experimental access to the lattice dynamics of LHPs can enable
elucidating the microscopic origin of the slow carrier dynamics in LHPs.

Time-resolved diffraction techniques are ideally-suited for this task. They
offer the most direct measurement of nonthermal phonon populations in photo-
excited materials 22, 23, 24. Recently, they have also emerged as powerful
methods to probe polaronic effects 25, 26, 27. Several time-resolved
diffraction studies have already reported light-induced lattice dynamics of
the soft lattice in LHPs 28, 29, 25, 30. Femtosecond electron diffraction
(FED) was successfully employed to monitor the formation of a rotationally
disordered halide octahedral structure over several picoseconds 28. More
recently, time-resolved X-ray diffuse scattering revealed transient strain
fields building over tens of picoseconds after polaron formation 25. Using
time-resolved X-ray absorption spectroscopy, Cannelli and co-workers were able
to identify the photo-induced polaronic distortion of the lattice tens of
picoseconds after photo-excitation 30. These works clearly demonstrate the
benefits of direct structural probes of the soft LHP lattice. However, while
these studies have mainly focused on processes on several picosecond
timescales, investigating the sub-picosecond lattice dynamics is extremely
relevant as well, as competition between hot-carrier thermalization and
polaron formation is expected to occur on these timescales.

Here we employ femtosecond electron diffraction (FED) to probe the sub-
picosecond lattice dynamics in weakly confined CsPbBr3 nanocrystals (NCs)
after photo-excitation above the electronic band gap. Both hot electron
cooling via the lattice and polaron formation can be expected to occur under
our excitation conditions 31, 32. The data directly reveal the emergence of a
light-induced structural distortion, which builds up with a time constant
ranging from 380 to 1200 femtoseconds depending on the excitation density (0.7
to 5.6 $\times$ 1019 cm-3). This observation is consistent with the
establishment of an exciton-polaron population in the NCs. Combining structure
factor analysis and simulations of diffraction patterns for distorted
structures, we find that our data are qualitatively consistent with specific
motions of the [PbBr3]- octahedral cage, in particular Pb-Br2 bond lengthening
(see Figure 1(a)). Furthermore, all the observables in our data are well-
modeled by a similar sub-picosecond time constant. The fluence dependence of
this sub-picosecond time constant can be explained by slower hot carrier
cooling at high fluences, reported in several previous studies 33, 10, 19, 34.
These results suggest hot electron cooling and the creation of an exciton-
polaron population occur in a coupled fashion. Our work demonstrates the value
of measuring the lattice dynamics directly to probe the interplay of the
various competing effects at the origin of long carrier lifetimes in LHPs.

##  1 Results and Discussion

CsPbBr3 nanocrystals were synthesized following previously established
procedures 2, 35. The linear absorption spectrum of the NCs dispersed in
toluene is shown as the black curve in Figure 1(b), featuring a bandgap of 2.5
eV (496 nm). The inset of this panel shows a representative TEM image of the
nanocrystals. The size of the nanocrystals is $\simeq$ 10 nm, indicating weak
quantum confinement effects, since the exciton Bohr radius is $\simeq$ 7 nm
for CsPbBr3 2. The linear photoluminescence spectrum, shown as the solid green
line in Figure 1(b), is redshifted by a Stokes shift of about 30 meV.

Figure 1: (a) Orthorhombic crystal structure of CsPbBr3 from Ref. 36, with the
two inequivalent Bromine atoms labeled. (b) Linear absorption (plain black
line) and photoluminescence (filled green) spectra of the CsPbBr3 NCs
dispersed in toluene. Inset: TEM picture showing the NCs in real space. (c)
Schematic illustration of the FED experiment, with an example diffraction
pattern of the NCs as collected by our detector. (d) Diffraction profile of
the CsPbBr3 NCs (dark line), obtained by azimuthally averaging the pattern
shown in (c). An empirical function was employed to remove background
contributions. The orange curve represents the simulated pattern using the
structure from Ref. 36 and assuming the NCs lie flat on their faces.

Following basic optical characterisation of the samples, the NCs were drop-
casted on a 10 nm thick Quantifoil TEM membrane (Plano GmbH) for the FED
measurements. The NCs film thickness is estimated to be around 60 nm based on
transmission measurements performed in an optical microscope with a narrow
bandpass filter at 400 nm and previously determined values of intrinsic
absorption coefficients in CsPbBr3 nanocrystals 37. An example of an
equilibrium transmission electron diffraction pattern of the perovskite NCs is
presented in Figure 1(c). Due to averaging over a wide range of orientations
of the NCs probed by the electron beam, the diffraction pattern exhibits
Debye-Scherrer rings typical of polycrystalline samples. For further analysis,
the diffraction pattern is azimuthally averaged and the inelastic background
arising from the substrate is removed (see supplementary Figure 1). An
azimuthally averaged and background-subtracted diffraction profile is shown in
Figure 1(d).

The thermal equilibrium structure of perovskite NCs is characterized by a
complex structural landscape, featuring local polar fluctuations among
different noncubic structures 38, significant local distortions of the PbX6
octahedra 39, structural defects and twin boundaries 40. We find that the
experimental pattern in Figure 1(d) is best reproduced by simulating the
pattern for the $Pnma$ orthorhombic structure, assuming that the NCs lie on
one of their faces 36 (see Supplementary Figure 2). The simulated pattern is
shown as the orange curve in Figure 1(d). Within the limit of the coherence
length of our electron beam, the positions of the Bragg reflections in our
measured diffraction pattern are consistent with the simulated pattern as well
as previous experimental studies 29, 41. The Miller indices corresponding to
the peaks are labeled in Figure 1(d). In the remainder of this work we will
refer to the peaks as 1-5 for convenience.

###  1.1 FED results

FED was previously applied successfully to other types of NCs 42, 43, 44, 45,
46. A schematic illustration of the experiment is shown in Figure 1(c): a
femtosecond laser pulse is used to impulsively excite the electrons in the
material. After a controllable time delay $t$, an electron pulse diffracts off
the lattice. The resulting diffraction pattern encodes the non-equilibrium
state of the lattice at $t$. By varying the time delay between the pump and
the probe, the ultrafast lattice dynamics following photo-excitation can be
monitored. Further details about the FED instrument are available elsewhere
47. Here, the CsPbBr3 NCs are photo-excited with a 50-fs light pulse with
central photon energy h$\nu$ = 3.1 eV (400 nm), roughly 0.6 eV above band
edge. All measurements are performed at room temperature. The incident fluence
on the sample is varied in the range from 0.09 to 0.70 mJ/cm2, and the
resulting initial density of photo-excited carriers induced by the pump pulse
is estimated to be in the range from ne = 0.7 - 5.6$\times$ 1019 cm-3 (see
Supporting Information). At these carrier densities, we estimate that each NCs
hosts multiple excited charge carriers (see Supporting Information). After
photo-excitation of the CsPbBr3 NCs, excited electrons and holes are
generated. These hot charge carriers subsequently couple to the lattice via
electron-phonon coupling, and we follow the ensuing lattice dynamics by
investigating changes in the diffraction patterns as a function of pump-probe
delay.

Figure 2: (a) Same as in Figure 1(d), reproduced for convenience. (b) Relative
intensity difference map, shown here for an excitation density of 2.8$\times$
1019 cm-3.

Figure 2 presents an overview of the photo-induced lattice dynamics, in the
form of relative intensity difference maps. These difference maps are obtained
as [I(t) - I(t<t0)]/I(t<t0), where I(t) is the diffraction profile at time
delay t and t0 is time zero. As shown in Supplementary Figure 3, the observed
lattice dynamics remain qualitatively the same for all measured excitation
densities. We verified that no time-resolved signal could be detected from the
Quantifoil substrate (Supplementary Figure 4) under the same experimental
conditions. In addition, the observed dynamics are reproducible over multiple
scans acquired at different laboratory times (Supplementary Figure 5). The
data in Figure 2 reflect complex lattice dynamics in addition to simple
lattice heating. The latter was estimated to be only about 2 K for an
excitation density of 2.8$\times$1019 cm-3 (see Supporting Information).
Thermal heating leads to an intensity decrease of all Bragg peaks as per the
Debye-Waller effect, see for instance Ref. 48. Such a response is clearly not
observed here for peaks 1, 2 and 4. Furthermore, peaks 2 and 5 shift to a
lower scattering vector after photo-excitation, while other peaks do not
follow this behaviour. Hence the data is also inconsistent with simple thermal
expansion, where all peaks would go to lower $Q$ vectors. This simple overview
of the data therefore suggests that the photo-induced lattice dynamics reflect
some more complex light-induced structural distortion arising from electron-
phonon interactions.

Figure 3: (a) Time-resolved relative diffraction intensities of the CsPbBr3
NCs for various excitation densities, obtained by averaging the raw
diffraction signals over some regions of interest (see Supplementary Figure 6
for more details on how they were obtained). The color code is matched to that
of panels (b) and (c), which display the values of the corresponding
excitation densities on their x-axis. Inset: example of a time-resolved trace
over the 100 ps time range. Following the drop in intensity, the subsequent
recovery indicates the onset of lattice cooling to the substrate. (b) Time
constant $\tau_{1}$ extracted from a bi-exponential fit to the data as a
function of excitation density. The errors correspond to 68% confidence
intervals of the fits. (c) Amplitude $A_{1}$ extracted from the same fit as a
function of excitation density.

Figure 3 shows the time-resolved relative diffraction intensities of the
CsPbBr3 NCs for various excitation densities, obtained by averaging the raw
diffraction signals over the regions of interest (ROIs) shown in Supplementary
Figure 6. All the ROIs exhibit the same dynamic response. An extended time
range is presented in the inset of panel (a). The transient diffraction
intensity can be fitted to a bi-exponential function convolved with a Gaussian
(FHWM of 300 fs) to account for the finite temporal resolution of the
experiment, see solid curves in Figure 3(a). The fit results reveal that the
lattice dynamics are well-captured by two time constants: a sub-picosecond
time constant $\tau_{1}$ associated with the initial decrease in peak
intensity, and a slow time constant $\tau_{2}$ of around 20 ps. We assign the
slow time constant to heat transfer from the NCs to the Quantifoil substrate,
and do not analyze it further. The fast time constant $\tau_{1}$ is intrinsic
to the CsPbBr3 NCs and reveals the reponse of the lattice to the excitation.
Figures 3(b) and (c) show the evolution of $\tau_{1}$ and the associated fit
amplitude $A_{1}$ as a function of excitation density. We observe that
$\tau_{1}$ rises with increasing excitation density, from 0.38 $\pm$ 0.13 ps
at 0.7$\times$1019 cm-3 to 1.17 $\pm$ 0.03 ps at 5.6$\times$1019 cm-3.
Meanwhile, the fit amplitude of the decay, $A_{1}$, increases from about
0.5$\%$ to around 4$\%$. This indicates, as expected, that the effect becomes
more pronounced at high excitation densities.

To complement the analysis shown in Figure 3, we determine the fluence
dependence of the peak position variation of peak 2 (Supplementary Figure 7).
Here also, we fit the peak position dynamics to a bi-exponential function
convolved with a Gaussian. We observe similar values and trend for the fast
time constant, $\tau_{1}^{\textrm{p2}}$, compared to those in Figure 3(b). In
contrast, no clear trend is seen for the amplitude of the peak shift,
$A_{1}^{\textrm{p2}}$, as a function of fluence. Together, the results of
Figure 3 and Supplementary Figure 7 suggest all observables in the data
(relative intensities and peak positions) follow the same sub-picosecond
dynamics and do not reflect independent processes. In the Supplementary
Information, we further show that such fast peak position changes do not
violate speed of sound propagation in the specific case of NCs, owing to their
high surface to volume ratio.

###  1.2 Analysis of the structural distortion

We next evaluate possible real-space atomic motions at the origin of the
structural distortion. We investigate commonly observed distortions in
perovskites and whether they can give rise to the lattice dynamics in Figure 2
49. Specifically we consider tilting and distortions of the octahedra (e.g.
changes in the Pb-Br bonds). Octahedral tilting, in particular, was reported
to occur in response to ultrafast photo-excitation in other perovskites such
as SrTiO3 50, 51. For the analysis, we follow a similar approach as in Ref.
26. We use the fact that atomic motions perpendicular to a lattice plane
($hkl$) modify the corresponding scattering intensity Ihkl, but in-plane
motions do not. We start from peak 2, since it shows the clearest signature.
Peak 2 is only sensitive to the (040) and (202) Miller planes, shown in
Supplementary Figure 10. Having shown that the observed peak shift cannot be
reproduced by intensity distribution changes between the (040) and (202)
reflections, we list the possible atomic motions contributing to the signal.
For the (040) plane, for instance, either a modification of the Pb-Br2 bond or
a tilting of the octahedra along the $c$ or $a$ axes of the crystal would
change I${}_{\textrm{040}}$, see Figure 4(c). A similar reasoning can be
applied to the (202) plane. The octahedra tilting angle or bond length changes
can be estimated based on the shift of peak 2 at late delays (see
Supplementary Information). Each possible distortion is individually simulated
by modifying the unit cell according to these estimates, and diffraction
patterns are generated for the modified structures. This procedure enables us
to directly compare the simulated and experimental difference diffraction
patterns for the different cases. Examples of a few distortions and simulated
patterns are shown in Supplementary Figure 10. In Supplementary Figure 11, we
also simulate a phase transition from the orthorhombic to the cubic phase,
previously reported in a tr-XRD study on similar CsPbBr3 NCs by the authors of
Ref. 29.

The best agreement with the data is reached by a lengthening of the Pb-Br2
bond, see Figure 4(a-c). This distortion reproduces the peak shift of peak 2,
the intensity reduction in peak 3 and the intensity rise of peak 4. The
magnitude of the simulated relative difference is also in agreement with the
experimental relative difference. Overall, the agreement remains qualitative
due to heating effects being neglected (see Supporting information) and the
sheer complexity of the LHP lattice structure. However, our work strongly
suggests the involvement of Pb-Br cage motions in the build-up of the light-
induced distortion, and in particular changes in the Pb-Br2 bond.

Figure 4: (a) Experimental diffraction profile of the CsPbBr3 NCs (black), and
relative difference profile from the experiment (dashed red). (b) Simulated
diffraction profile of the CsPbBr3 NCs (black), and simulated relative
difference profile (dashed orange). More details about the distortion
simulations are found in the text and Supplementary Information. (c) The (040)
Miller plane is indicated (blue). The distortion simulated in panel (b)
consists of a lengthening of Pb-Br2 bond by 0.09 %, estimated from the
relative shift of peak 2. (d) Schematic illustration of the exciton-polaron
formation process. The collective lattice dynamics following photoexcitation
(red arrow) result in excited-state dynamics (orange wavepackets) on the
excited potential energy surface that evolve from an initial state toward
equilibrium.

The presence of polarons in LHPs has been claimed by multiple complementary
techniques, ranging from optical 7, 32, 52, 53, 54, 55, 8 and photoemission
spectroscopies 56 to structural probes 25, 30. Both the timescales and nature
of the lattice dynamics observed here are consistent with the polaron
formation picture 7, 31, 55. Furthermore, several studies have also suggested
the involvement of [PbBr3]- cage motions in polaron formation 7, 32, 57, 58,
59, and atomic motion along the Pb-Br2 direction 30. Figure 4(d) summarizes
our interpretation of the data, in which lattice reorganization follows
photoexcitation (red arrow), i.e. the lattice evolves from an initial state
toward a new equilibrium. We note that for the investigated excitation
densities here, each NC hosts several exciton-polarons whose radii may overlap
60.

###  1.3 Interplay between hot carrier cooling and the creation of an exciton-
polaron population

In addition to the light-induced structural distortion, there are lattice
heating contributions to the data arising from carrier cooling. We estimate
that the Debye-Waller effect generates between 0.2 and 1$\%$ peak intensity
losses depending on the scattering vector and excitation density (see
Supporting Information). Thus, while heating may not dominate the lattice
dynamics, it also cannot be neglected. Within our instrument response function
of 300 fs, we do not observe hot carrier cooling and the emergence of an
exciton-polaron population to occur in a two-step fashion. The peak shift
dynamics - which can be assumed to reflect primarily the polaronic signatures
- exhibit very similar time constants compared to the integrated ROIs, where
lattice heating as a result of carrier cooling should clearly play a role.
Therefore, our data suggest that hot carrier thermalization and exciton-
polaron population build-up occur in a coupled fashion.

The increase of $\tau_{1}$ with increasing excitation density seen in Figure
3(b) and in Supplementary Figure 7(b) shows that the structural distortion
exhibits longer time constants at higher fluences. We note that the intensity
variations in our experiments reflect the population dynamics of exciton-
polarons, which depend on both exciton-polaron formation and hot carrier
cooling times 32. Multiple studies have reported a slowing down of carrier
cooling on the sub-picosecond timescale at high fluences 33, 10, 19, 34. Such
trend could arise from carrier screening effects at high excitation densities,
which are known to occur in polar semiconductors and would reduce the rate of
phonon emission 61. Alternatively, from a simple two-temperature model, one
would also expect an increase of the lattice heating time with increasing
initial change in electronic temperature 54, 62. Finally, the same trend would
also be observed in the case of nonthermal phonon populations, which are
likely present in our sample on the sub-picosecond timescale given the strong
dependence of the time constant with fluence. Regardless of the origin of this
dependence, the slower creation of the distortion at high fluences in our data
is fully consistent with the slower hot carrier cooling rates observed by
others. Further measurements pumping the NCs at the bandedge, where cooling
effects are minimized, may isolate the exciton-polaron formation time in the
future.

Even at the highest fluences, our measurements do not display signatures of
lattice heating over a timescale of several picoseconds. At high excitation
densities (> 1018 cm-3), several transient-absorption (TA) studies have
reported slow components in the spectral dynamics, with time constants ranging
from a few picoseconds 15, 13, 12, 10, 17 to tens or even hundreds of
picoseconds 9, 14, 11. The interpretation of these slow components is
controversial and lacks a commonly accepted picture 14, 63, 15, 16, with some
studies assigning the slow dynamics to the hot-phonon bottleneck effect 9,
other studies assigning them to Auger relaxation processes 63. For our
inorganic NCs, the time-resolved diffraction data do not exhibit signatures of
a long-lived hot-phonon bottleneck.

Our study has revealed the sub-picosecond lattice dynamics of photo-excited
CsPbBr3 NCs. The main observation was the direct observation of a structural
distortion within hundreds femtoseconds, which we assigned to the lattice
signature of an emerging population of exciton-polarons. The distortion is
consistent with atomic motions of the [PbBr3]- cage. We further observed that
the exciton-polaron population takes more time to build-up at high fluences,
which we attributed to slower hot carrier cooling. No hot-phonon bottleneck
effect lasting several picoseconds was observed for the investigated
excitation densities, which nearly reached the damage threshold of the NCs.
Our study sheds light on the microscopic origins of slow carrier cooling in
LHPs by helping discriminate between the competing scenarios proposed in the
literature.

This work was funded by the Max Planck-EPFL-Center for Molecular Nanoscience
and Technology, the Max Planck Society, the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation program (Grant
Agreement Number ERC-2015-CoG-682843), and partially by the Deutsche
Forschungsgemeinschaft (DFG) - Projektnummer 182087777 - SFB 951. H.S.
acknowledges support by the Swiss National Science Foundation under Grant No.
P2SKP2_184100. V.T. acknowledges financial support from the Alexander von
Humboldt Foundation. M.I.B. thanks for the support from the Swiss National
Science Foundation (grant number 200021_192308, project Q-Light). M. I. B. and
M.V.K acknowledge support by the Research and Innovation Foundation of Cyprus,
under the "New Strategic Infrastructure Units-Young Scientists" Program, grant
agreement number "INFRASTRUCTURES/1216/0004", Acronym "NANOSONICS". M. I. B.
and M.V.K are grateful for the use of the Empa Electron Microscopy Center
facilities. Y.W.W acknowledges funding from the DFG within the Emmy Noether
program under Grant No. RE 3977/1, as well as the TRR 227 (Project A10).

Simulated diffraction profiles and Bragg peaks assignment; Calculation of
excitation density; Estimate of temperature rise; Estimate of the Debye-Waller
effect; Estimate of average distance of atom to nanocrystal surface;
Determination of octahedral tilt angle.

## References

  * Zhang et al. 2016 Zhang, W.; Eperon, G. E.; Snaith, H. J. Metal halide perovskites for energy applications. _Nature Energy_ 2016, _1_ , 16048
  * Protesescu et al. 2015 Protesescu, L.; Yakunin, S.; Bodnarchuk, M. I.; Krieg, F.; Caputo, R.; Hendon, C. H.; Yang, R. X.; Walsh, A.; Kovalenko, M. V. Nanocrystals of Cesium Lead Halide Perovskites (CsPbX3, X = Cl, Br, and I): Novel Optoelectronic Materials Showing Bright Emission with Wide Color Gamut. _Nano Letters_ 2015, _15_ , 3692–3696
  * Sutherland and Sargent 2016 Sutherland, B. R.; Sargent, E. H. Perovskite photonic sources. _Nature Photonics_ 2016, _10_ , 295–302
  * Correa-Baena et al. 2017 Correa-Baena, J.-P.; Saliba, M.; Buonassisi, T.; Grätzel, M.; Abate, A.; Tress, W.; Hagfeldt, A. Promises and challenges of perovskite solar cells. _Science_ 2017, _358_ , 739–744
  * Zhu et al. 2016 Zhu, H.; Miyata, K.; Fu, Y.; Wang, J.; Joshi, P. P.; Niesner, D.; Williams, K. W.; Jin, S.; Zhu, X.-Y. Screening in crystalline liquids protects energetic carriers in hybrid perovskites. _Science_ 2016, _353_ , 1409–1413
  * Niesner et al. 2016 Niesner, D.; Zhu, H.; Miyata, K.; Joshi, P. P.; Evans, T. J. S.; Kudisch, B. J.; Trinh, M. T.; Marks, M.; Zhu, X.-Y. Persistent Energetic Electrons in Methylammonium Lead Iodide Perovskite Thin Films. _Journal of the American Chemical Society_ 2016, _138_ , 15717–15726
  * Miyata et al. 2017 Miyata, K.; Meggiolaro, D.; Trinh, M. T.; Joshi, P. P.; Mosconi, E.; Jones, S. C.; Angelis, F. D.; Zhu, X.-Y. Large polarons in lead halide perovskites. _Science Advances_ 2017, _3_ , e1701217
  * Buizza and Herz 2021 Buizza, L. R. V.; Herz, L. M. Polarons and Charge Localization in Metal-Halide Semiconductors for Photovoltaic and Light-Emitting Devices. _Advanced Materials_ 2021, _33_ , 2007057
  * Yang et al. 2015 Yang, Y.; Ostrowski, D. P.; France, R. M.; Zhu, K.; van de Lagemaat, J.; Luther, J. M.; Beard, M. C. Observation of a hot-phonon bottleneck in lead-iodide perovskites. _Nature Photonics_ 2015, _10_ , 53–59
  * Price et al. 2015 Price, M. B.; Butkus, J.; Jellicoe, T. C.; Sadhanala, A.; Briane, A.; Halpert, J. E.; Broch, K.; Hodgkiss, J. M.; Friend, R. H.; Deschler, F. Hot-carrier cooling and photoinduced refractive index changes in organic–inorganic lead halide perovskites. _Nature Communications_ 2015, _6_ , 8420
  * Yang et al. 2017 Yang, J. et al. Acoustic-optical phonon up-conversion and hot-phonon bottleneck in lead-halide perovskites. _Nature Communications_ 2017, _8_ , 14120
  * Papagiorgis et al. 2017 Papagiorgis, P.; Protesescu, L.; Kovalenko, M. V.; Othonos, A.; Itskos, G. Long-Lived Hot Carriers in Formamidinium Lead Iodide Nanocrystals. _The Journal of Physical Chemistry C_ 2017, _121_ , 12434–12440
  * Mondal et al. 2018 Mondal, A.; Aneesh, J.; Ravi, V. K.; Sharma, R.; Mir, W. J.; Beard, M. C.; Nag, A.; Adarsh, K. V. Ultrafast exciton many-body interactions and hot-phonon bottleneck in colloidal cesium lead halide perovskite nanocrystals. _Physical Review B_ 2018, _98_ , 115418
  * Fu et al. 2017 Fu, J.; Xu, Q.; Han, G.; Wu, B.; Huan, C. H. A.; Leek, M. L.; Sum, T. C. Hot carrier cooling mechanisms in halide perovskites. _Nature Communications_ 2017, _8_ , 1300
  * Butkus et al. 2017 Butkus, J.; Vashishtha, P.; Chen, K.; Gallaher, J. K.; Prasad, S. K. K.; Metin, D. Z.; Laufersky, G.; Gaston, N.; Halpert, J. E.; Hodgkiss, J. M. The Evolution of Quantum Confinement in CsPbBr3 Perovskite Nanocrystals. _Chemistry of Materials_ 2017, _29_ , 3644–3652
  * Chan et al. 2021 Chan, C. C. S.; Fan, K.; Wang, H.; Huang, Z.; Novko, D.; Yan, K.; Xu, J.; Choy, W. C. H.; Lončarić, I.; Wong, K. S. Uncovering the Electron-Phonon Interplay and Dynamical Energy-Dissipation Mechanisms of Hot Carriers in Hybrid Lead Halide Perovskites. _Advanced Energy Materials_ 2021, _11_ , 2003071
  * Nie et al. 2020 Nie, Z.; Gao, X.; Ren, Y.; Xia, S.; Wang, Y.; Shi, Y.; Zhao, J.; Wang, Y. Harnessing Hot Phonon Bottleneck in Metal Halide Perovskite Nanocrystals via Interfacial Electron–Phonon Coupling. _Nano Letters_ 2020, _20_ , 4610–4617
  * Shi et al. 2020 Shi, H.; Zhang, X.; Sun, X.; Zhang, X. Strong hot-phonon bottleneck effect in all-inorganic perovskite nanocrystals. _Applied Physics Letters_ 2020, _116_ , 151902
  * Verkamp et al. 2021 Verkamp, M.; Leveillee, J.; Sharma, A.; Lin, M.-F.; Schleife, A.; Vura-Weis, J. Carrier-Specific Hot Phonon Bottleneck in CH3NH3PbI3 Revealed by Femtosecond XUV Absorption. _Journal of the American Chemical Society_ 2021, 20176–20182
  * Chen et al. 2019 Chen, J.; Messing, M. E.; Zheng, K.; Pullerits, T. Cation-Dependent Hot Carrier Cooling in Halide Perovskite Nanocrystals. _Journal of the American Chemical Society_ 2019, _141_ , 3532–3540
  * Monahan et al. 2017 Monahan, D. M.; Guo, L.; Lin, J.; Dou, L.; Yang, P.; Fleming, G. R. Room-Temperature Coherent Optical Phonon in 2D Electronic Spectra of CH3NH3PbI3 Perovskite as a Possible Cooling Bottleneck. _The Journal of Physical Chemistry Letters_ 2017, _8_ , 3211–3215
  * Stern et al. 2018 Stern, M. J.; de Cotret, L. P. R.; Otto, M. R.; Chatelain, R. P.; Boisvert, J.-P.; Sutton, M.; Siwick, B. J. Mapping momentum-dependent electron-phonon coupling and nonequilibrium phonon dynamics with ultrafast electron diffuse scattering. _Physical Review B_ 2018, _97_ , 165416
  * Zahn et al. 2020 Zahn, D.; Hildebrandt, P.-N.; Vasileiadis, T.; Windsor, Y. W.; Qi, Y.; Seiler, H.; Ernstorfer, R. Anisotropic Nonequilibrium Lattice Dynamics of Black Phosphorus. _Nano Letters_ 2020, _20_ , 3728–3733
  * Seiler et al. 2021 Seiler, H.; Zahn, D.; Zacharias, M.; Hildebrandt, P.-N.; Vasileiadis, T.; Windsor, Y. W.; Qi, Y.; Carbogno, C.; Draxl, C.; Ernstorfer, R.; Caruso, F. Accessing the Anisotropic Nonthermal Phonon Populations in Black Phosphorus. _Nano Letters_ 2021, _21_ , 6171–6178
  * Guzelturk et al. 2021 Guzelturk, B. et al. Visualization of dynamic polaronic strain fields in hybrid lead halide perovskites. _Nature Materials_ 2021, _20_ , 618–623
  * Seiler et al. 2021 Seiler, H.; Krynski, M.; Zahn, D.; Hammer, S.; Windsor, Y. W.; Vasileiadis, T.; Pflaum, J.; Ernstorfer, R.; Rossi, M.; Schwoerer, H. Nuclear dynamics of singlet exciton fission in pentacene single crystals. _Science Advances_ 2021, _7_ , eabg0869
  * de Cotret et al. 2022 de Cotret, L. P. R.; Otto, M. R.; Pöhls, J.-H.; Luo, Z.; Kanatzidis, M. G.; Siwick, B. J. Direct visualization of polaron formation in the thermoelectric SnSe. _Proceedings of the National Academy of Sciences_ 2022, _119_ , e2113967119
  * Wu et al. 2017 Wu, X. et al. Light-induced picosecond rotational disordering of the inorganic sublattice in hybrid perovskites. _Science Advances_ 2017, _3_ , e160238
  * Kirschner et al. 2019 Kirschner, M. S.; Diroll, B. T.; Guo, P.; Harvey, S. M.; Helweh, W.; Flanders, N. C.; Brumberg, A.; Watkins, N. E.; Leonard, A. A.; Evans, A. M.; Wasielewski, M. R.; Dichtel, W. R.; Zhang, X.; Chen, L. X.; Schaller, R. D. Photoinduced, reversible phase transitions in all-inorganic perovskite nanocrystals. _Nature Communications_ 2019, _10_ , 504
  * Cannelli et al. 2021 Cannelli, O. et al. Quantifying Photoinduced Polaronic Distortions in Inorganic Lead Halide Perovskite Nanocrystals. _Journal of the American Chemical Society_ 2021, _143_ , 9048–9059
  * Evans et al. 2018 Evans, T. J. S.; Miyata, K.; Joshi, P. P.; Maehrlein, S.; Liu, F.; Zhu, X.-Y. Competition Between Hot-Electron Cooling and Large Polaron Screening in CsPbBr3 Perovskite Single Crystals. _The Journal of Physical Chemistry C_ 2018, _122_ , 13724–13730
  * Bretschneider et al. 2018 Bretschneider, S. A.; Ivanov, I.; Wang, H. I.; Miyata, K.; Zhu, X.; Bonn, M. Quantifying Polaron Formation and Charge Carrier Cooling in Lead-Iodide Perovskites. _Advanced Materials_ 2018, _30_ , 1707312
  * Hopper et al. 2018 Hopper, T. R.; Gorodetsky, A.; Frost, J. M.; Müller, C.; Lovrincic, R.; Bakulin, A. A. Ultrafast Intraband Spectroscopy of Hot-Carrier Cooling in Lead-Halide Perovskites. _ACS Energy Letters_ 2018, _3_ , 2199–2205
  * Diroll et al. 2019 Diroll, B. T.; Mannodi-Kanakkithodi, A.; Chan, M. K. Y.; Schaller, R. D. Spectroscopic Comparison of Thermal Transport at Organic–Inorganic and Organic-Hybrid Interfaces Using CsPbBr3 and FAPbBr3 (FA = Formamidinium) Perovskite Nanocrystals. _Nano Letters_ 2019, 8155−8160
  * Bodnarchuk et al. 2018 Bodnarchuk, M. I.; Boehme, S. C.; ten Brinck, S.; Bernasconi, C.; Shynkarenko, Y.; Krieg, F.; Widmer, R.; Aeschlimann, B.; Günther, D.; Kovalenko, M. V.; Infante, I. Rationalizing and Controlling the Surface Structure and Electronic Passivation of Cesium Lead Halide Nanocrystals. _ACS Energy Letters_ 2018, _4_ , 63–74
  * Stoumpos et al. 2013 Stoumpos, C. C.; Malliakas, C. D.; Peters, J. A.; Liu, Z.; Sebastian, M.; Im, J.; Chasapis, T. C.; Wibowo, A. C.; Chung, D. Y.; Freeman, A. J.; Wessels, B. W.; Kanatzidis, M. G. Crystal Growth of the Perovskite Semiconductor CsPbBr3: A New Material for High-Energy Radiation Detection. _Crystal Growth & Design_ 2013, _13_ , 2722–2727
  * Maes et al. 2018 Maes, J.; Balcaen, L.; Drijvers, E.; Zhao, Q.; Roo, J. D.; Vantomme, A.; Vanhaecke, F.; Geiregat, P.; Hens, Z. Light Absorption Coefficient of CsPbBr3 Perovskite Nanocrystals. 2018, _9_ , 3093–3097
  * Yaffe et al. 2017 Yaffe, O.; Guo, Y.; Tan, L. Z.; Egger, D. A.; Hull, T.; Stoumpos, C. C.; Zheng, F.; Heinz, T. F.; Kronik, L.; Kanatzidis, M. G.; Owen, J. S.; Rappe, A. M.; Pimenta, M. A.; Brus, L. E. Local Polar Fluctuations in Lead Halide Perovskite Crystals. 2017, _118_ , 136001
  * Worhatch et al. 2008 Worhatch, R. J.; Kim, H.; Swainson, I. P.; Yonkeu, A. L.; Billinge, S. J. L. Study of Local Structure in Selected Organic–Inorganic Perovskites in the Pm3̅m Phase. 2008, _20_ , 1272–1277
  * Bertolotti et al. 2017 Bertolotti, F.; Protesescu, L.; Kovalenko, M. V.; Yakunin, S.; Cervellino, A.; Billinge, S. J. L.; Terban, M. W.; Pedersen, J. S.; Masciocchi, N.; Guagliardi, A. Coherent Nanotwins and Dynamic Disorder in Cesium Lead Halide Perovskite Nanocrystals. 2017, _11_ , 3819–3831
  * Cannelli et al. 2022 Cannelli, O.; Wiktor, J.; Colonna, N.; Leroy, L.; Puppin, M.; Bacellar, C.; Sadykov, I.; Krieg, F.; Smolentsev, G.; Kovalenko, M. V.; Pasquarello, A.; Chergui, M.; Mancini, G. F. Atomic-Level Description of Thermal Fluctuations in Inorganic Lead Halide Perovskites. _The Journal of Physical Chemistry Letters_ 2022, _13_ , 3382–3391
  * Wang et al. 2016 Wang, X.; Rahmani, H.; Zhou, J.; Gorfien, M.; Plaskus, J. M.; Li, D.; Voss, R.; Nelson, C. A.; Lei, K. W.; Wolcott, A.; Zhu, X.; Li, J.; Cao, J. Ultrafast lattice dynamics in lead selenide quantum dot induced by laser excitation. _Applied Physics Letters_ 2016, _109_ , 153105
  * Mancini et al. 2016 Mancini, G. F.; Latychevskaia, T.; Pennacchio, F.; Reguera, J.; Stellacci, F.; Carbone, F. Order/Disorder Dynamics in a Dodecanethiol-Capped Gold Nanoparticles Supracrystal by Small-Angle Ultrafast Electron Diffraction. _Nano Letters_ 2016, _16_ , 2705–2713
  * Vasileiadis et al. 2018 Vasileiadis, T.; Waldecker, L.; Foster, D.; Silva, A. D.; Zahn, D.; Bertoni, R.; Palmer, R. E.; Ernstorfer, R. Ultrafast Heat Flow in Heterostructures of Au Nanoclusters on Thin Films: Atomic Disorder Induced by Hot Electrons. _ACS Nano_ 2018, _12_ , 7710–7720
  * Vasileiadis et al. 2019 Vasileiadis, T.; Skountzos, E. N.; Foster, D.; Coleman, S. P.; Zahn, D.; Krečinić, F.; Mavrantzas, V. G.; Palmer, R. E.; Ernstorfer, R. Ultrafast rotational motions of supported nanoclusters probed by electron diffraction. _Nanoscale Horizons_ 2019, _4_ , 1164–1173
  * Krawczyk et al. 2021 Krawczyk, K. M.; Sarracini, A.; Green, P. B.; Hasham, M.; Tang, K.; Paré-Labrosse, O.; Voznyy, O.; Wilson, M. W. B.; Miller, R. J. D. Anisotropic, Nonthermal Lattice Disordering Observed in Photoexcited PbS Quantum Dots. _The Journal of Physical Chemistry C_ 2021, _125_ , 22120–22132
  * Waldecker et al. 2015 Waldecker, L.; Bertoni, R.; Ernstorfer, R. Compact femtosecond electron diffractometer with 100 keV electron bunches approaching the single-electron pulse duration limit. _Journal of Applied Physics_ 2015, _117_ , 044903
  * Zahn et al. 2021 Zahn, D.; Seiler, H.; Windsor, Y. W.; Ernstorfer, R. Ultrafast lattice dynamics and electron–phonon coupling in platinum extracted with a global fitting approach for time-resolved polycrystalline diffraction data. _Structural Dynamics_ 2021, _8_ , 064301
  * Woodward 1997 Woodward, P. M. Octahedral Tilting in Perovskites. I. Geometrical Considerations. _Acta Crystallographica Section B Structural Science_ 1997, _53_ , 32–43
  * Porer et al. 2018 Porer, M. et al. Ultrafast Relaxation Dynamics of the Antiferrodistortive Phase in Ca Doped SrTiO3. _Physical Review Letters_ 2018, _121_ , 055701
  * Porer et al. 2019 Porer, M. et al. Ultrafast transient increase of oxygen octahedral rotations in a perovskite. _Physical Review Research_ 2019, _1_ , 012005(R)
  * Thouin et al. 2019 Thouin, F.; Valverde-Chávez, D. A.; Quarti, C.; Cortecchia, D.; Bargigia, I.; Beljonne, D.; Petrozza, A.; Silva, C.; Kandada, A. R. S. Phonon coherences reveal the polaronic character of excitons in two-dimensional lead halide perovskites. _Nature Materials_ 2019, _18_ , 349–356
  * Lan et al. 2019 Lan, Y.; Dringoli, B. J.; Valverde-Chávez, D. A.; Ponseca, C. S., Jr; Sutton, M.; He, Y.; Kanatzidis, M. G.; Cooke, D. G. Ultrafast correlated charge and lattice motion in a hybrid metal halide perovskite. _Science Advances_ 2019, _5_ , eaaw5558
  * Allen 1987 Allen, P. B. Theory of thermal relaxation of electrons in metals. _Phys. Rev. Lett._ 1987, _59_ , 1460–1463
  * Seiler et al. 2019 Seiler, H.; Palato, S.; Sonnichsen, C.; Baker, H.; Socie, E.; Strandell, D. P.; Kambhampati, P. Two-dimensional electronic spectroscopy reveals liquid-like lineshape dynamics in CsPbI3 perovskite nanocrystals. 2019, _10_ , 4962
  * Puppin et al. 2020 Puppin, M. et al. Evidence of Large Polarons in Photoemission Band Mapping of the Perovskite Semiconductor CsPbBr3. 2020, _124_ , 206402
  * Park et al. 2018 Park, M.; Neukirch, A. J.; Reyes-Lillo, S. E.; Lai, M.; Ellis, S. R.; Dietze, D.; Neaton, J. B.; Yang, P.; Tretiak, S.; Mathies, R. A. Excited-state vibrational dynamics toward the polaron in methylammonium lead iodide perovskite. _Nature Communications_ 2018, _9_ , 2525
  * Ambrosio et al. 2018 Ambrosio, F.; Wiktor, J.; Angelis, F. D.; Pasquarello, A. Origin of low electron–hole recombination rate in metal halide perovskites. _Energy & Environmental Science_ 2018, _11_ , 101–105
  * Schlipf et al. 2018 Schlipf, M.; Poncé, S.; Giustino, F. Carrier lifetimes and polaronic mass enhancement in the hybrid Halide perovskite CH_{3}NH_{3}PbI_{3} from multiphonon Fröhlich coupling. _Phys. Rev. Lett._ 2018, _121_ , 086402
  * Frost et al. 2017 Frost, J. M.; Whalley, L. D.; Walsh, A. Slow Cooling of Hot Polarons in Halide Perovskite Solar Cells. _ACS Energy Letters_ 2017, _2_ , 2647–2652
  * Yoffa 1981 Yoffa, E. J. Screening of hot-carrier relaxation in highly photoexcited semiconductors. _Physical Review B_ 1981, _23_ , 1909–1919
  * S.I. et al. 1967 S.I., A.; Bonch-Bruevich,; El’yashevich,; Imas,; Pavlenko,; Romanov, Effect of Powerful Light Fluxes on Metals. _Sov. Phys.-Tech. Phys._ 1967, _11_ , 945
  * Li et al. 2017 Li, M.; Bhaumik, S.; Goh, T. W.; Kumar, M. S.; Yantara, N.; Grätzel, M.; Mhaisalkar, S.; Mathews, N.; Sum, T. C. Slow cooling and highly efficient extraction of hot carriers in colloidal perovskite nanocrystals. _Nature Communications_ 2017, _8_ , 14350


# MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large
Language Models

Thao Minh Nguyen Phan  ,  Cong-Tinh Dao  National Yang Ming Chiao Tung
UniversityHsinchuTaiwan pnmthaoct<EMAIL_ADDRESS>,  Chenwei Wu
University of MichiganMichiganUSA<EMAIL_ADDRESS>,  Jian-Zhe Wang
National Yang Ming Chiao Tung UniversityHsinchuTaiwan<EMAIL_ADDRESS>,  Shun Liu  Shanghai University of Finance and EcomonicsShanghaiChina
<EMAIL_ADDRESS>,  Jun-En Ding  Stevens Institute of Technology
HobokenNew JerseyUSA<EMAIL_ADDRESS>,  David Restrepo  Massachusetts
Institute of TechnologyMassachusettsUSA<EMAIL_ADDRESS>,  Feng Liu  Stevens
Institute of Technology HobokenNew JerseyUSA<EMAIL_ADDRESS>,  Fang-Ming
Hung  Far Eastern Memorial HospitalNew TaipeiTaiwan<EMAIL_ADDRESS>and  Wen-Chih Peng  National Yang Ming Chiao Tung UniversityHsinchuTaiwan
<EMAIL_ADDRESS>
(2024; 20 February 2024; 12 March 2024; 5 June 2024)

###### Abstract.

Electronic health records (EHRs) are multimodal by nature, consisting of
structured tabular features like lab tests and unstructured clinical notes. In
real-life clinical practice, doctors use complementary multimodal EHR data
sources to get a clearer picture of patients’ health and support clinical
decision-making. However, most EHR predictive models do not reflect these
procedures, as they either focus on a single modality or overlook the inter-
modality interactions/redundancy. In this work, we propose MEDFuse, a
Multimodal EHR Data Fusion framework that incorporates masked lab-test
modeling and large language models (LLMs) to effectively integrate structured
and unstructured medical data. MEDFuse leverages multimodal embeddings
extracted from two sources: LLMs fine-tuned on free clinical text and masked
tabular transformers trained on structured lab test results. We design a
disentangled transformer module, optimized by a mutual information loss to 1)
decouple modality-specific and modality-shared information and 2) extract
useful joint representation from the noise and redundancy present in clinical
notes. Through comprehensive validation on the public MIMIC-III dataset and
the in-house FEMH dataset, MEDFuse demonstrates great potential in advancing
clinical predictions, achieving over 90% F1 score in the 10-disease multi-
label classification task.

Computer-aided Diagnosis; Large Language Model Fine-tuning; Electronic Health
Records

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
email; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Applied computing Health care information systems††ccs: Computing
methodologies Artificial intelligence††ccs: Information systems Data mining

##  1\. Introduction

Electronic Health Records (EHRs) are widely adopted in healthcare, documenting
a wealth of heterogeneous patient data comprised of tabular records and
unstructured clinical notes. Tabular records encompass essential medical
concepts such as diagnoses, medications, and laboratory test results,
providing a structured overview of a patient’s health. In contrast, clinical
notes are extensive, free-text documents written by healthcare providers,
offering a more detailed and nuanced account of the patient’s history,
clinical findings, and progress. The vast volume and diversity of multimodal
data within EHRs present a unique opportunity for deep learning technologies
to improve the prediction and management of diseases (Ding et al., 2024;
Restrepo et al., 2024). Nevertheless, the heterogeneous nature and large
quantity of redundancy in multimodal EHR inputs pose significant challenges
for medical AI practitioners to effectively distill and fuse clinically
meaningful information for disease prediction.

The primary question at hand is: can we effectively obtain and integrate
useful representations for different EHR modalities to improve clinical
predictions? Current research in deep EHR modeling (Luo et al., 2020; Ma et
al., 2017) often focuses on single data modalities, often neglecting the
integration of significant insights from unstructured medical notes and lab
tests. This oversight can limit the model from learning a more comprehensive
view of patient health conditions. Lab tests consist of high-dimensional,
usually discrete tabular data; however, the conventional approach models
structured EHR data as numerical vectors, overlooking complex interactions
between individual variables and does not consider their interactions (Choi et
al., 2016; Li et al., 2020; Zhang et al., 2020; Luo et al., 2020; Ma et al.,
2017). More recent work has moved towards deep learning architectures like
Bert and LLMs. LLMs fine-tuned on clinical data have shown promise in
unstructured clinical notes in understanding tasks like answering medical
questions and making few-shot predictions (Thirunavukarasu et al., 2023).
However, there is a large body of evidence showing that LLMs are still having
a hard time capturing the nuances of numerical lab test data and are
underperforming on tabular prediction tasks (Grinsztajn et al., 2022; Bellamy
et al., 2023; Hegselmann et al., 2023).

Another significant challenge in fusing information from different types of
EHR data is How do we distill the overlapping clinically important features
from both modalities? The information contained within different modalities
can be categorized as either modality-specific or modality-shared (Liang et
al., 2023). For example, a patient’s dietary habits would be considered
information specific to the clinical notes modality; hypertension record and
lab test value would be regarded as modality-shared information. Existing
efforts like multimodal EHR contrastive learning (Cai et al., 2024) have
primarily focused on integrating the modality-shared information by
emphasizing the inherent consistency through alignment techniques. However,
this approach often leads to the common information dominating the alignment
and integration process, resulting in the distinctive perspectives offered by
each modality being disregarded. Lab tests and clinical notes also possess
highly different noise-to-information ratios, making it hard to distill useful
joint representation from the noises and redundancy present in EHR. Therefore,
there is an urgent need for methods to extract the diverse yet collaborative
perspectives both modalities offer for informing therapeutic decision-making.

In this work, we propose MEDFuse, a novel Multimodal EHR Data Fusion
diagnostic model consisting of modality-specific embedding extractors followed
by a disentangled transformer for multimodal fusion. Our model integrates
embeddings between fine-tuned LLMs on unstructured clinical text and masked
lab-test modeling models pre-trained on structured laboratory results. We
further utilize a disentangled transformer optimized by mutual information
loss to decouple modality-specific and modality-common information and learn
meaningful joint representations for downstream prediction tasks. The key
contributions of our work are as follows:

  * •

We propose a novel diagnostic model integrating structured lab test data and
unstructured clinical notes, utilizing embeddings from fine-tuned LLMs and
Masked Lab-Test Modeling, enhancing understanding of diverse clinical
information.

  * •

We improved joint patient representation by incorporating a disentangled
transformer module to effectively separate and integrate modality-specific and
shared information, leading to better prediction outcomes across multiple
diseases.

  * •

We conducted empirical evaluations to illustrate our model’s effectiveness
through EHR datasets on various metrics.

##  2\. Related work

### 2.1 EHR For Multi-Label Disease Prediction

Most recent works in medical Multi-Label Text Classification (MLTC) entirely
rely on medical texts. For instance, Kim et al. (Chen, 2015) introduced a
convolutional attention network designed to extract meaningful document
representations across varying text lengths. Recent developments in LLMs, such
as those discussed by Luo et al. (2022) and Elliot et al. (Bolton et al.,
2024), utilize extensive data from medical literature for domain-specific
tasks such as natural language inferencing. Additionally, some studies have
employed graph neural networks (GNNs) to organize sequences from Electronic
Medical Records (EMR) into hierarchical graphs (Wu et al., 2021), or to
integrate entity relationships from text using attention mechanisms in neural
networks (Chen et al., 2019; Dun et al., 2021). Nevertheless, many of these
studies overlook the potential advantages of integrating medical expert
knowledge from official guidelines and critical blood tests. A combined
approach that harnesses both unstructured and structured data could offer
extra help to offset issues like label and data scarcity in the medical
domain.

### 2.2 Extraction of Clinical Relevant Information from Multimodal EHR

Recent work has leveraged self-supervised learning methods, like contrastive
pretraining of clinical notes (Cai et al., 2024) and prompt-based large
language modeling (Ding et al., 2024; Hegselmann et al., 2023), to facilitate
multimodal learning of EHR data. The former encourages the alignment between
paired patient data via contrastive loss, and the latter usually directly
converts the structured data into text by prompt templates and feeds it into
LLMs. However, if the data fusion process focuses solely on aligning the
common information, such as diabetes history (text) and blood glucose levels
(lab), the rich, modality-specific insights like exercise habits may be
overlooked. This can limit understanding of the patient’s health and impact
predictive models and clinical decisions. Therefore, it is essential to
develop multimodal EHR data fusion techniques that can effectively capture and
integrate both modality-specific and modality-shared information.

Figure 1. The proposed model architecture.

##  3\. Method

### 3.1 Overview

Given the clinical notes and lab tests of the patient’s current and historical
visits, MEDFuse integrates clinical notes and lab test data to create a
comprehensive patient representation for accurate multi-disease prediction.
Firstly, as illustrated in Figure 1, in the Multimodal Embedding Extraction
stage, textual data from clinical notes, including detailed patient
information and medical history, are filtered and structured. Simultaneously,
abnormal numerical data from various lab tests, such as triglyceride levels,
HDL cholesterol, ALT (SGPT), and glucose levels, are extracted and formatted
into textual data by prompt templates. The filtered clinical text is then
processed by fine-tuned LLMs to generate embeddings that capture its semantic
meaning. In parallel, the raw structured tabular data are processed using a
domain-specific masked lab-test model to create embeddings representing the
quantitative lab data. The two embeddings are then passed through the
disentangled transformer module for multimodal fusion and final disease
prediction.

### 3.2 Multimodal Embedding Extraction

####  3.2.1. Fine-tuning LLMs on Unstructured Text

Clinical notes, comprising diverse fields derived from physicians’ diagnoses,
form the textual component of the dataset. We filtered the text by Chief
Complaint, Present Illness, Medical History, and Medication on Admission.
These specific fields are crucial for accurately predicting a patient’s
disease. To integrate this tabular data with the textual clinical notes, we
converted the tabular data into a textual format, a process referred to as
tabular feature extraction. This method involves extracting abnormal lab test
results and formatting them into a text template — “These are abnormal results
recorded: ITEMID ¡ITEMID¿: ¡VALUE¿ ¡VALUEUOM¿; ITEMID ¡ITEMID¿: ¡VALUE¿
¡VALUEUOM¿; …;”. Here, ¡ITEMID¿ refers to the specific lab test names, ¡VALUE¿
indicates the test values, and ¡VALUEUOM¿ denotes the units of measure for the
test values.

Inspired by the recent success of fine-tuning the language models (Devlin et
al., 2018; Liu et al., 2019) for classification purposes, we fine-tuned
various LLMs for disease prediction. Our best-performing backbone is the
publicly accessible Medical-Llama3-8B model (ruslanmv, 2024), which is fine-
tuned from Meta-Llama-3-8B (AI@Meta, 2024b). It is trained on a comprehensive
medical chatbot dataset and optimized for addressing health-related inquiries.
We extracted latent vector representations from the final layer of the Llama
decoder, which was originally engineered for autoregressive prediction of
subsequent tokens. These extracted vectors were subsequently processed through
feed-forward neural layers, effectively transforming them into a label space.
The output from these transformations, in the form of logits, was then
utilized to perform discriminative classification based on labels. This method
aims to harness the latent embedding of LLMs to achieve targeted, efficient
task adaptation.

####  3.2.2. Masked Lab-Test Modeling

The Masked Lab-Test Modeling (MLTM) module extends the Masked Autoencoders
(MAE) (He et al., 2022; Song et al., 2023; Chen et al., 2024) framework to
reconstruct masked components based on observed components in EHR data. MLTM
consists of a encoder that maps observed values to their representations and a
decoder that reconstructs the masked values from the latent representations.
To account for the inherent incompleteness in the imputation task, MLTM
employs an additional masking approach in the training to make sure a uniform
75% value is masked out. The encoder applies a learnable linear encoding
function $wx+b$ to each unmasked $x$ and passes through a transformer
architecture, while the decoder operates on the embeddings of both observed
and masked values. Positional encoding is added to the embeddings to preserve
the lab test positions. The reconstruction loss is defined as the mean square
error between the reconstructed and original values on the re-masked and
unmasked sets. MLTM is designed with an asymmetric architecture, using a deep
encoder and a shallow decoder to extract useful lab-test representations.

### 3.3 Disentangled Transformer Module

Table 1. Training and Validation Performance Comparison of Various Models on the MIMIC-III Dataset. Model | Precision | Recall | F1 macro | F1 weighted | Accuracy
---|---|---|---|---|---
Bert | 0.8333 / 0.6790 | 0.2000 / 0.2000 | 0.1818 / 0.1618 | 0.3686 / 0.3162 | 0.6515 / 0.2692
LoRA Mistral-7B-v0.1 | 0.8759 / 0.8616 | 0.8459 / 0.8289 | 0.8449 / 0.8274 | 0.9007 / 0.8886 | 0.9089 / 0.8974
LoRA Llama-2-7B-hf | 0.8828 / 0.8585 | 0.8592 / 0.8364 | 0.8559 / 0.8301 | 0.9097 / 0.8924 | 0.9168 / 0.9004
LoRA Meta-Llama2-13B | 0.9153 / 0.8732 | 0.8852 / 0.8430 | 0.8874 / 0.8414 | 0.9297 / 0.8990 | 0.9363 / 0.9071
LoRA Meta-Llama3-8B | 0.8899 / 0.8667 | 0.8569 / 0.8306 | 0.8579 / 0.8305 | 0.9121 / 0.8935 | 0.9211 / 0.9040
LoRA Medical-Llama3-8B | 0.9283 / 0.8807 | 0.9008 / 0.8474 | 0.9026 / 0.8466 | 0.9367 / 0.9003 | 0.9417 / 0.9068
MEDFuse | 0.9375 / 0.9025 | 0.9217 / 0.8534 | 0.9216 / 0.8615 | 0.9462 / 0.9103 | 0.9535 / 0.9122
Table 2. Training and Validation Performance on the FEMH Dataset. Model | Precision | Recall | F1 macro | F1 weighted | Accuracy
---|---|---|---|---|---
LoRA Medical-Llama3-8B | 0.8702 / 0.8691 | 0.8496 / 0.8478 | 0.8453 / 0.8435 | 0.9182 / 0.9167 | 0.9267 / 0.9252
MEDFuse | 0.8839 / 0.8823 | 0.8707 / 0.8670 | 0.8637 / 0.8607 | 0.9260 / 0.9243 | 0.9311 / 0.9296
Table 3. Ablation Study on Training and Validation Performance on the MIMIC-III Dataset. Model | Precision | Recall | F1 macro | F1 weighted | Accuracy
---|---|---|---|---|---
MEDFuse w/o (MLTM & LABTEXT) | 0.8882 / 0.8580 | 0.8663 / 0.8406 | 0.8620 / 0.8321 | 0.9100 / 0.8901 | 0.9148 / 0.8955
MEDFuse w/o (MLTM & TEXT) | 0.6553 / 0.6224 | 0.6461 / 0.6203 | 0.6282 / 0.5980 | 0.7869 / 0.7627 | 0.8239 / 0.8008
MEDFuse w/o TEXT | 0.7730 / 0.7666 | 0.7912 / 0.7923 | 0.7600 / 0.7573 | 0.8331 / 0.8230 | 0.8331 / 0.8271
MEDFuse w/o Disentangled Transformer | 0.9330 / 0.8974 | 0.9164 / 0.8483 | 0.9162 / 0.8564 | 0.9417 / 0.9074 | 0.9489 / 0.9082
MEDFuse | 0.9375 / 0.9025 | 0.9217 / 0.8534 | 0.9216 / 0.8615 | 0.9462 / 0.9103 | 0.9535 / 0.9122

Initially, features from each modality are multiplied by the Kronecker product
to approximate a joint distribution, $C=A\bigotimes
B\in\mathbb{R}^{(m\times(a)\times(b))}$, effectively capturing the pairwise
interactions. Self-attention is applied to $Z_{a}$ and $Z_{b}$ to obtain
$S_{a}$ and $S_{b}$, controlling the expressivity of each modality and
preventing noisy features. Subsequently, the common information of the joint
distribution is extracted via cross attention of $Q_{c}$, $K_{c}+K_{a}+K_{b}$,
and $V_{c}+V_{a}+V_{b}$ to model modality-common features $S_{c}$. To preserve
modality-specific information, we minimize the Mutual Information (MI) loss
between concatenated $S_{a}+S_{b}$ and $S_{c}$. As the computation of mutual
information is intractable, we calculate a variational upper bound called
contrastive log-ratio upper bound (vCLUB) as an MI estimator to achieve MI
minimization. Given two variables $a$ and $b$, the $L_{v}^{CLUB}(a,b)$ is
calculated as follows (Zhang et al., 2024):

$\displaystyle L_{v}^{CLUB}(a,b)$ $\displaystyle=\mathbb{E}_{p}(a,b)\left[\log
q_{\theta}(b|a)\right]-\mathbb{E}_{p}(a)\mathbb{E}_{p}(b)\left[\log
q_{\theta}(b|a)\right]$ (1)
$\displaystyle=\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}\left[\log
q_{\theta}(b_{i}|a_{i})-\log q_{\theta}(b_{j}|a_{i})\right]$

We employ an MLP $q_{\theta}(b|a)$ to provide a variational approximation of
$q_{\theta}(b|a)$, which can be optimized by maximizing the log-likelihood
(Zhang et al., 2024): $L_{\text{estimator}}(a,b)=\frac{1}{N}\sum_{i=1}^{N}\log
q_{\theta}(b_{i}|a_{i})$. The MI Loss is then calculated as: $\text{MI
Loss}=L_{v}^{CLUB}(S_{a}+S_{b})+L_{\text{estimator}}(S_{a}+S_{b},S_{c})$.

After optimizing the mutual information between the modality-specific
information and the modality-common information, we utilize dense fusion
(Holste et al., 2023) to enable denser interaction between modalities. Instead
of directly connecting a prediction classifier on top of the fused
representation $S_{c}$, we learn deeper representations of the clinical notes
and lab test features and add skip connections to concatenate with the fused
representation, forming a final fused embedding: $h_{a}=f_{a}(S_{a})\text{ and
}h_{b}=f_{b}(S_{b})$ where $f_{a}$ and $f_{b}$ are fully-connected layers.
This final representation not only aggregates the modality-specific features
but also incorporates the modality-common representation from the previous
stage of the network: $h_{final}=concat(h_{a},S_{c},h_{b})$. Finally, a dense
block $g$ is used to generate $y=g(h_{final})$, and the model is trained by
optimizing the prediction loss (focal loss for multilabel prediction). This
allows for dense interaction of features from each modality, aggregating
information across different stages of the network. The final loss optimizes a
combination of the prediction objective and the mutual information loss,
controlled by a hyperparameter $\lambda$ with a value range of [0,1]. In this
case, we choose a value of 0.1.
$\text{Loss}_{\text{final}}=L_{\text{objective}}(g(h_{\text{final}}))+\lambda*\text{MI}(\text{concat}(S_{a},S_{b}),S_{c})$

##  4\. EXPERIMENTS

### 4.1 Datasets and Metrics

To evaluate the performance of the methods under comparison, we employed two
real-world EHR datasets: MIMIC-III (Johnson et al., 2016) and FEMH. We
collected five years of EHRs from the Far Eastern Memorial Hospital (FEMH) in
Taiwan from 2017 to 2021. The dataset includes 1,420,596 clinical notes,
387,392 lab results, and over 1,505 lab test items. The FEMH Research Ethics
Review Committee 111https://www.femhirb.org/ approved the study, and all data
were de-identified. We selected patients with at least two recorded visits
from each dataset.

For the multi-label classification task in MIMIC-III, we identified the top 10
most prevalent conditions: “Hypertension, uncomplicated”, “Cardiac
arrhythmias”, “Fluid and electrolyte disorders”, “Congestive heart failure”,
“Diabetes w/o chronic complications”, “Chronic pulmonary disease”, “Valvular
disease”, “Renal failure”, “Hypertension, complicated”, and “Other
neurological disorders”. In the FEMH dataset, the top 10 most common diseases
include “Hypertension”, “Diabetes”, “Heart disease”, “Cancer”,
“Cerebrovascular Disease”, “Kidney Disease”, “Liver Disease”, “Asthma”,
“Hyperlipidemia”, and “Lung Disease”. We applied several established multi-
label classification metrics to assess model performance such as Macro-average
and weighted-average F1-Scores, precision, recall, and accuracy on the test
dataset (Lipton et al., 2014; Hossin and Sulaiman, 2015; Palacio-Niño and
Berzal, 2019).

### 4.2 Experimental Results

Table 1 and Table 2 illustrate the training and validation performance of
various models, highlighting the effectiveness of our proposed method on the
MIMIC-III and FEMH datasets, respectively. In Table 1, our approach
outperforms baseline models such as Bert (Devlin et al., 2018),
Mistral-7B-v0.1 (Jiang et al., 2023), Llama-2-7B-hf (Touvron et al., 2023),
Meta-Llama2-13B (AI@Meta, 2024a), Meta-Llama3-8B (AI@Meta, 2024b), and
Medical-Llama3-8B (ruslanmv, 2024) across all key metrics. Specifically,
MEDFuse shows significant improvements over the best-performing LoRA fine-
tuned LLM, Medical-Llama3-8B. On the test set, our model performs 1.49% better
in macro F1 score, and similar trends are observed in other metrics. Table 2
shows MEDFuse consistently outperforms Medical-Llama3-8B on the FEMH dataset.
For example, training and validation in precision is a 1.53% increase, in the
recall is a 2.07% increase, the training accuracy is 0.9311 (0.55% increase),
and validation accuracy is 0.9296 (0.41% increase). These results validate the
robustness and generalizability of our approach, underscoring its potential
for accurate and reliable clinical predictions across diverse datasets.

### 4.3 Ablation Study

We conducted an ablation study to examine the contributions of various
components in our proposed method, which integrates Medical-Llama3 with a
transformer module, utilizing lab tests (LABTEXT) and clinical notes (TEXT).
The results highlight clear performance contrast when any component is
omitted. Removing both the transformer and LABTEXT results in a 4.81% drop in
training precision and a 4.40% decrease in validation precision. The most
substantial performance reduction occurs when both the transformer and TEXT
are excluded, leading to a 29.76% decrease in training precision and a 30.66%
decrease in validation precision. This underscores the indispensable role of
TEXT and the transformer in our method. Even when only TEXT is removed,
performance significantly deteriorates, with a 17.14% decline in training
precision and a 14.60% decline in validation precision. These findings
illustrate that each component contributes significantly to the model’s
overall efficacy. Our full model, combining LLMs and MLTM, demonstrates the
highest performance, with a training accuracy of 0.9535 and a validation
accuracy of 0.9122.

##  5\. Conclusion

In conclusion, we have presented a novel multi-disease diagnostic model that
integrates multimodal data, closely mirroring real-life clinical decision-
making. By combining fine-tuned LLMs with domain-specific transformers, we
achieved enhanced synthesis of structured and unstructured medical data. Using
a disentangled transformer further refined this integration, significantly
improving disease prediction accuracy. Our experimental results across two
practical EHR datasets demonstrated the proposed model’s robustness and
effectiveness. In future work, we aim to extend our model to cover more
complex and rare diseases, enhance its interpretability for clinical use, and
evaluate its performance on larger, more varied datasets. We will also explore
the integration of real-time and other data modalities (Zheng et al., 2021) to
further align our model with dynamic clinical environments.

## References

  * (1)
  * AI@Meta (2024a) AI@Meta. 2024a.  Llama 2 Model Card.  (2024).  https://huggingface.co/meta-llama/Llama-2-13b
  * AI@Meta (2024b) AI@Meta. 2024b.  Llama 3 Model Card.  (2024).  https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
  * Bellamy et al. (2023) David R Bellamy, Bhawesh Kumar, Cindy Wang, and Andrew Beam. 2023.  Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data.  _arXiv preprint arXiv:2312.11502_ (2023).
  * Bolton et al. (2024) Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, et al. 2024\.  BioMedLM: A 2.7 B Parameter Language Model Trained On Biomedical Text.  _arXiv preprint arXiv:2403.18421_ (2024).
  * Cai et al. (2024) Tianxi Cai, Feiqing Huang, Ryumei Nakada, Linjun Zhang, and Doudou Zhou. 2024.  Contrastive Learning on Multimodal Analysis of Electronic Health Records.  _arXiv preprint arXiv:2403.14926_ (2024).
  * Chen et al. (2019) Jindong Chen, Yizhou Hu, Jingping Liu, Yanghua Xiao, and Haiyun Jiang. 2019.  Deep short text classification with knowledge powered attention. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 6252–6259.
  * Chen (2015) Yahui Chen. 2015.  _Convolutional neural network for sentence classification_.  Master’s thesis. University of Waterloo.
  * Chen et al. (2024) Yinda Chen, Haoyuan Shi, Xiaoyu Liu, Te Shi, Ruobing Zhang, Dong Liu, Zhiwei Xiong, and Feng Wu. 2024.  TokenUnify: Scalable Autoregressive Visual Pre-training with Mixture Token Prediction.  _arXiv preprint arXiv:2405.16847_ (2024).
  * Choi et al. (2016) Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016.  Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.  _Advances in neural information processing systems_ 29 (2016).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Ding et al. (2024) Jun-En Ding, Nguyen Minh Thao Phan, Wen-Chih Peng, Jian-Zhe Wang, Chun-Cheng Chug, Min-Chen Hsieh, Yun-Chien Tseng, Ling Chen, Dongsheng Luo, Chenwei Wu, et al. 2024\.  Large Language Multimodal Models for New-Onset Type 2 Diabetes Prediction using Five-Year Cohort Electronic Health Records.  (2024).
  * Dun et al. (2021) Yaqian Dun, Kefei Tu, Chen Chen, Chunyan Hou, and Xiaojie Yuan. 2021.  Kan: Knowledge-aware attention network for fake news detection. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 81–89.
  * Grinsztajn et al. (2022) Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. 2022.  Why do tree-based models still outperform deep learning on typical tabular data?  _Advances in neural information processing systems_ 35 (2022), 507–520.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022.  Masked autoencoders are scalable vision learners. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 16000–16009.
  * Hegselmann et al. (2023) Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023.  Tabllm: Few-shot classification of tabular data with large language models. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 5549–5581.
  * Holste et al. (2023) Gregory Holste, Douwe van der Wal, Hans Pinckaers, Rikiya Yamashita, Akinori Mitani, and Andre Esteva. 2023.  Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision. In _2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 1–5.
  * Hossin and Sulaiman (2015) Mohammad Hossin and Md Nasir Sulaiman. 2015.  A review on evaluation metrics for data classification evaluations.  _International journal of data mining & knowledge management process_ 5, 2 (2015), 1.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023\.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_ (2023).
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ 3, 1 (2016), 1–9.
  * Li et al. (2020) Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi. 2020.  BEHRT: transformer for electronic health records.  _Scientific reports_ 10, 1 (2020), 7155.
  * Liang et al. (2023) Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2023.  Quantifying & modeling feature interactions: An information decomposition framework.  _arXiv e-prints_ (2023), arXiv–2302.
  * Lipton et al. (2014) Zachary C Lipton, Charles Elkan, and Balakrishnan Naryanaswamy. 2014.  Optimal thresholding of classifiers to maximize F1 measure. In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II 14_. Springer, 225–239.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Luo et al. (2020) Junyu Luo, Muchao Ye, Cao Xiao, and Fenglong Ma. 2020.  Hitanet: Hierarchical time-aware attention networks for risk prediction on electronic health records. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 647–656.
  * Ma et al. (2017) Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, and Jing Gao. 2017.  Dipole: Diagnosis prediction in healthcare via attention-based bidirectional recurrent neural networks. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 1903–1911.
  * Palacio-Niño and Berzal (2019) Julio-Omar Palacio-Niño and Fernando Berzal. 2019.  Evaluation metrics for unsupervised learning algorithms.  _arXiv preprint arXiv:1905.05667_ (2019).
  * Restrepo et al. (2024) David Restrepo, Chenwei Wu, Constanza Vásquez-Venegas, Luis Filipe Nakayama, Leo Anthony Celi, and Diego M López. 2024.  DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era.  _arXiv preprint arXiv:2404.12278_ (2024).
  * ruslanmv (2024) ruslanmv. 2024.  Medical-Llama3-8B-16bit: Fine-Tuned Llama3 for Medical Q&A.  (2024).  https://huggingface.co/ruslanmv/Medical-Llama3-8B
  * Song et al. (2023) Xingchen Song, Di Wu, Binbin Zhang, Zhendong Peng, Bo Dang, Fuping Pan, and Zhiyong Wu. 2023.  Zeroprompt: Streaming acoustic encoders are zero-shot masked lms.  _arXiv preprint arXiv:2305.10649_ (2023).
  * Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023.  Large language models in medicine.  _Nature medicine_ 29, 8 (2023), 1930–1940.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Wu et al. (2021) Haoran Wu, Wei Chen, Shuang Xu, and Bo Xu. 2021.  Counterfactual supporting facts extraction for explainable medical record based diagnosis with graph network. In _Proceedings of the 2021 conference of the north American chapter of the association for computational linguistics: human language technologies_. 1942–1955.
  * Zhang et al. (2020) Xianli Zhang, Buyue Qian, Shilei Cao, Yang Li, Hang Chen, Yefeng Zheng, and Ian Davidson. 2020.  INPREM: An interpretable and trustworthy predictive model for healthcare. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 450–460.
  * Zhang et al. (2024) Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. 2024.  Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction.  _arXiv preprint arXiv:2401.01646_ (2024).
  * Zheng et al. (2021) Lijuan Zheng, Zihan Wang, Junqiang Liang, Shifan Luo, and Senping Tian. 2021.  Effective compression and classification of ECG arrhythmia by singular value decomposition.  _Biomedical Engineering Advances_ 2 (2021), 100013.