# Attribute Structuring Improves LLM-Based Evaluation
of Clinical Text Summaries

Zelalem Gero, Chandan Singh, Yiqing Xie, Sheng Zhang,
Tristan Naumann, Jianfeng Gao, Hoifung Poon

Microsoft Research

###### Abstract

Summarizing clinical text is crucial in health decision-support and clinical
research. Large language models (LLMs) have shown the potential to generate
accurate clinical text summaries, but still struggle with issues regarding
grounding and evaluation, especially in safety-critical domains such as
health. Holistically evaluating text summaries is challenging because they may
contain unsubstantiated information. Here, we explore a general mitigation
framework using Attribute Structuring (AS), which structures the summary
evaluation process. It decomposes the evaluation process into a grounded
procedure that uses an LLM for relatively simple structuring and scoring
tasks, rather than the full task of holistic summary evaluation. Experiments
show that AS consistently improves the correspondence between human
annotations and automated metrics in clinical text summarization.
Additionally, AS yields interpretations in the form of a short text span
corresponding to each output, which enables efficient human auditing, paving
the way towards trustworthy evaluation of clinical information in resource-
constrained scenarios. We release our code, prompts, and an open-source
benchmark at https://github.com/microsoft/attribute-structuring/.

(Hide=100, Hang=true, Progressive=2ex, Style*=$\bullet$ , Style2*=$\circ$)

Attribute Structuring Improves LLM-Based Evaluation
of Clinical Text Summaries

Zelalem Gero, Chandan Singh, Yiqing Xie, Sheng Zhang, Tristan Naumann,
Jianfeng Gao, Hoifung Poon Microsoft Research

##  1 Introduction

Automatic medical text summarization has emerged as a critical task for real-
world impact in natural-language processing, amidst widespread physician
fatigue and burnout. Meanwhile, pre-trained large language models (LLMs) have
shown impressive proficiency in a range of complex natural language processing
tasks [1, 2, 3], including in text summarization [4, 5]. However, evaluating
these summarizations is challenging, as they rely on human effort to track
different attributes and facts.

Here, we propose using LLMs for evaluating medical text summaries, guided by a
simple structuring process. LLMs have previously been used to provide
practical evaluations in a variety of evaluation settings [6, 7]. However,
these evaluations have been imperfect: yielding relatively low correlation
with human judgments, showing bias [8, 9], or more generally struggling with
long-form nuanced evaluation, as is often the case in summarization [10].
Structuring the scoring process can help to ground the evaluation, and help an
LLM decompose a complex evaluation process into simpler, more accurate steps.

We introduce a simple structuring process, which we call Attribute Structuring
(AS) (Fig. 1). Given a groundtruth summary and a summary to be evaluated, we
leverage an LLM to score a summary through prompting. Rather than directly
asking the LLM for a score, we ask it to provide a score for each pair of
attributes, one from the groundtruth and another from the summary to be
evaluated. We extract 17 attributes derived from a clinical ontology for each
summary; the attributes are shown to define the elements of quality in the
context of a clinical discharge summary in a large scale study [11]. Unlike
previous methods such as FActScore [12], this does not require a reliable
knowledge source to verify each attribute, but instead relies on the pre-
specified ontology to capture relevant aspects of a summary. DocLens [13],
similar to our work, performs a fine-grained evaluation of medical text using
various generation models and show that such scoring has higher agreement with
the judgements of medical experts. However, DocLens relies on the model to
generate the important attributes rather than a clinical ontology.

Figure 1:  Overview of Attribute Structuring LLM evaluation for clinical
summarization. AS uses a human-given ontology to extract attributes from
summaries and then to individually score them.

We perform experiments on clinical summarization tasks using discharge notes
from MIMIC-III [14] and various LLMs, including GPT-4 [3], GPT-3.5 [15], and
LLaMA-2 [2]. We find that that Attribute Structuring helps reduce the gap
between automated metrics and human annotators. In addition, AS enables
auditing the evaluation for a particular summary, by providing grounding for
each attribute, in the form of a short text span in the input.

##  2 Methods and experimental setup

###  2.1 Methods: Attribute Structuring

Fig. 1 shows the various steps of the Attribute Structuring (AS) pipeline. The
pipeline takes in two summaries: a groundtruth summary and a generated summary
(which we are trying to evaluate). Additionally, it uses a set of attributes
that are important for the task at hand. These attributes comes from an
ontology that utilizes domain knowledge to define the elements of quality, in
our case the context of a clinical discharge summary in a large scale study
[11], e.g. Admission diagnosis or Goals of care.

Each step in the pipeline is achieved by prompting an LLM. In the first step,
the Attribute Structuring pipeline first extracts text for each attribute in
the ontology from both summaries. Then, for each pair of extracted attributes,
it generates a similarity score by prompting an LLM to compare them on a scale
of 1–5 (all prompts are provided in Appendix A). Finally, an average of these
scores is returned as the evaluation score for the generated summary (usually
after normalizing to range from 0–100) .

In a final (optional) interpretation step, an LLM can be used to ground the
scores for each attribute using text spans in the input. This can be achieved
by directly prompting an LLM to find a text span that led to the extracted
text for each attribute. This step is made possible by the Attribute
Structuring structuring process.

###  2.2 Experimental setup

#### Dataset and benchmark

We construct a benchmark consisting of discharge summaries for 300 randomly
selected MIMIC-III notes [14] and release scripts to process and use this
data. To be fair to LLMs with different context lengths, we select MIMIC notes
with no more than 4,000 tokens (measured with tiktoken’s p50_base
tokenizer111https://github.com/openai/tiktoken). As it is derived from MIMIC,
this benchmark data should not be used outside of research contexts.

#### Models

We evaluate 6 different LLMs: GPT-4 [3] gpt4-0314, GPT-3.5 [15]
gpt-3.5-turbo-0613, both accessed securely through the Azure OpenAI API. We
also evaluate LLaMA-2 [7B,70B] [2], Mistral-7B [16], and Mixtral (8x7B) [17].
We evaluate each LLM both for generating summaries and for scoring summaries.
We compare methods for scoring with three popular metrics: BERT-Score [18],
and ROUGE-1/ROUGE-L [19].

Figure 2:  Distribution of scores across different attributes. Attribute
Structuring enables auditing which aspects of discharge summaries are
generally accurate (e.g. Discharge test) and which are often inaccurate (e.g.
Hospital consults).

#### Annotations and metrics

On a subset of our benchmark, three blinded human annotators evaluate the
similarities between the extracted text for each attribute. They score 17
attributes for 30 total documents (20 test summaries generated by GPT-4 and 10
test summaries generated by Mixtral), for a total of 510 attribute labels.
Each label is scored on a 4-point scale: Not similar, Somewhat similar, Very
similar, Essentially the same. We evaluate the match between the manually
annotated attribute labels and the automated attribute labels using Pearson
correlation, Spearman (rank) correlation, and root mean-squared-error (RMSE).

##  3 Results

#### Main result: AS improves evaluation quality

Figure 3: Example interpretation of document similarity. Scores for each element of the ontology can be analyzed and grounded in quotes from the source summaries for better understanding. Table 1: Main result: Attribute Structuring improves the match between automated scoring metrics and human annotators. For all methods, attribute structuring is performed by prompting GPT-4.  | RMSE ↓ |  Pearson correlation ↑ |  Spearman correlation ↑
---|---|---|---
| Default/AS | Default/AS | Default/AS
ROUGE-1 | 0.17 / 0.17 | 0.36 / 0.64 | 0.34 / 0.48
ROUGE-L | 0.26 / 0.18 | 0.35 / 0.59 | 0.32 / 0.44
BERT-Score | 0.48 / 0.18 | 0.17 / 0.71 | 0.23 / 0.69
GPT-4 | 0.18 / 0.07 | 0.46 / 0.84 | 0.38 / 0.77
Table 2:  Automatic evaluation scores using different LLMs for generating summaries and for evaluating. The Benchmark column uses Mixtral for structuring summaries as well as evaluating them, making it fully reproducible using open source tools. The remaining columns use GPT-4 for structuring and different LLMs for scoring the attributes. The relative rankings of different LLMs generators is preserved across different columns.  |  | Benchmark | Scorer (with GPT-4 structuring)
---|---|---|---
|  |  Mixtral (8x7B) | AVG |  LLaMA-2 (7B) |  Mistral (7B) |  LLaMA-2 (70B) |  Mixtral (8x7B) | GPT-3.5 | GPT-4
Generator | LLaMA-2 (7B) | 36.2 | 45.97 | 65.4 | 47.1 | 49.0 | 38.5 | 40.3 | 35.5
Mistral | 37.4 | 46.37 | 66.1 | 47.7 | 49.2 | 38.7 | 40.8 | 35.7
LLaMA-2 (70B) | 38.4 | 49.70 | 66.7 | 50.6 | 54.3 | 43.4 | 43.6 | 39.6
Mixtral (8x7B) | 39.2 | 51.30 | 68.4 | 52.4 | 54.9 | 46.0 | 44.9 | 41.2
GPT-3.5 | 39.4 | 52.78 | 73.1 | 53.7 | 56.7 | 45.9 | 45.7 | 41.6
GPT-4 | 45.1 | 56.02 | 73.8 | 56.9 | 61.2 | 51.2 | 48.8 | 44.2

Table 1 shows how well different scoring methods match the annotations
provided by humans. Attribute Structuring yields a considerable improvement
for all metrics (except for ROUGE-1 when measured with RMSE). Scoring with
GPT-4 yields the best match with human annotators. The Pearson correlation
coefficient of 0.84 for GPT-4 closely approaches the mean inter-annotator
Pearson correlation of 0.90, and the Spearman correlation coefficient of 0.77
similarly nears the mean inter-annotator Spearman correlation of 0.87. The
enhanced correlations across all metrics substantiate that attribute-based,
fine-grained scoring outperforms holistic methodologies.

Fig. 2 shows how similarities break down across attributes for different
attributes / metrics. Some attributes, such as Discharge test are more often
correctly identified in the test summaries, whereas others, such as Hospital
consults are rarely correct. Aggregating these attribute evaluations through
AS allows for improved evaluation compared to evaluating an entire summary at
once.

#### Summaries from stronger LLMs yield higher AS benchmark scores

We now evaluate how well different LLMs perform under an automatic evaluation
using Attribute Structuring. The first column of Table 2 shows the benchmark
scores achieved by different models; the benchmark scores rely only on the
open-source Mixtral model to perform both structuring and scoring, enabling
full reproducibility. GPT-4 achieves the highest score, followed by GPT-3.5
and Mixtral (8x7B).

We additionally evaluate how consistent the benchmark scores are with scores
provided using different language models for structuring and scoring. The
right section of Table 2 shows evaluation scores when structuring with GPT-4
and scoring with various different LLMs. We find that the absolute scores in
different columns vary widely, but the relative ordering of LLM generators is
almost perfectly preserved between the benchmark and all other settings (the
only exception is that the Mixtral scorer slightly prefers the Mixtral
generator to the GPT-3.5 generator).

#### Analysis: Attribute Structuring facilitates interpretation

Going beyond aggregate interpretations, Fig. 3 shows an example of how
similarity can be audited for a single pair of summaries. Unlike posthoc
feature importance [20, 21], or intrinsically interpretable models [22, 23],
our interpretable grounding comes directly from an LLM, similar to recent
works that use LLMs to generate explanations [24, 25] and ground those
explanations in evidence [26, 27]. Since AS decomposes similarity into a
series of attributes, the attributes can each be audited by directly prompting
an LLM (here, GPT-4) to find a text span that serves as evidence for the
extracted attribute in each document. Fig. 3 shows text spans for two
discrepancies: Admission diagnosis and Pertinent lab tests, which are only
partially retained in the test summary.

##  4 Discussion

Attribute Structuring constitutes an important step towards safely deploying
LLMs in healthcare settings. As LLMs continue to generally improve, their
ability to produce and evaluate clinical summaries with AS is likely to
improve as well. Going forward, Attribute Structuring can be harnessed in a
variety of ways to improve clinical NLP summarization, e.g. for evaluating
free-form text such as conversations [28], improving model distillation by
helping to filter effective summaries [29, 30], and improving the
explainability of the process [31].

## Limitations

One limitation of Attribute Structuring is that it incurs a high computational
cost as multiple LLM calls are chained together; however, these costs may
continue to decrease as models become more efficient [32]. Another limitation
is that LLMs (and thereby Attribute Structuring) continue to be sensitive to
prompts, increasing the need for methods to make LLMs more amenable to
prompting [15, 33, 34] and to make finding strong prompts easier [35, 36, 37,
38]. Additionally, AS relies on a human-given ontology for evaluation, but
could be made more general by developing methods to automatically define
extract and define an ontology for a given domain. While Attribute Structuring
helps to improve the automatic evaluation of summaries, there are nevertheless
risks present with fully relying on LLMs without human supervision.

## References

  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _ArXiv_ , 2023.  URL https://arxiv.org/pdf/2303.08774.pdf.
  * Liu et al. [2022] Yixin Liu, Alexander R Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, et al.  Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation.  _arXiv preprint arXiv:2212.07981_ , 2022.
  * Zhang et al. [2024] Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto.  Benchmarking large language models for news summarization.  _Transactions of the Association for Computational Linguistics_ , 12:39–57, 2024.
  * Liu et al. [2023] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_ , 2023.
  * Fu et al. [2023] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu.  Gptscore: Evaluate as you desire.  _arXiv preprint arXiv:2302.04166_ , 2023.
  * Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging LLM-as-a-judge with MT-bench and chatbot arena.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=uccHPGDlao.
  * Wang et al. [2023] Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey E Kuehl, Erin Bransom, and Byron C Wallace.  Automated metrics for medical multi-document summarization disagree with human evaluations.  _arXiv preprint arXiv:2305.13693_ , 2023.
  * Zhong et al. [2022] Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han.  Towards a unified multi-dimensional evaluator for text generation.  _arXiv preprint arXiv:2210.07197_ , 2022.
  * Savvopoulos et al. [2018] Stavros Savvopoulos, Tara Sampalli, Ruth Harding, Gail Blackmore, Sandra Janes, Kothai Kumanan, Rick Gibson, and Chris MacKnight.  Development of a quality scoring tool to assess quality of discharge summaries.  _Journal of Family Medicine and Primary Care_ , 7(2):394, 2018.
  * Min et al. [2023] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  _arXiv preprint arXiv:2305.14251_ , 2023.
  * Xie et al. [2024] Yiqing Xie, Sheng Zhang, Hao Cheng, Pengfei Liu, Zelalem Gero, Cliff Wong, Tristan Naumann, Hoifung Poon, and Carolyn Rose.  Doclens: Multi-aspect fine-grained evaluation for medical text generation, 2024.
  * Johnson et al. [2016] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=TG8KACxEON.
  * Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Jiang et al. [2024] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ , 2024.
  * Zhang et al. [2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.
  * Lin [2004] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Lundberg and Lee [2017] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _Advances in Neural Information Processing Systems_ , pages 4768–4777, 2017.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  Why should i trust you?: Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 1135–1144. ACM, 2016.
  * Rudin et al. [2021] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong.  Interpretable machine learning: Fundamental principles and 10 grand challenges.  _arXiv preprint arXiv:2103.11251_ , 2021.
  * Singh et al. [2023] Chandan Singh, Armin Askari, Rich Caruana, and Jianfeng Gao.  Augmenting interpretable models with large language models during training.  _Nature Communications_ , 14(1):7913, 2023.
  * MacNeil et al. [2022] Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang.  Generating diverse code explanations using the gpt-3 large language model.  In _Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2_ , pages 37–39, 2022.
  * Chen et al. [2024] Yanda Chen, Chandan Singh, Xiaodong Liu, Simiao Zuo, Bin Yu, He He, and Jianfeng Gao.  Towards consistent natural-language explanations via explanation-consistency finetuning.  _arXiv preprint arXiv:2401.13986_ , 2024.
  * Xue et al. [2023] Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, and Heng Ji.  Rcot: Detecting and rectifying factual inconsistency in reasoning by reversing chain-of-thought.  _arXiv preprint arXiv:2305.11499_ , 2023.
  * Gero et al. [2023] Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon.  Self-verification improves few-shot clinical information extraction.  _arXiv preprint arXiv:2306.00024_ , 2023.
  * Tu et al. [2024] Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al.  Towards conversational diagnostic ai.  _arXiv preprint arXiv:2401.05654_ , 2024.
  * Wu et al. [2023] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Pmc-llama: Further finetuning llama on medical papers.  _arXiv preprint arXiv:2304.14454_ , 2023.
  * Toma et al. [2023] Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry B Rubin, and Bo Wang.  Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding.  _arXiv preprint arXiv:2305.12031_ , 2023.
  * Singh et al. [2024] Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao.  Rethinking interpretability in the era of large language models.  _arXiv preprint arXiv:2402.01761_ , 2024.
  * Dao et al. [2022] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ , 35:16344–16359, 2022.
  * Zhou et al. [2023] Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen.  Context-faithful prompting for large language models.  _arXiv preprint arXiv:2303.11315_ , 2023.
  * Zhang et al. [2023] Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao.  Tell your model where to attend: Post-hoc attention steering for llms.  _arXiv preprint arXiv:2311.02262_ , 2023.
  * Shin et al. [2020] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  _arXiv preprint arXiv:2010.15980_ , 2020.
  * Singh et al. [2022] Chandan Singh, John X Morris, Jyoti Aneja, Alexander M Rush, and Jianfeng Gao.  Explaining patterns in data with language models via interpretable autoprompting.  _arXiv preprint arXiv:2210.01848_ , 2022.
  * Xu et al. [2023] Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang.  $k$ nn prompting: Beyond-context learning with calibration-free nearest neighbor inference.  _arXiv preprint arXiv:2303.13824_ , 2023.
  * Morris et al. [2023] John X Morris, Chandan Singh, Alexander M Rush, Jianfeng Gao, and Yuntian Deng.  Tree prompting: efficient task adaptation without fine-tuning.  _arXiv preprint arXiv:2310.14034_ , 2023.

##  Appendix A Appendix

Here we include the prompts we used to:

  * •

Generate summaries

  * •

Structure to extract attributes ,and

  * •

Score each pair of attributes.

  * •

A prompt to score without structuring as a baseline.

1\. Generation prompt:
Prompt = "You are tasked with generating a high quality clinical discharge
summary for the provided input text INPUT_DOC. The summary has to be very
relevant to the input document and cover the most important aspects of the
input. Follow the following steps:
Step 1: Generate the most common sections that usually appear in a clinical
discharge summary. Step 2: Not all the sections from Step 1 will have content.
See which of those sections have contents from the INPUT_DOC. Remove the
sections that don’t have information.
Step 3 : Use the following concepts to generate appropriate content. These are
not Sections. Concepts: patient information and service type, diagnosis given
at the time of admission, brief summary of initial presentation and diagnostic
evaluation, pertinent physical findings relevant to diagnoses, goals of care;
level of treatment,code status(e.g. curative,life-prolonging palliative, and
symptomatic palliative), course in hospital; synotpic,problem-based
description of sequential events and respective evaluations, treatments, and
prognoses, hospital consults; description of specialty and/or allied health
consults, procedures in hospital; a list of procedures with key findings and
date, principal discharge diagnosis or main reason for admission and all
additional pertinent diagnoses where applicable, discharge medications with
specific description of new, altered, and discontinued medications and
rationale for changes, lab tests and investigative results, tests ordered
during the hospitalization that are pending at the time of discharge, outcome
of care/condition at discharge; sense of the patient health status at
discharge includes functional status, and cognitive status, outstanding issues
for follow-up and recommendations to a recipient health-care provider during
discharge, appointments after discharge including person responsible for
scheduling, care provider, discharge instructions; list of
information/education provided to the patient during discharge, main author of
the discharge summary or attending clinician.

Step 4: Put the generated content in a coherent order. Format in such a way
that the dishcarge summary has an excellent coherence, fluency, and
consistency. Remember to use the sections you generated in Step 1, don’t make
the concepts in Step 2 as a section, they are just suggestive concepts not
sections.

Step 5: Remove sections with no information. Dont put ’not specified’ or ’not
mentioned’ or ’none specified’ in a section. Just remove everything for that
section including the section header.
Step 5: Return the final discharge summary with all the remaining sections
that have contents. Remember to remove sections with no information. Context :
"

2\. Structuring prompt:
We use ResponseSchema from langchain to inform the model what each attribute
description is.

ad_diag_schema

= ResponseSchema(name = ’ad_diag’, description=’preliminary or working
diagnosis given at the time of admission’)

dc_diag_schema

= ResponseSchema(name = ’dc_diag’, description=’the list of principal
discharge diagnosis or main reason for admission and all additional pertinent
diagnoses where applicable’)

main_diag_schema

= ResponseSchema(name = ’main_diag’, description=’diagnosis mostly accountable
for the largest portion of the patient"s stay, responsible for the greatest
part of the length of stay ’)

history_schema

= ResponseSchema(name = ’history’, description=’a brief summary of initial
presentation and diagnostic evaluation’)

physical_schema

= ResponseSchema(name = ’physical’, description=’pertinent physical findings
relevant to diagnoses ’)

goals_schema

=ResponseSchema(name = ’goals’, description=’goals of care; level of
treatment,code status(e.g. curative,life-prolonging palliative, and
symptomatic palliative)’)

course_schema

= ResponseSchema(name = ’course’, description=’course in hospital;
synotpic,problem-based description of sequential events and respective
evaluations, treatments, and prognoses ’)

consults_schema

= ResponseSchema(name = ’consults’, description=’hospital consults;
description of specialty and/or allied health consults’)

procedures_schema

= ResponseSchema(name = ’procedures’, description=’procedures in hospital; a
list of procedures with key findings and date’)

ds_med_schema

= ResponseSchema(name = ’ds_med’, description=’a list of all discharge
medications with specific description of new, altered, and discontinued
medications and rationale for changes’)

lab_schema

= ResponseSchema(name = ’lab’, description=’pertinent lab tests and
investigative results’)

ds_test_schema

= ResponseSchema(name = ’ds_test’, description=’tests ordered during the
hospitalization that are pending at the time of discharge ’)

ds_status_schema

= ResponseSchema(name = ’ds_status’, description=’outcome of care/condition at
discharge; sense of the patient health status at discharge includes functional
status, and cognitive status’)

followup_schema

= ResponseSchema(name = ’followup’, description=’outstanding issues for
follow-up and recommendations to a recipient health-care provider during
discharge’)

appt_schema

= ResponseSchema(name = ’appt’, description=’appointments after discharge
including person responsible for scheduling, care provider ’)

instruct_schema

= ResponseSchema(name = ’instruct’, description=’discharge instructions; list
of information/education provided to the patient during discharge’)

author_schema

= ResponseSchema(name = ’author’, description=’main author of the discharge
summary or attending clinician’)

prompt = "You are an expert in information extraction and structuring from
clinical notes. Given a clinical note, create a structured output. For a given
variable, if you can not determine/find a value, return NONE. Dont add any
extra text, just the structured value. Here is the note: INPUT_DOC.
format_instructions "

3\. Structured attribute Scoring Prompt:

prompt = "You will be given a python dictionary containing a clinical variable
as a key and a list containing two values for the variable as values.

Your task is to rate how similar the values are given the variable. Compare
value1 and value2 for semantic similarity(similarity in meaning) given the
variable and the criteria below. Two values can be very similar in meaning
even if they are phrased differently. Also remember that this is a clinical
document, take that into account. When scoring the similarity between two
clinical terminologies, assign a value from 1 to 4, where 1 signifies a lack
of similarity and 4 indicates identical meanings. Consider the context,
clinical relevance, and semantic alignment between the terminologies. If the
terminologies convey vastly different meanings, assign a score of 1. A score
of 2 is appropriate for terminologies that, while related, represent different
concepts or emphasize distinct elements. A score of 3 should be used for
terminologies with substantial semantic overlaps but minor differences.
Finally, if the terminologies are semantically equivalent and interchangeable,
warranting no clinical distinction, assign a score of 4. Ensure that the
assessment reflects the degrees of similarity in meaning, irrespective of
syntactical differences, context, or minor variances in expression.

Please make sure you read and understand these instructions carefully. Return
the similarity score. Return only the score, don’t include any other extra
text. "

4\. Document scoring without structuring prompt

prompt = """You will be given two clinical discharge summaries, summary1 and
summary2.

Your task is to rate how similar the two summaries are from 1-10. 1 is not
similar while 10 is essentially identical. Focus on the following important
clinical variables when performing the comparison: How similar is the
diagnosis, how similar are the goals, how similar is hospital course and
history, how similar are the medications administered, how similar are the
physical condition diagnoses, how similar are the followup consults and
procedures, how similar are the lab tests performed, how similar are the
patients discharge status, is the follow-up instructions similar, are there
any similar appointments, and instructions only return the score, dont add any
extra text Here are the inputs discharge summaries: summary1: [value1]
summary2: [value2]. """

###  A.1 Prompts given to human annotators

Fig. A1 shows a sample interface given to human annotators. Human annotators
were presented values for attributes through a web interface along with the
prompt “For each attribute listed on the left, two possible values are
provided(different colors). Please rate the semantic similarity between the
values.”. Annotators are three machine learning researchers that are authors
of the current study.

Figure A1: Annotation interface screenshot.


# A NOVEL METHOD FOR SCHIZOPHRENIA CLASSIFICATION USING NONLINEAR FEATURES AND
NEURAL NETWORKS

Hari Prasad SV1<EMAIL_ADDRESS>St. Stephen’s College,Delhi1

###### Abstract

One notable method for recording brainwaves to identify neurological problems
is electroencephalography (hereafter EEG). A trained neuro physician can learn
more about how the brain functions through the use of EEGs. However
conventionally, EEGs are only used to examine neurological problems (Eg.
Seizures). But abnormal links to neurological circuits can also exist in
psychological illnesses like Schizophrenia. Hence EEGs can be an alternate
source of data for detection and classification of psychological disorders. A
study on the classification of EEG data obtained from healthy individuals and
individuals experiencing schizophrenia is conducted. The inherent nonlinear
nature of brain waves are made use for the dimensionality reduction of the
data. Nonlinear parameters such as Lyapunov exponent (LE) and Hurst exponent
(HE) were selected as essential features. The EEG data was obtained from the
openly available EEG database of MV. Lomonosov Moscow State university. To
perform Noise reduction of the data, a more recently developed Tunable Q
factor based wavelet transform (TQWT) is used . Finally for the
classification, the 16 channel EEG time series is converted into spatial
heatmaps using the aforementioned features. A convolutional neural network
(CNN) is designed and trained with the modified data format for
classification.

##  I INTRODUCTION

Schizophrenia is classified as a psychological disorder, where an individual
loses cognition and faces significant impairment in judgment and thought
processes. It is characterized by hallucinations , Delusions, Inability in
social interaction etc[1] . The aim of this paper is to evaluate schizophrenia
as a neurological disorder by examining it using non linear techniques[2] .
Previous work indicates the use of hybrid LSTM/CNN based networks[3], use of
support vector machines (SVMs) [4] and other various types of classifiers,
with varying accuracies. A comprehensive list of previous classifiers built
for schizophrenia classification is documented by Sun et al [3]. One major
difference of the following work from other documented work is the novel
approach for the dimensionality reduction of the data. For convenience, This
paper is divided into multiple sections, the second section describes the EEG
data that is used for analysis. The third section shows how the data is
converted into an image form so that it can be analysed by a Convolution
Neural Network along with the different NLD features that are used to
understand the data. Fourth is dedicated to discussing data augmentation and
the architecture of the neural network. Last section concludes with results
obtained along with discussion and remarks.

##  II Data

Electroencephalogram data is a time series that records the electrical
activity of the brain using electrodes placed in different parts of the scalp.
The data acquired by these electrodes are voltage signals ranging in
millivolts (mV) and showcase several characteristic brain functions. A trained
neuro physician can look at this data and can understand various defects and
abnormalities [5]. The data for this work is obtained from the openly
accessible Schizophrenia database hosted by the MV Lomonosov Moscow State
University [6]. The files are in the form of ‘.eea’ (standard ASCII data)
which includes time series numerical values obtained from the standard 10-20
placement system [7] (see figure). The two sets obtained are classified as
‘Healthy Adults ’ (39) and ‘Adults experiencing symptoms of Schizophrenia’
(45). Each datafile contains a column with 16 electrode positions. The column
can be separated into sections of 16 which contains 7680 data points. A few
samples of the data are shown (Fig. 2) for reference.

Figure 1: $10-20$ Placement system Figure 2: A sample candidate of the data
(Time series from only $5$ lobes are shown.)

The obtained data is subjected to a filtering algorithm for the reduction of
noise. Due to the complexity of the time series no discernible periodicity is
observed. Hence to analyse these signals, time localized wavelets are utilized
thus by converting into frequency domain [8] [9].

###  II-A Noise reduction and Wavelet Transforms

Wavelet transform is a more robust way of performing spatio-temporal analysis
on time series data. The generic idea of wavelet transform is to use a wavelet
(A spatially localized wave) to analyse non stationary, transient signals
[10]. Wavelet transforms are extensively used in biomedical signal analysis.
In the present case, instead of using it for converting the data into
frequency domain, it is used for performing noise reduction. The time series
signals will eventually be subjected for the calculation of non linear
features and it is essential that it is noise - free . So for this a recently
developed Tunable Q wavelet transform (TQWT) developed by Selesnick [11] is
used.

First, each channel time series is subjected to a frequency split by setting
the q value as 6 and redundancy r as 5. The series was then split into 10
energy sub bands. After the channels are done splitting, they are combined
together using an inverse TQWT algorithm based on the same technique. The
entire denoising and recompilation of data was done in python by using the
open source TQWT package available on github by Hajek-Lellmann. The immediate
results in the noise reduction can be seen in the heatmaps Fig. 3. All of the
time series signals are subjected to the same procedure accordingly
afterwards.

Figure 3: Heat maps before (left) and after (right) applying TQWT

##  III Heatmap generation and NLD features

The idea of the Heatmap generation using non linear features in the spatial
features is proposed by Kutepov, Ilya E et al.[12]. For these spatial heat
maps, First the required features are calculated which are mentioned in the
subsection(s) 3.1 and 3.2. After the features are calculated that of each
electrode, it is mapped into a 4x4 grid. These maps represent a topological
view of the scalp (Fig. 4).These heatmaps perform significant dimensionality
reduction of the data thus by reducing the complexity.

Figure 4: Electrode mapping in $4\times 4$ grid

###  III-A Calculation of features - Lyapunov Exponent

Lyapunov exponent (LE) is a crucial parameter in determining the stability
behaviour [13] of chaotic systems. LE is often regarded as the measure of
divergence from the initial trajectory of the dynamical variable. The concept
of LEs in dynamical systems was first proposed by Oseledec [14] as a measure
of the stability of the system itself. A quantitative idea of LE can be
mathematically characterized as follows; Consider a system with some
trajectory Z(t). The divergence of the trajectory can be related to the
initial trajectory . We can define a ‘measure of divergence’ as $\lambda$ and
can be related as.

$|\delta Z(t)|\approx e^{\lambda t}|\delta Z_{0}|$ (1)

From this. Lyapunov exponent (LE) can be defined as,

$\lambda(x_{o})=\lim_{n\to\infty}\frac{1}{n}\sum_{i=0}^{n-1}\ln\left|f^{\prime}(x_{i})\right|$
(2)

For a system, there can be a spectrum of LE values according to its evolution
and this spectrum LE values can be calculated (Eckmann et al. [15]).Here, only
the maximal value of the LEs are utilized, or often termed in literature as
the Maximal Lyapunov Exponent (MLE). The value of MLE is thus calculated using
Rosenstein’s Algorithm proposed by Rosenstein et al. [16] using the open
source python package ‘nolds’ [17]

###  III-B Calculation of features - Hurst Exponent

Hurst Exponent (HE) can be defined as a measure of the long term memory of the
series. The idea of the Hurst exponent was initially developed on the work
done on hydrology on optimum dam requirements on the Nile river by Hurst et
al[18]. Since the exponent generally measures the amount of autocorrelations
in a time series, it gives an idea of the evolution of the series and the lag
between the adjacent elements. It indicates the long term dependencies of a
series, thus the measure of it is often used to determine how much the system
is closely correlated to each other. The value of HE is usually between 0 and
1. A value of the exponent between 0 and 0.5 indicates the system can have
high and low correlations periodically for a longer range. 0.5 denotes no
correlation at all.

The Hurst exponent (HE) can be mathematically defined as,

$\left(\frac{R}{S}\right)_{s}=Ks^{H}$ (3)

Where H denotes the exponent, R and S indicate the range of first ‘s’
cumulative deviations from mean and the series sum for ‘s’ entries
respectively. K is an arbitrary constant. For the calculation of HE in case of
EEG time series, the previously mentioned ‘nolds’ package is utilised. After
the values of Lyapunov exponent and Hurst exponent are calculated, they are
used for the construction of the heat maps and fed into the classifier.

##  IV Data augmentation and CNN architecture

After the data has been prepared, a convolutional neural network (CNN) was
designed. Conventional Multilayer Perceptron feedforward (MLP) networks are
not capable enough to classify multidimensional data [19] . Hence much more
robust convolutional neural networks (CNNs) are used. CNNs are particularly
used in image classification (Fig. 5) or in deep neural networks [19].

###  IV-A Data augmentation

To train a neural network often requires tremendous amounts of training data;
for an improved classification and to prevent overfitting. The dataset used
for this analysis is fairly small. To address this issue, data augmentation
techniques are used. Data augmentation is a technique used when the source
data from training a network is comparatively minimal [20]. The classifier is
not intended to classify the specific spatial features of the EEG from the
heatmap. Elements like symmetry of the inputs can be ignored while fed into
the network. The heatmaps can be flipped horizontally along an axis as well
vertically, or can be zoomed in, etc., using image processing techniques. The
Tensorflow Keras [abadi2016tensorflow] framework allows standard data
augmentation to be performed in case of the input data. After sufficient data
is generated for feeding into the network, all images are reshaped into an
array of (224,224,3) to maintain uniformity along the training.

Figure 5: A conventional CNN performing image classification [19]

###  IV-B CNN architecture

The CNN is then designed accordingly. The hyperparameters of the network were
varied with some trial and error for the best results. The final CNN consisted
of 3 layers with input layer taking in shapes of (224,224,3) and output layer
of only one neuron. The input layer is a ‘MobilenetV2’ layer which has been
previously known to perform best in performing image classification in mobile
systems which have less computational efficiency [21]. The hidden layer
consists of 120 neurons and ReLU activation and the output layer was activated
by a sigmoid function. An adagrad optimizer, which is based on gradient
descent, was used and the loss function was set to be binary crossentropic.
The network was fully built and tested using Tensorflow keras framework
[abadi2016tensorflow].

##  V Results and conclusions

The accuracy of the neural network is evaluated after training, the training
accuracy was 75%. The accuracy of the network improved over the course of the
training epochs. The plots (Fig.6) shows an increase in accuracy and steady
decrease in loss.

Figure 6: Accuracy Figure 7: Loss

The results suggest the accuracy only to be 75% which compared to that of the
existing classifiers is fairly less. A low accuracy for this network was
expected due to the very small training data pool. The network accuracy can be
improved significantly by increasing the training data size as well as fine
tuning the hyperparameters of the network. The use of other non linear
features are also a possible solution to the problem. Here the results are
based on the HE based heatmap data, which exhibited much more distinguishable
features than that of the Lyapunov maps. A hybrid network is also proposed
which takes in multiple heatmaps as inputs which represents multiple features;
Non linear as well as statistical.

##  VI Acknowledgements

The author thanks for the suggestions provided by Dr. Geetanjali Sethi, from
the Department of Physics, St. Stephen’s College, Delhi. Fellow colleagues Ms.
Sandra Elsa Sanjai and Mr. Vedanta Thapar (St. Stephen’s College, Delhi) are
also thanked for their valuable thoughts and comments on this work.

## References

  * [1] K. R. Patel, J. Cherian, K. Gohil, and D. Atkinson, “Schizophrenia: Overview and treatment options,” P & T : a peer-reviewed Journal for Formulary Management, vol. 39, pp. 638–45, 2014.
  * [2] M. Breakspear, “The nonlinear theory of schizophrenia,” Aust. N. Z. J. Psychiatry, vol. 40, no. 1, p. 20–35, 2006.
  * [3] J. Sun, R. Cao, M. Zhou, W. Hussain, B. Wang, J. Xue, and J. Xiang, “A hybrid deep neural network for classification of schizophrenia using eeg data,” Scientific Reports, vol. 11, 02 2021.
  * [4] T. Bose, S. D. Sivakumar, and B. Kesavamurthy, “Identification of schizophrenia using eeg alpha band power during hyperventilation and posthyperventilation,” Journal of Medical and Biological Engineering, vol. 36, pp. 901–911, 2016.
  * [5] Niedermeyer’s electroencephalography: Basic principles, clinical applications, and related fields.  Oxford Medicine Online, Oxford University Press.
  * [6] “Eeg database - schizophrenia,” 2019.
  * [7] R. W. Homan, J. Herman, and P. Purdy, “Cerebral location of international 10–20 system electrode placement,” Electroencephalography and Clinical Neurophysiology, vol. 66, pp. 376–382, 1987.
  * [8] N. Kumar, K. Alam, and A. H. Siddiqi, “Wavelet transform for classification of eeg signal using svm and ann,” Biomedical and Pharmacology Journal, vol. 10, p. 2061–2069, 12 2017.
  * [9] A. Hamad, E. Houssein, A. E. Hassanien, and A. Fahmy, Feature extraction of epilepsy EEG using discrete wavelet transform.  12 2016.
  * [10] E. D. Übeyli, “Wavelet/mixture of experts network structure for eeg signals classification,” Expert Syst. Appl., vol. 34, pp. 1954–1962, 2008.
  * [11] I. W. Selesnick, “Wavelet transform with tunable qfactor,” IEEE Transactions on Signal Processing, vol. 59, pp. 3560–3575, 0.
  * [12] I. E. Kutepov, V. V. Dobriyan, M. V. Zhigalov, M. F. Stepanov, A. V. Krysko, T. V. Yakovleva, and V. A. Krysko, “Eeg analysis in patients with schizophrenia based on lyapunov exponents,” Informatics in Medicine Unlocked, vol. 18, p. 100289, 2020.
  * [13] A. M. Liapounoff, Probleme General de la Stabilite du Mouvement. (AM-17).  Princeton University Press, 12 1948.
  * [14] O. V. I, “A multiplicative ergodic theorem.liapunov characteristic number for dynamical systems,” Trans.Moscow Math.Soc., vol. 19, p. 197–231, 1968\.
  * [15] J. Eckmann, K. S. Oliffson, D. Ruelle, and S. Ciliberto, “Liapunov exponents from time series,” Phys. Rev. A, vol. 34, pp. 4971–4979, 12 1986.
  * [16] M. T. Rosenstein, J. J. Collins, and C. , “A practical method for calculating largest lyapunov exponents from small data sets,” Physica D: Nonlinear Phenomena, vol. 65, pp. 117–134, 1993.
  * [17] C. Schölzel, “nolds: Nonlinear measures for dynamical systems (based on one-dimensional time series).”
  * [18] H. E. Hurst, “The problem of long-term storage in reservoirs,” International Association of Scientific Hydrology. Bulletin, vol. 1, pp. 13–27, 01 1956.
  * [19] S. Driss, M. Soua, R. Kachouri, and M. Akil, A comparison study between MLP and Convolutional Neural Network models for character recognition.  05 2017.
  * [20] 2018 International Interdisciplinary PhD Workshop (IIPhDW), Data augmentation for improving deep learning in image classification problem, 0.
  * [21] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” arXiv:1801.04381 [cs], 03 2019\.