# Content Reduction, Surprisal and Information Density Estimation for Long
Documents

Shaoxiong Ji 1 Wei Sun 2 Pekka Marttinen 3
1 University of Helsinki, Finland 2 KU Leuven, Belgium 3 Aalto University,
Finland
shaoxiong.ji@helsinki.fi;sun.wei<EMAIL_ADDRESS>

###### Abstract

Many computational linguistic methods have been proposed to study the
information content of languages. We consider two interesting research
questions: 1) how is information distributed over long documents, and 2) how
does content reduction, such as token selection and text summarization, affect
the information density in long documents. We present four criteria for
information density estimation for long documents, including surprisal,
entropy, uniform information density, and lexical density. Among those
criteria, the first three adopt the measures from information theory. We
propose an attention-based word selection method for clinical notes and study
machine summarization for multiple-domain documents. Our findings reveal the
systematic difference in information density of long text in various domains.
Empirical results on automated medical coding from long clinical notes show
the effectiveness of the attention-based word selection method.

##  1 Introduction

Long document comprehension is an arduous task in human language
understanding. Information redundancy is becoming prevalent with the
digitalization of individuals’ records and the generation of massive user
content. Natural language encodes information with words and syntax. From the
viewpoint of information theory (Shannon, 1948), language transmits
information over a bandwidth-limited noisy channel. Redundant information in
long documents increases the cognitive load of readers, hinders the processing
of texts, and probably affects the classification performance, especially for
complex examples, in downstream domains. A rational language user tends to use
information-dense phrases (Levy and Jaeger, 2006). The redundancy also
increases the length of sequences, leading to extra computational costs for
neural text encoders. Redundancy is linked to a reduced form of original
content without sacrificing comprehension or cognition. For example, given a
news categorization task, the sentence “The state of medical health records,
and what deep learning can do to help”, we can infer this category is about
health and technology even with some key phrases such as “medical health
records” and “deep learning” who have low word probabilities (high surprisal)
in Figure 1(a).

(a) A text example and its word probability

(b) Examples of clinical notes and their word probability distributions

Figure 1: Illustrations of a) a text snippet and its surprisal modeled by the
pretrained language model, and b) word probability distributions of three
examples of clinical notes

Information redundancy in long documents has been observed as a critical
problem. Taking health text as an example, text redundancy in Electronic
Health Records (EHR) has been widely recognized (Wrenn et al., 2010). We
illustrate the word probabilities of three examples of clinical notes in
Figure 1(b) using a pretrained BERT base model (Devlin et al., 2019), where
informative words and less informative words (tend to be redundant) are
distributed at the two ends of the plots with large densities. Electronic
clinical notes suffer from information redundancy mainly due to copy-and-paste
in clinical notes. Moreover, different expressions exist for the same thing in
the clinical context. A study on 23,630 clinical notes shows that 46% and 36%
of text are copied and imported, respectively (Wang et al., 2017). Massive
redundant information in clinical notes can lead to clinicians’ burnout and
increase medical coder working hours (Montgomery et al., 2019). More
worsening, it can lead to other harms, such as inconsistency in texts and
error propagation during decision-making.

Human reading comprehension can be robust to errors and redundancy (Hahn et
al., 2019), while a robust neural text encoding model that can achieve human-
level comprehension is a challenging research topic. Recent contextualized
language models such as BERT (Bidirectional Encoder Representations from
Transformers) (Devlin et al., 2019) have achieved exciting performance on many
natural language processing tasks. Like human comprehension from prior
experience or education, pretrained models have prior exposure to specific
training corpora. However, most of them are limited to processing the short
sequence due to the quadratic complexity of the self-attention mechanism. For
example, BERT is pretrained with a length of 128. Several efficient
transformers have been proposed to solve the complexity issue to some extent
Tay et al. (2022). Nevertheless, an interesting question is how language
models pretrained with short sequences transfer to long document
representation learning.

This paper studies the problem of long document encoding. We view text
understanding as an abstraction process. Specifically, we mimic the
abstraction via two concrete text processing approaches, i.e., attention-based
word selection and (abstractive) machine summarization. We investigate two
aspects: 1) content reduction to shorten long documents via attention-based
word selection and automated text summarization with pretrained models; 2)
information density estimation for original and content-reduced texts via a
pretrained language model, surprisal model, entropy, uniform information
density, and lexical density. Our contributions are as follows.

  * •

We investigate the systematic difference in information density in different
domains (i.e., clinical texts, movie reviews, and news articles) before and
after content reduction.

  * •

We propose a simple pipeline-based method powered by the label attention
mechanism to select informative words from lengthy clinical notes and perform
automated medical coding.

  * •

Our findings show that content reduction redistributes the information density
of less standard text, such as clinical notes and movie reviews, and the
information density reflects the downstream classification performance. Our
empirical results also show that attention-based word selection can improve
the performance of medical coding from clinical notes.

##  2 Information Density Estimation

Similar to the definition of mass density in physics, information density in
computational linguistics measures the human-readable information encoded per
linguistic unit. One common metric to measure information density is the
lexical density (Section 2.4), which describes the proportion of content words
in a given corpus (Kalinauskaitė, 2018). Psycholinguistic experiments have
shown a link between information density and other issues such as readability
and memory Howcroft and Demberg (2017). Generally, more grammatical words give
less information, and lexical words such as nouns and verbs are more
informative. We investigate the long document embeddings through the lens of
information-theoretic estimation. Several measurements inspired by the
information theory are adopted in this study, including surprisal model
(Section 2.1), entropy (Section 2.2), and uniform information density (Section
2.3).

###  2.1 Surprisal Model

The surprisal model of human language processing describes the surprisal of a
word given its prefix. Intuitively, cognitive efforts involved with text
understanding should be proportional to word surprisal. The lexical-based
surprisal measure (Eq. 1) in psycholinguistic evaluation (Hale, 2001; Levy,
2008) is defined as the negative logarithmic conditional probability given
preceding words of $w_{k+1}$ (or its so-called context). The surprisal is
calculated as:

$S=-\log P\left(w_{k+1}\mid w_{1}\ldots w_{k}\right),$ (1)

where $w_{k}$ is the $k$-th word. The surprisal score values the amount of
surprise. A higher surprisal value means a word difficult to process or
comprehend. An error word should be more surprising than the correct word. For
example, the misspelled word artial and letter-transposed word atrila produce
more surprisal and are more difficult to comprehend than the correct word
atrial. Demberg et al. (2013) summarized two ways to estimate surprisal in
psycholinguistic evaluation, i.e., lexical surprisal and structural surprisal.
Lexical surprisal further considers two levels of word and part-of-speech,
while structural surprisal depends on the syntax of sentence prefixes.

Given a sentence $u=[w_{1},\dots,w_{t},\dots,w_{n}]$ and a pretrained
contextualized language model such as BERT parameterized by
$\boldsymbol{\theta}$, we can calculate the conditional probability of $t$-th
word $w_{t}$ by applying the softmax transform on the $t$-th hidden
representation $\mathbf{h}_{t}$ as

$p_{\boldsymbol{\theta}}\left(w_{t}\mid
w_{<n}\right)=\operatorname{softmax}\left(\mathbf{W}\mathbf{h}_{t}+\mathbf{b}\right),$
(2)

where $\mathbf{W}\in\mathbb{R}^{|\mathcal{S}|\times d_{h}}$,
$\mathbf{b}\in\mathbb{R}^{|\mathcal{S}|}$, and $|\mathcal{S}|$ is the
vocabulary size of the target corpus $\mathcal{S}$. Accordingly, there are two
approaches to computing sentence-level surprisal. We use the n-gram model. For
example, the 3-gram model is defined as

$\displaystyle P(s)$ $\displaystyle=P\left(w_{1}\right)\times P\left(w_{2}\mid
w_{1}\right)\times P\left(w_{3}\mid w_{2}w_{1}\right)\times$
$\displaystyle\prod_{i=4}^{n}P\left(w_{n}\mid w_{n-1}w_{n-2}w_{n-3}\right).$

Noise in text, like typos or errors, can degrade the context of a word,
leading to increased surprise and increasing the difficulty of comprehension
(Hahn et al., 2019). We investigate the surprisal level of texts from
different domains, long documents, particularly to understand the behavior of
neural text encoders.

###  2.2 Entropy

Entropy estimate has been studied in many ways. Genzel and Charniak (2002)
conducted a $n$-gram entropy estimate in three different ways, i.e., a
$n$-gram probabilistic model, a probabilistic model induced by a statistical
parser, and a non-parametric estimator. The authors proposed the constancy
rate principle governing language generation. However, their local entropy
estimate ignored the context. Bentz and Alikaniotis (2016) used entropy to
measure the average information content of natural languages and conducted a
quantitive analysis to investigate the systematic difference in word entropies
across different languages. Inspired by these two works, we estimate entropy
by utilizing pretrained contextualized language models to consider the context
information and study the systematic difference in word entropies for long
documents and their summaries across different domains. The entropy of text
$s$ is defined as

$H(s)=-\sum_{i=1}^{n}P\left(w_{i}\right)\log\left(P\left(w_{i}\right)\right),$
(3)

where $P(w_{i})$ is the probability of word $w_{i}$. $P(w_{i})$ can be
approximated as $P(w_{i})=\frac{f_{i}}{\sum_{j=1}^{n}f_{j}}$ from the
frequency viewpoint and $f_{i}=\operatorname{freq}(w_{i})$ is the frequency of
word $w_{i}$. Our study approximates it as the conditional probability
generated by a pretrained language model given its context.

###  2.3 Uniform Information Density

The uniform information density (UID) hypothesis asserts that information
encoding aims to transmit messages in a uniform way during the language
production Jaeger (2006, 2010). The intuition behind UID is to maximize the
information transmission and minimize comprehension difficulty. The UID
hypothesis aligns with the principle of language production, i.e., to avoid
information overloading or being uninformative. The context plays an important
role in the information density of sentences. If the context is considered,
the information density of sentences is uniform; otherwise, it experiences an
increase with the sentence number in local measures of entropy Genzel and
Charniak (2002). Meister et al. (2021) quantifies the linguistic uniformity by
defining the UID as

$\mathrm{UID}^{-1}({u})=\frac{1}{n}\sum_{i=1}^{n}\Delta\left(S\left(u_{i}\right),\mu_{c}\right)$
(4)

where $\mu_{c}$ is an average information rate and $\Delta(\dot{,}~{}\dot{)}$
is a per-unit distance metric.

From this viewpoint, UID can be regarded as a measure on how uniform the
sentence conveys its meaning. We investigate if the embeddings of long
documents from pretrained language models adhere to the uniform information
density hypothesis.

###  2.4 Lexical Density

We first use lexical readability to examine how difficult a document is to
understand. We apply the Flesch reading ease score that was introduced for
reading ease evaluation (Kincaid et al., 1975). It is formulated as:

$206.835-1.015\left(\frac{\text{ total words }}{\text{ total sentences
}}\right)-84.6\left(\frac{\text{ total syllables }}{\text{ total words
}}\right),$

where the coefficients come from user study. A higher score means easier to
read. A score of 100 indicates the text is effortless to read, while a score
ranging from 0 to 10 means the text is complicated to comprehend and needs
professional knowledge. We transfer the readability test to some specific
domains and provide a reference for lexical density estimation.

We study lexical richness, which basically measures to what extent different
words are used in the text. Many lexical richness measures calculate the
proportion of unique words to evaluate the lexical diversity (Wimmer and
Altmann, 1999). The widely used type-token ratio is calculated by the number
of types divided by the number of tokens. We use one of its variants called
the Herdan lexical richness measure proposed by Herdan (1960). Herdan lexical
richness is defined as:

$C=\frac{\log V(N)}{\log N},$

where $N$ is the number of tokens and $V$ is the number of types.

##  3 Content Reduction

We cast long document understanding as a generation process that comprehends
the long documents and digests key messages as latent states. As a result, the
understanding process generates some short versions of the original text but
preserves the subject matter of original long documents. Specifically, we
instantiate the generation process by two concrete instances, i.e., attention-
based word selection and abstractive text summarization. Attention-based word
selection in the previous section is similar to extractive text summarization.
However, it is not trained with a reference dataset with extraction-based
summaries.

###  3.1 Attention-based Word Selection

We propose a simple and efficient pipeline-based word selection method powered
by the label attention mechanism that prioritizes essential information in the
hidden representation relevant to medical codes. The label attention uses dot
product to calculate the attention score matrix
$\mathbf{A}\in\mathbb{R}^{n\times m}$ as:

$\mathbf{A}=\operatorname{Softmax}(\mathbf{H}\mathbf{U}),$ (5)

where $\mathbf{H}\in\mathbb{R}^{n\times h}$ is the hidden features,
$\mathbf{U}\in\mathbb{R}^{{h}\times m}$ is the parameter matrix of the query,
and $m$ is the number of medical codes. We use mean pooling to obtain the
attention vector $\mathbf{a}\in\mathbb{R}^{n}$ for word selection, i.e.,
$\mathbf{a}=\text{MeanPooling}(\mathbf{A})$. Given a threshold or $q$-th
quantile of the pooled attention score, we selected words whose attention
scores meet the selection criteria, and other words in a text are filtered
out. This pipeline can be extended to various text feature extractors that
capture sequential dependency and utilize label-aware representations from the
label attention mechanism.

###  3.2 Text Summarization

Automated text summarization transforms lengthy documents into shortened
paragraphs while preserving the overall meaning. Abstractive summarization
summarizes the text differently rather than extracting some key sentences from
the document.

We utilize two advanced abstractive summarization models, i.e., pretrained
BART (Lewis et al., 2020) that is trained by learning to predict the
arbitrarily corrupted text and T5 (Raffel et al., 2020) based on a text-to-
text framework. These two representative models have shown superior
performance on several text summarization benchmarks. However, there exists
one limitation to this study. As the reference dataset with human
summarization is not available, we can not tell which machine summarization
model is the best.

##  4 Results and Analyses

###  4.1 Tasks and Datasets

We conduct experiments on a long document classification task with three
public datasets from different domains. A statistical summary of datasets is
shown in Table 1.

Dataset | Avg. Length | Train | Validation | Test
---|---|---|---|---
BBC News | 419 | 1,424 | 356 | 445
IMDB | 698 | 1,553 | 350 | 1,791
MIMIC-III | 1,883 | 8,066 | 1,573 | 1,729
Table 1: A statistical summary of datasets

#### Medical Coding

Medical coding is a multi-label multi-class classification task that takes
clinical notes from electronic health records as inputs and predicts medical
codes of standard disease classification systems (Ji et al., 2022). We use
clinical notes from the MIMIC-III dataset (Johnson et al., 2016) and adopt the
data split of top-50 codes from Mullenbach et al. (2018) that assigns
frequently used ICD-9 codes to discharge summaries. We use the BERT text
encoder as the neural backbone to get text representations and learn label-
aware features with a label attention mechanism to boost the performance of
medical coding.

#### News Topic Classification

The BBC News dataset (Greene and Cunningham, 2006) contains news articles in
BBC News from 2004-2005. It is used for news topic classification in five
topical areas, i.e., business, entertainment, politics, sport, and technology.
Notice that we use our data split as there is no standard data partition.

#### Movie Review Sentiment Analysis

The IMDB movie review dataset (Maas et al., 2011) has movie reviews posted on
the IMDB website. As the average length of this dataset is relatively short,
we select long reviews from the training and testing sets of the original
data. Then, we split an additional validation set for the IMDB long review
data.

###  4.2 Results of Attention-based Selection for Medical Coding

We present the results of the attention-based word selection method for
medical coding. Specifically, we use this method to obtain selected texts
whose sequence length is shorter than the original text. For the text encoder
and code classifier, we adopt a recent medical coding model that utilizes
recalibrated feature aggregation and multitask learning with focal loss (Sun
et al., 2023). After the word selection, we input the shortened text into a
BERT-based medical coding model. We compare this pipeline-based model with the
following models. The first category is the convolutional or recurrent neural
network-based models. They are CAML (Mullenbach et al., 2018) that uses a text
convolutional neural network and label attention mechanism, GatedCNN-NCI (Ji
et al., 2021b) that adopts gated convolutions and a note-code interaction
module, and JointLAAT (Vu et al., 2021) that utilizes bidirectional long
short-term memory networks and a structured attention mechanism. The second
category is based on BERT-based classifiers. We compare the truncated and
hierarchical BERT (Ji et al., 2021a) with three domain adaptive BERT models,
i.e., PubMedBERT (Gu et al., 2020), BioBERT (Lee et al., 2020) and
ClinicalBERT (Alsentzer et al., 2019). The third category is enhanced BERT-
based models. MDBERT (Zhang and Jankowski, 2022) considers three-level
hierarchical encoding. Two variants of MDBERT are also compared. MDBERT-SBERT
removes sentence BERT, and MDBERT+avg uses model ensemble. Our method achieves
better performance than simple BERT-based classifiers and comparable
performance than simple MDBERT without sentence BERT, although slightly worse
than the ensemble-based method (MDBERT+avg). Different word selection
strategies also affect the performance of our method. Selection with the
$q$-th quantile ($q=0.875$) is more flexible in selecting informative words
and achieves better performance than the variant that only selects words with
a fixed threshold for all documents. We choose the threshold that can obtain a
content-reduced text with an average length of 250.

Model | AUC-ROC | F1 | P@5
---|---|---|---
Macro | Micro | Macro | Micro
CAML | 87.5 | 90.9 | 53.2 | 61.4 | 60.9
GatedCNN-NCI | 91.5 | 93.8 | 62.9 | 68.6 | 65.3
JointLAAT | 92.5 | 94.6 | 66.1 | 71.6 | 67.1
PubMedBERT | 82.1 | 84.4 | 52.6 | 57.3 | 55.7
BioBERT | 81.8 | 84.3 | 50.5 | 55.4 | 54.5
ClinicalBERT | 82.3 | 85.3 | 50.6 | 56.9 | 55.7
MDBERT-SBERT | 91.1 | 93.1 | 64.4 | 68.1 | 64.3
MDBERT | 91.8 | 93.6 | 65.9 | 69.2 | 65.4
MDBERT+avg | 92.8 | 94.6 | 67.2 | 71.7 | 67.4
Ours (fixed) | 90.6 | 92.9 | 58.2 | 65.3 | 64.0
Ours ($q$ quantile) | 91.6 | 93.5 | 64.6 | 68.5 | 64.6
Table 2: Attention word selection for medical coding on MIMIC-III dataset.
“+avg” means averaging-based model ensemble. “fixed” and “$q$ quantile” denote
that we select words by a fixed threshold and $q$ quantile.

We compare the model performance by applying a scaling factor to the
embeddings of selected words to verify the effect of attention-based word
selection further. Specifically, the word embeddings multiplied by the scaling
factor is penalized as a restricted input signal to the model. Intuitively, we
use the scaled word embeddings to control the strength of selected words. A
higher scaling factor means that the embeddings of selected words weigh more
in the input text. Results in Figure 2 show that selected words with rich
information are contributing more to representing the content, and the
predictive performance is getting better with the increase of the scaling
factor. Then, we apply the scaling factor to both selected words and those not
selected. Table 3 shows the performance drops after applying a scaling factor
of 0.1. The results reveal that penalizing the signal strength of selected
words with the scaling factor leads to a more significant performance drop
than downweighting those not selected words. These two studies indicate that
attention-based word selection can extract important words to the model
prediction.

Figure 2: Predictive performance on MIMIC-III dataset with different scaling factors applied on the embeddings of selected words Scaling | AUC-ROC | F1 | P@5
---|---|---|---
Macro | Micro | Macro | Micro
Non-selected Words | -2.9 | -2.1 | -10.6 | -7.5 | -3.3
Selected Words | -6.0 | -5.0 | -19.9 | -16.6 | -11.7
Table 3: Performance drop on MIMIC-III dataset when applying a scaling factor
of 0.1 on selected and non-selected words

###  4.3 Results of Classification on Abstractive Summary

Text summarization can be a way to alleviate the reader’s workload by
automatically extracting critical information from the text. We test the
performance of text classification on the abstractive summary. Experimental
results show it is hard to achieve better performance on summaries than the
original texts. In several cases, the performance drop can even be 10%. We do
not report those negative results in detail to avoid verbosity.

(a) MIMIC-III

(b) IMDB

(c) BBC News

Figure 3: Kernel density estimation plots for document-level mean surprisal

###  4.4 Analysis of Surprisal, Entropy, and Uniform Information Density

#### Surprisal

We draw the kernel density estimation plot for document-level mean surprisal
in Figure 3. The figure also shows histograms normalized to the same scale as
the density curves. The smooth density estimation curve is generated by
summing the Gaussians of individual points of word probabilities. The figures
show that long original documents tend to have a higher mean surprisal than
attention-selected texts and summaries. T5 summarization model significantly
reduces the surprisal of MIMIC-III clinical notes and IMDB movie reviews.

These results also align with the performance of downstream document
classification tasks. With close surprisal distributions (Figure 3(c)), the
downstream classification performance on the original text and summaries of
BBC News is very close. In contrast, the performance on IMDB summary drops up
to 10% when compared with the performance on original texts, which can partly
be explained by the redistributed surprisal as shown in Figure 3(b).

#### Entropy

Entropy describes the amount of information required to represent an event
randomly drawn from the distribution. We estimate the document-level entropy
to understand the information required to represent the text encoded by a
language model. Figure 4 shows the kernel density estimation plots for
document-level entropy of three datasets. We can clearly see that the original
texts contain more information than summaries of all three datasets and
content-reduced text via attention selection in the MIMIC-III dataset.

(a) MIMIC-III

(b) IMDB

(c) BBC News

Figure 4: Kernel density estimation plots for document-level entropy

#### Uniform Information Density

(a) MIMIC-III

(b) IMDB

(c) BBC News

(d) Corpus-level mean

Figure 5: Uniform information density. (a-c) document-level kernel density
estimation plots and (d) mean UID in corpus level

Uniform information density describes how uniformly the information is
distributed through the communication channel. Figure 5 shows the kernel
density estimation of UID of three datasets. In MIMIC-III, the UID
distributions of the original text and attention-selected text overlap (Figure
5(a)), and the corpus mean UID of attention-selected text is slightly bigger
than the original text. Thus, we can conclude that attention selection can
extract informative parts from the original text and maintain the level of
uniformity. Figure 5(b) shows the summarization model generated IMDB review
text with information density more uniformly distributed, while Figure 5(c)
shows the overlapped UID of BBC News. Figure 5(d) of corpus-level UID further
verifies summarization models generate texts with more uniform information
density.

###  4.5 Analysis of Lexical Density

We conduct a quantitative analysis of the lexical structure of the original
texts and content-reduced texts and summaries. Many metrics have been used to
evaluate the complexity of language. We choose two from those widely used
metrics. They are lexical readability and lexical richness which measure how
readable and diverse the text is. We conduct this analysis to examine the
change in lexical complexity before and after the content reduction via
attention selection and machine summarization.

We illustrate the lexical readability of each instance in the MIMIC-III
dataset (Figure 6(a)) and the corpus mean of three datasets (Figure 6(c))
measured by the Flesch reading ease score. We use the instance indices of
sorted scores of the original dataset to plot each instance. Figure 6(a) shows
that MIMIC-III clinical notes are extremely hard to comprehend with negative
readability scores. With content reduction via attention-based selection and
summarization, the readability scores of MIMIC-III clinical notes improve
significantly but are still very hard to read. As for text in the domains of
newswire and movie review, abstractive summarization generates texts that are
slightly easier or the same as easy to read. Movie reviews contain several
documents hard to read. Formal language in the news is more readable than
clinical notes and user-generated movie reviews, which aligns with the finding
of the Flesch test. The Flesch test tends to give a higher score for text with
easy words.

Figure 6(b) and Figure 6(d) illustrate the Herdan lexical richness of each
instance in the MIMIC-III dataset and all three datasets at the corpus level.
We can see that summarization with the BART model increases lexical richness
by a considerable margin in IMDB and BBC News. Furthermore, attention-based
word selection improves the richness of MIMIC-III clinical notes (Figure
6(b)). T5 tends to generate less affluent summaries.

Content reduction increases the lexical density to some extent, especially for
less standard text such as clinical notes. However, the behavior of different
summarization models varies. Considering the performance boost brought by the
attention-based word selection, we summarize that a simple pipeline method
that condenses the lexical density of noisy texts (e.g., clinical notes)
benefits the downstream classification task to some extent. More investigation
is needed for the text summarization method via transfer learning.

(a) Readability of MIMIC

(b) Richness of MIMIC

(c) Mean readability

(d) Mean richness

Figure 6: Lexical readability measured by Flesch reading ease score and Herdan
lexical richness of MIMIC-III at instance level and all three datasets at
corpus level

###  4.6 Discussion, Limitations and Future Work

This study sheds light on the investigation of an attention-based word
selection method for clinical notes and information density estimation on
various summarization texts. Extractive summarization by attention selection
has shown good downstream performance on the medical coding task. However,
some critical issues remain unexplored; for example, what if attention-based
word selection filters out negations and breaks the syntactic structure?
Besides, statistical significance cannot be thoroughly evaluated given the
limited number of training instances and domain data. We leave these
unexplained problems for future work. We tried some amendments for some
existing limitations. There are no ground-truth values of surprisal. An
alternative we used is to approximate it via pretrained language models. Also,
we cannot directly evaluate the quality of word selection and summarization
due to the lack of a reference dataset. Instead, we evaluate it through
downstream classification tasks.

##  5 Related Work

Processing long documents with redundant information is burdensome. Many
efforts have been made to estimate the redundancy in clinical notes and study
the potential risks of redundancy in a retrospective manner. Wrenn et al.
(2010) quantified the redundancy in the clinical document by measuring the
amount of new information and showed information duplication between document
types. Zhang et al. (2011) studied several methods for measuring the
redundancy in clinical texts. In the clinical domain, vocabulary and errors
are relatively rare compared with generic texts. Searle et al. (2021) showed
that clinical text is less efficient in encoding information than open-domain
text from the perspective of information theory and observed that some
clinical notes in the MIMIC database could be 97-98% redundant.

Levy and Jaeger (2006) investigated the possibility of uniformity maximization
of information density through syntactic reduction. (Meister et al., 2021)
revisited the uniform information density hypothesis and interpreted the
hypothesis as the regression to the mean information of a language.
Information density estimation is an important research task. Several works
have been done to automatically measure text information density, such as from
the perspective of lexical and syntactic features (Kalinauskaitė, 2018). Horn
et al. (2013) used open information extraction system to extract facts and
applied factual density, calculated as the number of facts divided by the
document size, to measure the informativeness of web documents.

##  6 Conclusion

Long document processing is challenging for many reasons, such as the
difficulty of capturing long-term dependency and noises in the long text. This
paper studies the encoding of long documents via information density
estimation and empirical analyses on content reduction. We systematically show
the difference in information density between original long documents and
content-reduced texts. We improve the performance of automated medical coding
by using selected words as inputs when compared with simple baselines that use
the same neural backbone. We validate that careful word selection can obtain
words that can redistribute the distribution of word probability and entropy.
Our study takes a positive step towards understanding language model-based
long document encoding.

## Limitations

As an empirical study, this paper did not outperform the state-of-the-art
method, such as the ensemble-based hierarchical model (Zhang and Jankowski,
2022). Our analyses focus on standard self-attention-based transformer
networks and masked language models. Recent efficient transformers and
pretrained models with other language modeling objectives are not considered
in this study. We leave them in the future work.

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly Available Clinical BERT Embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78.
  * Bentz and Alikaniotis (2016) Christian Bentz and Dimitrios Alikaniotis. 2016.  The word entropy of natural languages.  _arXiv preprint arXiv:1606.06996_.
  * Demberg et al. (2013) Vera Demberg, Frank Keller, and Alexander Koller. 2013.  Incremental, predictive parsing with psycholinguistically motivated tree-adjoining grammar.  _Computational Linguistics_ , 39(4):1025–1066.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT_.
  * Genzel and Charniak (2002) Dmitriy Genzel and Eugene Charniak. 2002.  Entropy rate constancy in text.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 199–206.
  * Greene and Cunningham (2006) Derek Greene and Pádraig Cunningham. 2006.  Practical solutions to the problem of diagonal dominance in kernel document clustering.  In _Proceedings of 23rd International Conference on Machine learning_ , pages 377–384.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.  _arXiv preprint arXiv:2007.15779_.
  * Hahn et al. (2019) Michael Hahn, Frank Keller, Yonatan Bisk, and Yonatan Belinkov. 2019.  Character-based surprisal as a model of reading difficulty in the presence of errors.  In _41st Annual Meeting of the Cognitive Science Society_ , pages 401–407. Cognitive Science Society.
  * Hale (2001) John Hale. 2001.  A probabilistic earley parser as a psycholinguistic model.  In _NAACL_.
  * Herdan (1960) Gustav Herdan. 1960.  _Type-token mathematics_ , volume 4.  Mouton.
  * Horn et al. (2013) Christopher Horn, Alisa Zhila, Alexander Gelbukh, Roman Kern, and Elisabeth Lex. 2013.  Using factual density to measure informativeness of web documents.  In _Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013)_ , pages 227–238.
  * Howcroft and Demberg (2017) David M Howcroft and Vera Demberg. 2017.  Psycholinguistic models of sentence processing improve sentence readability ranking.  In _Proceedings of EACL_ , pages 958–968.
  * Jaeger (2010) T Florian Jaeger. 2010.  Redundancy and reduction: Speakers manage syntactic information density.  _Cognitive psychology_ , 61(1):23–62.
  * Jaeger (2006) Tim Florian Jaeger. 2006.  _Redundancy and syntactic reduction in spontaneous speech_.  Ph.D. thesis, PhD dissertation, Stanford University.
  * Ji et al. (2021a) Shaoxiong Ji, Matti Hölttä, and Pekka Marttinen. 2021a.  Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study.  _Computers in Biology and Medicine_.
  * Ji et al. (2021b) Shaoxiong Ji, Shirui Pan, and Pekka Marttinen. 2021b.  Medical code assignment with gated convolution and note-code interaction.  In _Findings of ACL-IJCNLP_.
  * Ji et al. (2022) Shaoxiong Ji, Wei Sun, Hang Dong, Honghan Wu, and Pekka Marttinen. 2022.  A unified review of deep learning for automated medical coding.  _arXiv preprint arXiv:2201.02797_.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  MIMIC-III, a Freely Accessible Critical Care Database.  _Scientific Data_ , 3:160035.
  * Kalinauskaitė (2018) Danguolė Kalinauskaitė. 2018.  Detecting information-dense texts: towards an automated analysis.  In _CEUR Workshop Proceedings_.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975\.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.  Technical report, Naval Technical Training Command Millington TN Research Branch.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Levy (2008) Roger Levy. 2008.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126–1177.
  * Levy and Jaeger (2006) Roger Levy and T. Florian Jaeger. 2006.  Speakers optimize information density through syntactic reduction.  _Advances in neural information processing systems_ , 19.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Meister et al. (2021) Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021.  Revisiting the uniform information density hypothesis.  In _Proceedings of EMNLP_ , pages 963–980.
  * Montgomery et al. (2019) A Montgomery, E Panagopoulou, A Esmail, T Richards, and C Maslach. 2019.  Burnout in healthcare: the case for organisational change.  _Bmj_ , 366.
  * Mullenbach et al. (2018) James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018\.  Explainable Prediction of Medical Codes from Clinical Text.  In _Proceedings of NAACL-HLT_ , pages 1101–1111.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Searle et al. (2021) Thomas Searle, Zina Ibrahim, James Teo, and Richard Dobson. 2021.  Estimating redundancy in clinical text.  _Journal of biomedical informatics_ , 124:103938.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423.
  * Sun et al. (2023) Wei Sun, Shaoxiong Ji, Erik Cambria, and Pekka Marttinen. 2023.  Multitask balanced and recalibrated network for medical code prediction.  _ACM Transactions on Intelligent Systems and Technology_ , 14(1):1–20.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022.  Efficient transformers: A survey.  _ACM Computing Surveys_.
  * Vu et al. (2021) Thanh Vu, Dat Quoc Nguyen, and Anthony Nguyen. 2021.  A label attention model for icd coding from clinical text.  In _Proceedings of IJCAI_.
  * Wang et al. (2017) Michael D Wang, Raman Khanna, and Nader Najafi. 2017.  Characterizing the source of text in electronic health record progress notes.  _JAMA Internal Medicine_ , 177(8):1212–1213.
  * Wimmer and Altmann (1999) Gejza Wimmer and Gabriel Altmann. 1999.  On vocabulary richness.  _Journal of Quantitative Linguistics_ , 6(1):1–9.
  * Wrenn et al. (2010) Jesse O Wrenn, Daniel M Stein, Suzanne Bakken, and Peter D Stetson. 2010.  Quantifying clinical narrative redundancy in an electronic health record.  _Journal of the American Medical Informatics Association_ , 17(1):49–53.
  * Zhang and Jankowski (2022) Ning Zhang and Maciej Jankowski. 2022.  Hierarchical bert for medical document understanding.  _arXiv preprint arXiv:2204.09600_.
  * Zhang et al. (2011) Rui Zhang, Serguei Pakhomov, Bridget T McInnes, and Genevieve B Melton. 2011.  Evaluating measures of redundancy in clinical texts.  In _AMIA annual symposium proceedings_ , volume 2011, page 1612. American Medical Informatics Association.