# Finding the Needle in a Haystack:
Unsupervised Rationale Extraction from Long Text Classifiers

Kamil Bujel  Department of Computing, Imperial College London, U.K.
Department of Computer Science and Technology, University of Cambridge, U.K.
Andrew Caines  Department of Computer Science and Technology, University of
Cambridge, U.K.  The ALTA Institute, University of Cambridge, U.K.  Helen
Yannakoudakis  Department of Informatics, King’s College London, U.K.  The
ALTA Institute, University of Cambridge, U.K.  Marek Rei  Department of
Computing, Imperial College London, U.K.  The ALTA Institute, University of
Cambridge, U.K.

###### Abstract

Long-sequence transformers are designed to improve the representation of
longer texts by language models and their performance on downstream document-
level tasks. However, not much is understood about the quality of token-level
predictions in long-form models. We investigate the performance of such
architectures in the context of document classification with unsupervised
rationale extraction. We find standard soft attention methods to perform
significantly worse when combined with the Longformer language model. We
propose a compositional soft attention architecture that applies RoBERTa
sentence-wise to extract plausible rationales at the token-level. We find this
method to significantly outperform Longformer-driven baselines on sentiment
classification datasets, while also exhibiting significantly lower runtimes.

##  1 Introduction

Transformer-based architectures (Vaswani et al., 2017) have become ubiquitous
in natural language processing research. A key attribute to their success is
the multi-head self-attention mechanism (Michel et al., 2019). However, its
computational and memory requirements grow quadratically with input sequence
length. Therefore, models such as BERT (Devlin et al., 2019) commonly limit
the maximum sequence length to $512$ tokens. Longer documents are truncated
Devlin et al. (2019) or staggered position embeddings are used Jain et al.
(2020). This limitation motivated long-text transfomers, such as Big Bird
(Zaheer et al., 2020) and Longformer (Beltagy et al., 2020), which reduce the
complexity of self-attention through the use of sparse attention and improve
the performance of transformers on long documents.

The task of rationale extraction focuses on selecting a subset of input as a
justification for the model’s output. In particular, our work focuses on
extracting token-level rationales from long documents. The models presented
are supervised using document labels only, with no token-level annotations
used during training. This setting is more practical for longer texts, where
token-level annotations are often missing due to prohibitive costs of manual
labeling.

Rationale extraction for the vanilla transformer-based sentence and document
classifiers has already been examined (Pruthi et al., 2020; Jain et al., 2020;
Bujel et al., 2021). However, to the best of our knowledge, there is no work
that evaluates unsupervised rationales extracted from long-text transformers.
As we show in this work, methods that work best for standard transformers do
not necessarily perform as well on longer documents. We investigate various
methods to adapt long-text transformers to zero-shot rationale extraction. We
highlight that this work explicitly focuses on extracting plausible rationales
(agreeable to human annotators; DeYoung et al. (2020)), as opposed to faithful
explanations (true to the system’s computation; Rudin (2018)).

While self-attention layers have been shown to learn correct grammatical
relations (Clark et al., 2019) and rationales (Pruthi et al., 2020; Jain et
al., 2020), we find that the sparse self-attention present in Longformer
struggles to locate tokens that can serve as a plausible rationale for
document labels. For longer sentiment detection datasets, the quality of the
rationale extracted from the self-attention layers is not better than the
random baseline. We further show that a direct combination of Longformer with
a Weighted Soft Attention rationale extractor Bujel et al. (2021) does not
perform well either, as only a small proportion of tokens receive supervision
signal at each epoch. We propose Ranked Soft Attention, which ensures that the
weights of all tokens are updated at each epoch. We introduce a Compositional
Soft Attention architecture, which applies a vanilla RoBERTa model sentence-
wise (Liu et al., 2019) and composes the obtained token contextual embeddings
using Ranked Soft Attention. We find this model to outperform other
unsupervised approaches for rationale extraction on the longer sentiment
detection documents, while being significantly faster than Longformer-based
systems.

##  2 Soft Attention

Rei and Søgaard (2018) introduced a soft attention architecture for biLSTM
zero-shot sequence labelers, which Bujel et al. (2021) adapted to transformers
by introducing Weighted Soft Attention. We apply Weighted Soft Attention to
contextual embeddings $t_{i}\in\mathbb{R}^{h}$ for all tokens in the document:

$\displaystyle e_{i}=\tanh(W_{e}t_{i}+b_{e})\,\,\,\,\,\,\,\,\,$ (1)
$\displaystyle\widetilde{a_{i}}=\sigma(\widetilde{e_{i}})\,\,\,\,\,\,\,\,\,\,\widetilde{e_{i}}=W_{\tilde{e}}e_{i}+b_{\tilde{e}}$
(2)

where $h\in\mathbb{N}$ is the dimension of the contextual token embeddings,
$e_{i}\in\mathbb{R}^{h^{\prime}}$ is a vector,
$\widetilde{e_{i}}\in\mathbb{R}$ is a single scalar value and
$\widetilde{a_{i}}\in[0,1]$ is the token attention score. The scores are
converted to normalised attention weights $a_{i}$ to build document-level
representation $c$:

$a_{i}=\frac{\widetilde{a_{i}}^{\beta}}{\sum_{j=1}^{N}\widetilde{a_{j}}^{\beta}}\,\,\,\,\,\,\,\,\,\,c=\sum_{i=1}^{N}a_{i}t_{i}$
(3) $d=\tanh(W_{d}c+b_{d})\,\,\,\,\,\,\,\,\,\,y=\sigma(W_{y}d+b_{y})$ (4)

where $N\in\mathbb{N}$ is the number of tokens in a sentence,
$c\in\mathbb{R}^{h}$ is the combined document representation,
$d\in\mathbb{R}^{s}$ is the hidden document representation and
$y\in\mathbb{R}^{2}$ is the document prediction. $\beta\in\mathbb{R}$ is a
weight controlling the sharpness of the attention scores. For each document
$j$, we obtain document-level predictions $y^{(j)}\in[0,1]$ and the token-
level scores $0\leq\widetilde{a_{i}}\leq 1$. As the token-level scores are
preceded by a logistic activation function, we use a classification threshold
of $0.50$. The supervision of Weighted Soft Attention is described in Appendix
A.

###  2.1 Ranked Soft Attention

Empirically, we find that most of the token scores obtained using Weighted
Soft Attention are close to $1$ for long documents. We suspect this is caused
by an insufficient number of token scores receiving a supervision signal at
each epoch. Only minimum and maximum token scores are optimized. While working
well for individual sentences, such an approach does not scale well to longer
documents. We confirmed this hypothesis by measuring that on average, only 5%
of tokens receive supervision signal during training.

We propose Ranked Soft Attention, where $k\%$ of tokens with the highest
scores are supervised with the document label, while the remaining $100-k\%$
tokens are supervised with a $0$. In particular, we define two subsets,
$\widetilde{a}_{top}$ and $\widetilde{a}_{rest}$ which contain the top $k\%$
and the remaining $1-k\%$ of token attention scores $\widetilde{a_{i}}$
respectively, so that
$\widetilde{a}=\widetilde{a}_{top}+\widetilde{a}_{rest}$. We define the loss
function $L_{ranked}$ as follows:

$L_{ranked}=\sum_{j}\Bigg{(}\frac{1}{|\widetilde{a}_{top}|}\sum_{\widetilde{a_{i}}\in\widetilde{a}_{top}}(\widetilde{a_{i}}-\widetilde{y}^{(j)})^{2}\\\
\\\
+\frac{1}{|\widetilde{a}_{rest}|}\sum_{\widetilde{a_{i}}\in\widetilde{a}_{rest}}(\widetilde{a_{i}}-0)^{2}\Bigg{)}$
(5)

where $k$ is a hyperparameter that can be inferred based on the percentage of
annotations in the dataset (Jain et al., 2020) and total loss is
$L=L_{1}+\gamma(L_{2}+L_{3})+\gamma_{ranked}L_{ranked}$.

We hope this universal supervision approach alleviates the problems of
insufficient token-level supervision during training.

| FCE | BEA 2019
---|---|---
| Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | MAP | Time | Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | MAP | Time
Random Uniform | - | $12.13\pm 0.36$ | 13.70 | 15.91 | - | - | $11.36\pm 0.32$ | 12.00 | 16.68 | -
Longformer Weighted Soft Attention | 89.34 | $\textbf{24.81}\pm 1.42$ | 17.64 | 16.17 | 242 | 93.79 | $\textbf{21.22}\pm 0.17$ | 14.59 | 16.38 | 357
Longformer Self-Attention Top-K | 89.29 | $14.23\pm 2.25$ | 15.57 | 14.67 | 243 | 94.39 | $14.75\pm 3.21$ | 14.99 | 13.76 | 360
Longformer Ranked Soft Attention | 89.10 | $22.23\pm 2.44$ | 23.40 | 18.82 | 238 | 93.49 | $19.11\pm 2.92$ | 19.65 | 18.55 | 351
Compositional Soft Attention | 81.32 | $21.08\pm 2.02$ | 23.19 | 19.67 | 139 | 89.07 | $20.44\pm 1.73$ | 20.22 | 19.44 | 141
Table 1: Results for the Grammatical Error Detection Datasets. Doc $F_{1}$
represents document-level classification performance, while $F_{1}$, $F_{0.5}$
and MAP represent token-level metrics. Time represents an average seconds per
epoch the given model took to train. We see that both our Ranked Soft
Attention and Compositional Soft Attention perform significantly better on the
token-level than Weighted Soft Attention or Self-Attention baselines. We note
that Weighted Soft Attention exhibits high $F_{1}$ due to assigning $1$ to
most tokens and thus exhibiting a large recall. Compositional Soft Attention
is significantly faster than Longformer-based models.

###  2.2 Compositional Soft Attention

Given RoBERTa’s strong performance as a rationale extractor for individual
sentences Bujel et al. (2021); Pruthi et al. (2020), we decide to investigate
the feasibility of using RoBERTa together with Ranked Soft Attention to
extract token-level rationales from longer documents. In particular, we
propose to model intra-sentence token dependencies by applying RoBERTa to each
sentence individually. To extract rationale for the whole document, we use a
ranked soft attention layer that composes the individual token contextual
embeddings across different sentences. This is different from Hierarchical
Transformers (Pappagari et al., 2019), which focus on document-level
representation only.

More concretely, we propose a Compositional Soft Attention architecture that
uses a standard length transformer to build contextual token embeddings
$t_{k}^{\prime}\in\mathbb{R}^{N_{k}\times d}$ separately for each sentence
$s_{k}$, where $k\in\mathbb{N}$ and $k<m$:

$t_{k}^{\prime}=\text{Transfomer}(s_{k})$ (6)
$t=\text{Concat}[t_{0}^{\prime},...,t_{m}^{\prime}]$ (7)

where $t\in\mathbb{R}^{N\times d}$. We then provide this document
representation $t$ as input to a soft attention layer (Eq.1), which composes
tokens across all sentences to obtain a document-level representation and the
prediction $y^{(j)}$. We additionally obtain token-level attention scores
$\widetilde{a_{i}}$, which we use to extract rationales for document
classification tasks. An architecture visualization and pseudocode are given
in Appendix 1.

##  3 Datasets

We investigate the performance of our models on three different datasets. Each
of the datasets contains a document-level label, together with human
annotations on the token-level that are used for evaluation only.

We evaluate our models on Grammatical Error Detection (GED) datasets, which
contain texts written by learners of the English language. They are annotated
with token-level grammatical errors, which serve as rationale for document-
level proficiency scores. An annotated set of essays from the Write & Improve
feedback and assessment platform Yannakoudakis et al. (2018) was released as
part of the BEA 2019111https://www.cl.cam.ac.uk/research/nl/bea2019st/ shared
task Bryant et al. (2019). The essays were submitted online in response to
various prompts, and document-level labels indicate the CEFR proficiency
levels (A/B/C) of the English learners. We remove intermediate (B) essays and
treat the beginner (A) class as a positive document label, as opposed to the
advanced (C) class label. Since there is no publicly available test dataset,
we held out the development dataset for evaluation purposes and randomly
sampled $10\%$ of the training dataset for development.

The First Certificate in English222https://ilexir.co.uk/datasets/index.html
(FCE) dataset Yannakoudakis et al. (2011) contains essays written by non-
native English learners for an intermediate-level language proficiency exam.
Each student wrote $2$ essays, which we combine to increase the overall
document length. We split the dataset into beginner (score $<27$, equivalent
to a fail) and more advanced learners (score $>30$). We use the train/dev/test
split released by Rei and Yannakoudakis (2016). We note that both of the GED
datasets contain a relatively small proportion of documents exceeding $512$
tokens, the standard RoBERTa maximum sequence length (Table 3).

We also use the Sentiment Detection movie reviews
IMDB333https://www.tensorflow.org/datasets/catalog/movie_rationales dataset
Zaidan et al. (2007), which contains positive and negative movie reviews. We
focus on a subset of this dataset that has been annotated with rationales for
the reviews by human annotators. We split the dataset into IMDB-Pos and IMDB-
Neg, where the former contains evidence for only positive reviews, while the
latter contains evidence for negative ones. We use the train/dev/test split
published by Pruthi et al. (2020).

| IMDB-Pos | IMDB-Neg
---|---|---
| Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | MAP | Time | Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | MAP | Time
Random Uniform | - | $5.46\pm 0.18$ | 4.27 | 8.46 | - | - | $6.02\pm 0.23$ | 4.80 | 9.90 | -
Longformer Weighted Soft Attention | 93.41 | $6.89\pm 0.19$ | 4.43 | 7.81 | 603 | 93.52 | $8.34\pm 0.98$ | 5.40 | 10.59 | 599
Longformer Self-Attention Top-K | 94.42 | $5.90\pm 2.21$ | 5.19 | 7.85 | 606 | 94.60 | $5.71\pm 3.63$ | 5.02 | 9.03 | 603
Longformer Ranked Soft Attention | 92.63 | $14.13\pm 2.27$ | 11.92 | 11.97 | 618 | 93.92 | $19.62\pm 0.61$ | 16.98 | 16.44 | 608
Compositional Soft Attention | 91.85 | $\textbf{25.46}\pm 0.97$ | 20.12 | 26.82 | 436 | 90.82 | $\textbf{27.27}\pm 3.68$ | 22.70 | 29.78 | 433
Table 2: Results for the Sentiment Detection IMDB datasets. Doc $F_{1}$
represents document-level classification performance, while $F_{1}$, $F_{0.5}$
and MAP represent token-level metrics. Time represents an average seconds per
epoch the given model took to train. We note the low performance of previously
used Weighted Soft Attention and Longformer Self-Attention. Both of our Ranked
Soft Attention and Compositional Soft Attention perform significantly better
on the rationale extraction. Compositional Soft Attention performs best across
all token-level metrics, while also exhibiting lower runtimes.

##  4 Results

We report the $F_{1}$-measure and the $F_{0.5}$-measure on the token level. We
further evaluate the models in the ranking setting by using Mean Average
Precision (MAP) for returning positive tokens. MAP serves as a diagnostic
metric that removes the threshold dependency and indicates which models have
learned to rank the tokens best. We further report a document-level
$F_{1}$-measure.

As a baseline, we use the global attention scores from the <CLS> token in the
Longformer’s last self-attention layer. We classify top $k\%$ of token scores
as rationale, where the value of $k$ is based on the rationale proportion in
the dataset. This is equivalent to evaluating the plausiblity of rationale
extracted by the support model in FRESH (Jain et al., 2020).

Details of our experimental setup are described in Appendix D. Tables 1 and 2
present results for the Grammatical Error Detection and Sentiment Detection
datasets respectively.

We find that both the Longformer Self-Attention and Longformer Weighted Soft
Attention perform poorly on the task of token-level rationale extraction. On
the longer Sentiment Detection dataset, the performance of both methods is on-
par with a random baseline. For GED, we note that the high token-level $F_{1}$
score of Weighted Soft Attention is due to the model assigning scores of $1$
to most tokens, as evident by the substantially lower $F_{0.5}$ score and high
recall (Appendix E). We suspect this homogeneity of token scores is caused by
only $2$ tokens per document receiving supervision signal each epoch. This
encourages the token scores to stay close to the initial values and the model
is unable to learn to provide plausible rationales for its predictions.
Increasing the supervision signal through Ranked Soft Attention significantly
improves the token-level $F_{0.5}$ results ($5.06\%-11.58\%$ absolute
increase).

We suspect the poor performance of Longformer Self-Attention is partly caused
by the use of normalized global attention from the <CLS> token. As the token
attention scores across the whole document have to sum up to $1$, very few
tokens would be assigned high scores. This is evidenced by the significantly
lower token-level recall than for other methods. While some improvement could
be obtained through fine-tuning the classification threshold, self-attention
methods also do not learn to rank tokens well, as indicated by the low MAP
scores, which are below the random baseline.

We notice that Compositional Soft Attention models exhibit significantly lower
runtimes ($30\%-60\%$) than Longformer-based methods. It significantly
outperforms all architectures on the longer IMDB datasets, while also
achieving results on-par with Ranked Soft Attention on the GED datasets. It
appears that the compositional nature of this architecture allowed the RoBERTa
model to learn to provide meaningful token-level scores for each sentence
individually. This is in contrast with Longformer-based methods, which
directly focus on modeling global dependencies and struggle to provide
meaningful token-level predictions. We provide example token-level predictions
in Appendix F.

##  5 Conclusion

We investigated unsupervised rationale extraction for long document
classifiers. Our experiments showed that standard transformers methods do not
perform well on longer texts. We proposed Ranked Soft Attention that works
well in conjuction with Longformer by increasing the supervision signal
available to each token. We further introduced a novel Compositional Soft
Attention architecture that uses RoBERTa to support documents of arbitrary
length. We found Compositional Soft Attention to significatly outperform
Longformer-based systems on longer documents, while being $30\%-60\%$ faster
to fine-tune.

## Limitations

Our work aims to bridge the gap between rationale extraction for standard
length transformers and long documents. We improve the scalability of such
methods by sequentially applying RoBERTa to each sentence instead of relying
on slower long text transformers. However, it is important to underline that
this method still doesn’t not permit application to texts of arbitrary length,
as the memory of the GPU is the main limitation. We hope for future work to
address this.

We also note the limited evaluation for truly long documents. This is due to
small number of long text datasets with token-level annotations available. We
encourage the development of such new datasets in the future.

We believe that current approaches of evaluating rationale extraction as a
sequence labeling problem do not scale well to longer documents. It is
becoming harder to quantitatively evaluate such models. We encourage future
work to investigate alternative methods of evaluating plausibility of token-
level rationale extractors for long texts.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Bryant et al. (2019) Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019\.  The BEA-2019 shared task on grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 52–75, Florence, Italy. Association for Computational Linguistics.
  * Bujel et al. (2021) Kamil Bujel, Helen Yannakoudakis, and Marek Rei. 2021.  Zero-shot sequence labeling for transformer-based sentence classifiers.  In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_ , pages 195–205, Online. Association for Computational Linguistics.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019.  What does bert look at? an analysis of bert’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020.  ERASER: A benchmark to evaluate rationalized NLP models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458, Online. Association for Computational Linguistics.
  * Jain et al. (2020) Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020.  Learning to faithfully rationalize by construction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4459–4473, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one?  _Advances in neural information processing systems_ , 32.
  * Mosbach et al. (2020) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020.  On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines.  In _International Conference on Learning Representations_.
  * Pappagari et al. (2019) Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba, Yishay Carmiel, and Najim Dehak. 2019.  Hierarchical transformers for long document classification.  In _2019 IEEE automatic speech recognition and understanding workshop (ASRU)_ , pages 838–844. IEEE.
  * Pruthi et al. (2020) Danish Pruthi, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2020.  Weakly-and semi-supervised evidence extraction.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3965–3970.
  * Rei and Søgaard (2018) Marek Rei and Anders Søgaard. 2018.  Zero-shot sequence labeling: Transferring knowledge from sentences to tokens.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 293–302.
  * Rei and Yannakoudakis (2016) Marek Rei and Helen Yannakoudakis. 2016.  Compositional sequence labeling models for error detection in learner writing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1181–1191.
  * Rudin (2018) Cynthia Rudin. 2018.  Please stop explaining black box models for high stakes decisions.  _stat_ , 1050:26.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Yannakoudakis et al. (2018) Helen Yannakoudakis, Øistein E Andersen, Ardeshir Geranpayeh, Ted Briscoe, and Diane Nicholls. 2018.  Developing an automated writing placement system for esl learners.  _Applied Measurement in Education_ , 31(3):251–267.
  * Yannakoudakis et al. (2011) Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011.  A new dataset and method for automatically grading esol texts.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , pages 180–189.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  In _NeurIPS_.
  * Zaidan et al. (2007) Omar Zaidan, Jason Eisner, and Christine Piatko. 2007.  Using “annotator rationales” to improve machine learning for text categorization.  In _Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference_ , pages 260–267.

##  Appendix A Supervision of Weighted Soft Attention

We further recall that the Weighted Soft Attention architecture uses the
following loss functions:

$\displaystyle L_{1}$ $\displaystyle=\sum_{j}(y^{(j)}-\tilde{y}^{(j)})^{2}$
(8) $\displaystyle L_{2}$
$\displaystyle=\sum_{j}(min(\widetilde{a_{i}})-0)^{2}$ (9) $\displaystyle
L_{3}$ $\displaystyle=\sum_{j}(max(\widetilde{a_{i}})-\tilde{y}^{(j)})^{2}$
(10) $\displaystyle L$ $\displaystyle=L_{1}+\gamma(L_{2}+L_{3})$ (11)

where $L_{1}$ optimises the document-level performance, $L_{2}$ ensures the
minimum attention score is close to $0$ and $L_{3}$ optimises the maximum
attention score to be close to the document label $\widetilde{y}^{(j)}$.

##  Appendix B Compositional Soft Attention

Figure 1: Compositional Soft Attention architecture for rationale extraction
and document classification. A standard size transformer is applied to each
sentence individually and the contextual token embeddings are then combined to
build a document-level representation.

We present the algorithm for Compositional Soft Attention in Algorithm 1 and
the overview of the architecture in Figure 1.

Algorithm 1 Compositional Soft Attention

for sentence $s_{j,i}$ in document $doc_{j}$ do

$T_{j,i}\leftarrow\text{Transformer}(s_{j,i})$

end for

$T_{j}\leftarrow[T_{j,1},...,T_{j,n}]$

$\tilde{y}^{(j)},\,\,\widetilde{a}=\text{SoftAttention}(T_{j})$

##  Appendix C Datasets

We present the summary statistics for the datasets used in our experiments in
Table 3.

##  Appendix D Experimental Setup

We use a pre-trained RoBERTa-base Liu et al. (2019) and Longformer-base
Beltagy et al. (2020) models, available through the HuggingFace library Wolf
et al. (2019). All experiments are performed on Nvidia Tesla P100. Following
Mosbach et al. (2020), we train for $20$ epochs, with the best performing
checkpoint chosen. Each experiment is repeated $3$ times and the average
results are reported. We perform significance testing using a two-tailed
paired t-test ($a=0.05$). We set $k$ based on the percentage of evidence
present for the positive class.

###  D.1 Baselines

For reference, we include a random baseline, which samples token-level scores
from the standard uniform distribution and uses a threshold $k$ to classify
the scores. We also compare against a Longformer’s last self-attention layer,
following insight by Jain et al. (2020) that self-attention performs well as a
rationale extractor.

##  Appendix E Full Results

We present full results of the experiments in Tables 4, 5, 6 and 7.

##  Appendix F Example Predictions

Furthermore, we provide more sample predictions made by different models in
Figures 2 and 3.

| FCE | BEA 2019 | IMDB-Pos | IMDB-Neg
---|---|---|---|---
Number of train samples | $722$ | $1120$ | $1200$ | $1200$
Number of dev samples | $51$ | $280$ | $299$ | $299$
Number of test samples | $66$ | $200$ | $300$ | $300$
Average text length (words) | $441$ | $213$ | $686$ | $686$
Maximum text length (words) | $725$ | $655$ | $1935$ | $1935$
% of texts $>512$ words | $16\%$ | $2\%$ | $73\%$ | $73\%$
% positive samples | $49\%$ | $46\%$ | $50\%$ | $50\%$
% negative samples | $51\%$ | $54\%$ | $50\%$ | $50\%$
% evidence | $13\%$ | $9\%$ | $8\%$ | $8\%$
Table 3: Statistics for the datasets used. All measured on the development
datasets. We note the low proportion of long texts in FCE and BEA 2019.

| Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | $P$ | $R$ | MAP
---|---|---|---|---|---|---
Random Uniform | - | $11.36\pm 0.32$ | $12.00$ | $12.47$ | $10.42$ | $16.68$
Longformer Weighted Soft Attention | $93.79$ | $\textbf{21.22}\pm 0.17$ | $14.59$ | $12.08$ | 88.38 | $16.38$
Longformer Soft Attention Top-K | 94.39 | $14.75\pm 3.21$ | $14.99$ | $15.16$ | $14.40$ | $13.76$
Longformer Ranked Soft Attention | $93.49$ | $19.11\pm 2.92$ | $19.65$ | 20.12 | $18.53$ | $18.55$
Compositional Soft Attention | $89.07$ | $20.44\pm 1.73$ | 20.22 | $20.10$ | $20.95$ | 19.44
Table 4: Full results for BEA 2019. We note that while Weighted Soft Attention performs best on the token-level, it is largely due to the model assigning scores of $1$ to most tokens, as indicated by the high recall. Using $F_{0.5}$ as an evaluation metric highlights this issue. Our proposed Compositional Soft Attention performs best on the token-level in terms of both $F_{0.5}$ and MAP. | Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | $P$ | $R$ | MAP
---|---|---|---|---|---|---
Random Uniform | - | $12.13\pm 0.36$ | $13.70$ | $14.99$ | $10.18$ | $15.91$
Longformer Weighted Soft Attention | 89.34 | $\textbf{24.81}\pm 1.42$ | $17.64$ | $14.81$ | 83.35 | $16.17$
Longformer Soft Attention Top-K | $89.29$ | $14.23\pm 2.25$ | $15.57$ | $16.61$ | $12.46$ | $14.67$
Longformer Ranked Soft Attention | $89.10$ | $22.23\pm 2.44$ | 23.40 | $24.51$ | $21.33$ | $18.82$
Compositional Soft Attention | $81.32$ | $21.08\pm 2.02$ | $23.19$ | 25.21 | $18.78$ | 19.67
Table 5: Full results for FCE GED dataset. Similarly to BEA 2019, we note that the Weighted Soft Attention performs best on the token-level, as evaluated by $F_{1}$ score. However, that is mainly because the model assigns $1$ to most tokens, causing recall to be high. This is evident by the significantly lower $F_{0.5}$ metric. Our Ranked Soft Attention and Compositional Soft Attention models achieve similar performance, significantly better than Weighted Soft Attention if evaluated on the $F_{0.5}$ and MAP metrics. | Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | $P$ | $R$ | MAP
---|---|---|---|---|---|---
Random Uniform | - | $5.46\pm 0.18$ | $4.27$ | $3.72$ | $10.24$ | $8.46$
Longformer Weighted Soft Attention | $93.41$ | $6.89\pm 0.19$ | $4.43$ | $3.58$ | 92.39 | $7.81$
Longformer Soft Attention Top-K | 94.42 | $5.90\pm 2.21$ | $5.19$ | $4.80$ | $7.67$ | $7.85$
Longformer Ranked Soft Attention | $92.63$ | $14.13\pm 2.27$ | $11.92$ | $10.80$ | $20.54$ | $11.97$
Compositional Soft Attention | $91.85$ | $\textbf{25.46}\pm 0.97$ | 20.12 | 17.66 | $45.85$ | 26.82
Table 6: Full results for the IMDB-Pos Sentiment Detection dataset. We note that our Compositional Soft Attention architecture performs significantly better across all token-level metrics. | Doc $F_{1}$ | $F_{1}$ | $F_{0.5}$ | $P$ | $R$ | MAP
---|---|---|---|---|---|---
Random Uniform | - | $6.02\pm 0.23$ | $4.80$ | $4.23$ | $10.46$ | $9.90$
Longformer Weighted Soft Attention | $93.52$ | $8.34\pm 0.98$ | $5.40$ | $4.37$ | 93.38 | $10.59$
Longformer Soft Attention Top-K | 94.60 | $5.71\pm 3.63$ | $5.02$ | $4.65$ | $7.40$ | $9.03$
Longformer Ranked Soft Attention | $93.92$ | $19.62\pm 0.61$ | $16.98$ | $15.63$ | $27.27$ | $16.44$
Compositional Soft Attention | $90.82$ | $\textbf{27.27}\pm 3.68$ | 22.70 | 20.42 | $41.22$ | 29.78
Table 7: Full results for the IMDB-Neg Sentiment Detection dataset. Similarly
to IMDB-Pos, we note that our Compositional Soft Attention architecture
performs significantly better across all token-level metrics.

(a) Sample token-level predictions for BEA 2019 positive sample (beginner
learner). Compositional Soft Attention finds all evidence, but also scores
neighboring tokens highly. Ranked Soft Attention on the other hand attends to
less neighbouring tokens. We note that this might explain the performance
differences between Grammatical Error Detection and Sentiment Detection
datasets, as the the former annotations are more concentrated than the latter.

(b) Excerpt from an FCE positive document (beginner learner) without any
grammatical errors in the sentence. We note that both Weighted Soft Attention
and Ranked Soft Attention find a lot of false positives. On the other hand,
Compositional Soft Attention correctly does not much rationale.

(c) Excerpt from a positive FCE document (beginner learner). This sentence
includes grammatical errors. Ranked Soft Attention manages to pick up some
manually annotated rationale, while Compositional Soft Attention fails.
However, Ranked Soft Attention also finds many false positives.

(d) An excerpt from a FCE positive document (beginner learner), for a sentence
with grammatical errors. Ranked Soft Attention fails to provide any rationale,
while Compositional Soft Attention correctly finds "," as rationale, but fails
to attend to neighboring tokens.

Figure 2: Example predictions for Grammatical Error Detection datasets. W-SA
corresponds to Weighted Soft Attention, R-SA to Ranked Soft Attention, while
C-SA to Compositional Soft Attention. We highlight words that human annotators
marked as rationale, while also marking true positives and false positives.

(a) Sample predictions for a sample negative review in the IMDB-Neg dataset.
We find that Weighted Soft Attention assigns similar scores to most tokens,
while Ranked Soft Attention and Compositional Soft Attention manage to provide
more fine-grained predictions. Compositional Soft Attention appears to recover
spans of rationale better than Ranked Soft Attention.

(b) Sample prediction for a positive movie review in the IMDB-Pos review. Here
we note how Compositional Soft Attention learns the correct ranking of tokens,
as evidenced by the higher scores for true positives than false positives.
However, it still failed to optimize correctly for the classification
threshold, leading numerous false positives. This problem is not present in
the Ranked Soft Attention for this sample.

(c) Sample predictions for a negative review in the IMDB-Pos dataset. The gold
token-level labels are all $0$, as there are no positive rationale in the
negative review. Weighted Soft Attention still assigns scores close to $1$ to
most tokens, while both Ranked Soft Attention and Compositional Soft Attention
learns not to attend to any tokens. This shows how the increased token-level
supervision signal helps these architectures to learn to provide better token-
level rationale.

Figure 3: Example predictions for Sentiment Detection IMDB datasets. W-SA
corresponds to Weighted Soft Attention, R-SA to Ranked Soft Attention, while
C-SA to Compositional Soft Attention. We highlight words that human annotators
marked as rationale, while also marking true positives and false positives.


# Uniform Complexity for Text Generation

Joseph Marvin Imperial
Human Language Technology Lab (NU HLT)
National University
Manila, Philippines
<EMAIL_ADDRESS>

###### Abstract

Large pre-trained language models have shown promising results in a wide array
of tasks such as narrative generation, question answering, and machine
translation. Likewise, the current trend in literature has deeply focused on
controlling salient properties of generated texts including sentiment, topic,
and coherence to produce more human-like outputs. In this work, we introduce
Uniform Complexity for Text Generation or UCTG which serves as a challenge to
make existing models generate uniformly complex text with respect to inputs or
prompts used. For example, if the reading level of an input text prompt is
appropriate for low-leveled learners (ex. A2 in the CEFR), then the generated
text by an NLG system should also assume this particular level for increased
readability. In a controlled narrative generation task, we surveyed over 160
linguistic and cognitively-motivated features for evaluating text readability
and found out that GPT-2 models and even humans struggle in preserving the
linguistic complexity of input prompts used. Ultimately, we lay down potential
methods and approaches which can be incorporated into the general framework of
steering language models towards addressing this important challenge.

##  1 Introduction

In a narrative writing process, it is a general practice to maintain the
complexity of text as one tries to complete the story. As such, the
readability of a text depends largely on the writer’s capability to reduce and
simplify the structure of words and sentences Fountas and Pinnell (1999);
DuBay (2004). For example, when writing a book for young learners in first
grade, one must take note of acceptable words that are within the vocabulary
of a first grader to avoid frustration in reading that will hinder effective
comprehension Gickling and Armstrong (1978); Guevarra (2011). This challenge
can also be emulated to natural language generators to test whether they
maintain the textual complexity of prompt inputs. Figure 1 shows an ideal
scenario where a prompt given by a human has the same level of reading
difficulty (66.9) as the generated text by a GPT-2 model (61.9) based on the
Flesch formula Flesch (1948). The interpretation of Flesch scores obtained
means both texts can be read easily by Grade 8 and 9 learners.

Figure 1: An illustrated example of a prompt-continuation pair produced using
a GPT-2 model having the same reading level based on the Flesch formula.

Generally, the complexity of a narrative or any piece of text at hand can be
measured by a multitude of content-based and linguistic factors. Some measures
that have been explored through the years include syntactic complexity as
equivalence of high-proficiency writing through presence of qualified
syntactic phrases or chunks such as words from a noun or verb phrase Beers and
Nagy (2009); McNamara et al. (2010); Roemmele et al. (2017), discourse
complexity by aggregating the presence of entities of a text such as mentions
people, organizations, and locations which can lead to increased working
memory burden Feng et al. (2009), and vocabulary complexity by matching words
from the text associated with a specific age-based difficulty level from
developmental studies Kuperman et al. (2012); Vajjala and Meurers (2016) to
name a few. These factors, when extracted from texts for evaluating
complexity, usually follow a general linear property: the complexity of a text
increases as the value of a measured feature from the text increases.

In this study, we formally introduce the task of uniform complexity for text
generation or UCTG framed using an open-ended narrative generation experiment
or more commonly known as story generation. To facilitate this, we used the
WritingPrompts dataset from Reddit Fan et al. (2018). To cover a wide range of
measures used for approximating linguistic complexity, we extracted over 160
features drawing concepts from age-of-acquisition in developmental studies,
word and phrase level part-of-speech, discourse from entity mentions, and
formula-based readability indices Lee et al. (2021). We compare text
continuations from three models: humans, an off-the-shelf GPT-2 model, and a
finetuned GPT-2 model. We perform a statistical test of difference for each
model and for each linguistic feature. Our results show that continuations
from both humans and neural language models like GPT-2, regardless of applied
finetuning, generally fail to maintain the complexity levels of the prompts.
We outline several potential methods that can be adapted to control the
complexity of existing NLG systems. For reproducibility and transparency, we
will release the processed dataset and the code upon acceptance.

##  2 Task Description

The proposed evaluation task can be generally adapted to any NLG framework as
long as texts are both the input and output. In the case of a narrative
generation setting, given a series of prompts
$P=\\{p_{1},p_{2},\ldots,p_{n}\\}$ and their continuations
$C=\\{c_{1},c_{2},\ldots,c_{n}\\}$ by a model $M$, a linguistic feature
function $F$ processes instances from both the prompt and continuation sets to
get their corresponding measures $F(p_{n})\in\mathbb{R}$ and
$F(c_{n})\in\mathbb{R}$. All calculations performed by $F$ should be
normalized with either number of word counts or a similar variable to avoid
higher values from longer generated texts. This is a common practice in text
complexity research to mitigate effects of length in feature values Lu (2012).
Then, a statistical test is then applied using the two groups to measure
significant difference in each linguistic complexity feature variable.

##  3 Generation Setup

We describe the training recipes and resources used for exploring UCTG in
human and GPT-2 models.

###  3.1 Human Prompt and Continuation Data

For the compilation of prompts and human continuation as benchmark, we used
the WritingPrompts111www.reddit.com/r/WritingPrompts/ dataset collected by Fan
et al. (2018). This dataset is derived from the r/WritingPrompts community of
the Reddit platform where it allows users to post story premises in various
context and genre for other members to continue with their own ideas and
creativity. The current compilation is divided into train, test, and
validation splits. For nature of this task, we only used the test split which
contains 15,138 pairs. In addition to the data preprocessing steps done by
DeLucia et al. (2021), we handpicked prompts with at least 30 words in length
to ensure that the neural models will have enough context for the generation
phase while capped human continuation texts with a minimum of 150 words and
maximum of 300 words to avoid long-range memory limitation in the small GPT-2
model used Sun et al. (2021). Overall, we arrive at a total of 941 prompt-
human continuation pairs as benchmark for the task.

###  3.2 Generation Models

To test whether large neural language models are capable of UCTG, we selected
the GPT-2 model Radford et al. (2019) for analysis due to its notable
extensive use in the NLP community in the narrative generation task See et al.
(2019); Xu et al. (2020); Akoury et al. (2020); Chang et al. (2021). We used
two versions of GPT-2: the small, off-the-shelf version with 117M parameters
from Huggingface222https://huggingface.co/gpt2 and the finetuned model from
DeLucia et al. (2021) using the same WritingPrompts dataset with the train
split. We no longer needed to reiterate trying out various top-$p$ and top-$k$
values as this has been exhaustively explored in DeLucia et al. (2021).
According to the study, the best values for nucleus sampling or top-$p$ for
the narrative generation task ranges from 0.7 to 0.95 wherein generated
stories are more vivid and of better quality as evaluated through human and
automatic means. For this work, we set top-$p$ to 0.7.

##  4 Difference in Prompt and Continuation Complexities

For the first experiment, we test for possible significant differences in the
complexity values of the prompts compared with three groups: (a) the generated
texts from humans of the r/WritingPrompts community, (b) the base GPT-2 model,
and (c) the finetuned GPT-2 model by DeLucia et al. (2021). We used the Welch
t-test Welch (1947) with Bonferroni correction resulting to an alpha level of
0.0167 (0.05/3). For the variables of interest, we extracted over 160
linguistic features using the LingFeat tool by Lee et al. (2021) which are
often used as predictors in text readability research. The tool covers
extraction of surface, lexical, phrasal, tree structure, type token,
psycholinguistics, and formula-based readability features detailed in the
subsections below. As mentioned in Section 2, we only used the average-based
linguistic complexity features (ex. count of noun POS / total count of words)
instead of raw counts to avoid bias or reaching extremely high values for
longer sentences.

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Total token x Total sent | 0.0000 | 0.0000 | 0.0001
Sqrt Total token x Total sent | 0.0000 | 0.0000 | 0.0001
Log token / Log sent | 0.0001 | 0.0001 | 0.0001
Avr token sent | 0.0001 | 0.0001 | 0.0001
Avr Syll sent | 0.0001 | 0.0001 | 0.0001
Avr Syll token | 0.0001 | 0.0001 | 0.0001
Avr Chars sent | 0.0001 | 0.0001 | 0.0001
Avr Chars token | 0.0020 | 0.0001 | 0.0001
Table 1: Shallow based features.

###  4.1 Shallow Features

For our first feature set, we looked at 8 shallow or surface-based textual
features as shown in Table 1 such as product and square root of total tokens
by total sentences, log densities of tokens to sentences, average token count
per sentence, average syllable count per sentence and per token, and average
character count per sentence and token. These linguistic features have been
extensively used text complexity assessment in a wide range of languages such
as in English Flesch (1948), French François and Miltsakaki (2012) and
Filipino Imperial and Ong (2021). From the result, all of the mentioned
features for the three groups appear to be significantly different with
respect to the complexity of the prompt. This preempts how even humans
subconsciously do not follow a uniform pattern related to surface and
frequency-based properties of texts in a narrative generation task.

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Flesch-Kincaid | 0.0008 | 0.0831 | 0.0001
NARI | 0.3535 | 0.0111 | 0.0001
Coleman-Liau | 0.0001 | 0.0001 | 0.0001
SMOG | 0.0000 | 0.0001 | 0.0026
Gunning-Fog | 0.0001 | 0.0002 | 0.0001
Linsear | 0.0001 | 0.0005 | 0.0001
Table 2: Formula based features.

###  4.2 Traditional Readability Features

Formula-based features for readability assessment also stem from a
combinations of surface-based features such as word length, sentence length,
and occurrence from a pre-defined dictionary of words. We covered 6 metrics
such as Flesch-Kincaid Kincaid et al. (1975), New Automated Readability Index
(NARI) Senter and Smith (1967), Coleman-Liau Coleman and Liau (1975), SMOG Mc
Laughlin (1969), Gunning-Fog Gunning et al. (1952), and Linsear Klare (1974).
Due to their ease of use, formulas such as the Flesch-Kincaid Reading Ease are
integrated in most text-based applications. From the results in Table 2,
majority of the narrative continuations from humans, base GPT-2 model, and
finetuned GPT-2 model were significantly different except for two: the Flesch-
Kincaid score obtained by the baseline GPT-2 model and the NARI score by
humans. The formula for NARI is the only one which uses number of characters
as predictor which may signal that human-continued texts are sensitive to
character count compared to neural model-generated texts. However,
collectively, the non-uniformity of complexity levels for this group is
expected as the predictors used for each formula also leverages on surface-
based features as previously mentioned.

###  4.3 Part of Speech Features

We further the analysis by looking deeper into the text structure via part of
speech (POS) tags. For this study, syntactic concepts covering nouns, verbs,
adverbs, adjectives, subordinating clauses and conjunction, coordinating
clauses and conjunction and prepositional phrases were used as predictors to
calculate the densities of prompts and text continuations in both sentence and
token level aspect Heilman et al. (2007); Lu (2012). Table 3 details 47 ratio
and average-based predictors which quantifies POS complexities of human and
neural model text continuations. Overall, the finetuned GPT-2 model obtained
the least number of complexity features that are significantly different with
the prompt with 31 (16 features non-significant). This is followed by the
human-generated continuations with 33 (14 features non-significant) and the
baseline GPT-2 with 37 (10 features non-significant). This result suggests
that fine-tuning the baseline GPT-2 model with the best value for sampling-
based decoding (top-$p$ = 0.7) somehow provides a certain level uniformity of
usage of grammatical entities with respect to prompts. Looking at the overlap
of non-significant features with respect to the prompt, the baseline GPT-2
model only coincided with one feature that is the average of adjective POS per
sentence. On the other hand, the finetuned GPT-2 model obtained 7 features
such as average of coordinating conjunction POS per sentence and ratio of
coordinating conjunction POS to adjective POS that overlapped with complexity
values of the prompt. This further supports the inference on result of
finetuning that was previously mentioned.

###  4.4 Type Token Features

Aside from looking at the average or ratio-based densities of POS tags
locally, syntactic complexity can also be measured via densities of collective
(more than one) POS tags per sentence. Table 4 details 5 type-token ratio
(TTR) based measures: simple type-token ratio O’Loughlin (1995) , correlated
type-token ratio Carroll (1964), bilogarithmic type-token ratio Herdan (1960),
Uber index Dugast (1978), and simple lexical diversity. Type token features
provide a quantified measure of unique word types (ex. combination of nouns,
verbs, adjectives, and adverbs) normalized by the total number of words in a
segment of a language. The variations of TTR have been studied over the years
to minimize effects of sentence length when calculating the values Herdan
(1960); Tweedie and Baayen (1998). From the results, the human-continued text
measured by the Uber index, a controlled metric for lexical diversity, was the
only non-significant feature. These results may suggest that texts generated
by humans, base GPT-2, and finetuned GPT-2 model may assume notable difference
in densities of various basic grammatical components such as nouns, verbs,
adjectives, and adverbs with respect to prompts used.

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Avr Noun POS sent | 0.0001 | 0.0194 | 0.0001
Avr Noun POS token | 0.0001 | 0.0001 | 0.0001
Noun POS to Adj POS | 0.0002 | 0.1467 | 0.0025
Noun POS to Verb POS | 0.0001 | 0.0001 | 0.0001
Noun POS to Advrb POS | 0.0001 | 0.0001 | 0.3215
Noun POS to SubrdConj | 0.0001 | 0.0001 | 0.0096
Noun POS to CordConj | 0.0001 | 0.0814 | 0.0002
Avr Verb POS sent | 0.1531 | 0.0001 | 0.0001
Avr Verb POS token | 0.0001 | 0.6279 | 0.0001
Verb POS to Adj POS | 0.0244 | 0.0006 | 0.0001
Verb POS to Noun POS | 0.0001 | 0.0001 | 0.0001
Verb POS to Advrb POS | 0.2302 | 0.0001 | 0.0001
Verb POS to SubrdConj | 0.0001 | 0.0001 | 0.0001
Verb POS to CordConj | 0.0001 | 0.0001 | 0.0001
Avr Adj POS sent | 0.0557 | 0.0735 | 0.0001
Avr Adj POS token | 0.7609 | 0.0013 | 0.6105
Adj POS to Noun POS | 0.1982 | 0.0027 | 0.0001
Adj POS to Verb POS | 0.0001 | 0.0092 | 0.0024
Adj POS to Advrb POS | 0.0001 | 0.0001 | 0.0506
Adj POS to SubrdConj | 0.0001 | 0.0001 | 0.0408
Adj POS to CordConj | 0.0001 | 0.0011 | 0.5120
Avr Advrb POS sent | 0.0001 | 0.0411 | 0.0001
Avr Advrb POS token | 0.0001 | 0.0668 | 0.3566
Advrb POS to Adj POS | 0.0001 | 0.0726 | 0.0429
Advrb POS to Noun POS | 0.0001 | 0.0008 | 0.0001
Advrb POS to Verb POS | 0.0001 | 0.0047 | 0.0001
Advrb POS to SubrdConj | 0.0001 | 0.0001 | 0.0138
Advrb POS to CordCobj | 0.0001 | 0.0247 | 0.5228
Avr SubrdConj sent | 0.2546 | 0.0001 | 0.0001
Avr SubrdConj token | 0.4488 | 0.0001 | 0.0023
SubrdConj POS to Adj POS | 0.1985 | 0.0001 | 0.2602
SubrdConj POS to Noun POS | 0.2955 | 0.0001 | 0.0024
SubrdConj POS to Verb POS | 0.0002 | 0.0001 | 0.0001
SubrdConj POS to Advrb POS | 0.0002 | 0.0001 | 0.8923
SubrdConj POS to CordConj POS | 0.0001 | 0.0001 | 0.2257
Avr CordConj POS sent | 0.0001 | 0.0001 | 0.0001
Avr CordConj POS token | 0.2407 | 0.0001 | 0.4773
CordConj POS to Adj POS | 0.0208 | 0.0001 | 0.1113
CordConj POS to Noun POS | 0.2194 | 0.0001 | 0.0037
CordConj POS to Verb POS | 0.0001 | 0.0006 | 0.0001
CordConj POS to Advrb POS | 0.0004 | 0.0001 | 0.0220
CordConj POS to SubrdConj POS | 0.0001 | 0.0001 | 0.0457
Avr Content Words sent | 0.0017 | 0.6532 | 0.0001
Avr Content Words token | 0.0001 | 0.0001 | 0.4892
Avr Function Words token | 0.0198 | 0.0001 | 0.0001
Avr Function Words token | 0.0001 | 0.0001 | 0.0001
Content to Function Words | 0.0001 | 0.0001 | 0.0001
Table 3: Part of speech based features. Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Simple TTR | 0.0001 | 0.0000 | 0.0001
Correlated TTR | 0.0000 | 0.0109 | 0.0001
BiLogarithmic TTR | 0.0001 | 0.0001 | 0.0001
Uber Index | 0.6039 | 0.0001 | 0.0001
Lexical Diversity | 0.0001 | 0.0001 | 0.0001
Table 4: Type token based features.

###  4.5 Lexical Variation Features

In complement to calculating ratios and averages of word-level POS
complexities, lexical variation can also signal difficulty via densities of
unique grammatical components Lu (2012). Table 6 describes 12 lexical
variation-based features focusing on simple, squared, and corrected versions
of unique counts of POS such as nouns, verbs, adjectives and adverbs
normalized its total in a sentence. From the results, only the simple unique
adverb count per sentence from the human generated continuations and the
corrected unique verb variation count per sentence from the baseline GPT-2
model obtained unsiginicant results. This finding may suggest that the
densities of unique POS tags is completely different with the prompt values
regardless of any human or neural-based generation mechanism.

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Avr Noun phrs sent | 0.0280 | 0.0001 | 0.0001
Avr Noun phrs token | 0.0001 | 0.0121 | 0.0001
Noun phrs to Verb phrs | 0.0001 | 0.0001 | 0.0001
Noun phrs to SubClaus | 0.0001 | 0.0757 | 0.0222
Noun phrs to Prep phrs | 0.0047 | 0.0067 | 0.0001
Noun phrs to Adj phrs | 0.0001 | 0.0001 | 0.0001
Noun phrs to Adv phrs | 0.0001 | 0.0001 | 0.0001
Avr Verb phrs sent | 0.2050 | 0.0001 | 0.0008
Avr Verb phrs token | 0.0001 | 0.0001 | 0.0001
Verb phrs to Noun phrs | 0.0335 | 0.0001 | 0.0001
Verb phrs to SubClaus | 0.0001 | 0.0001 | 0.0001
Verb phrs to Prep phrs | 0.5188 | 0.0168 | 0.0001
Verb phrs to Adj phrs | 0.0001 | 0.0001 | 0.0001
Verb phrs to Adv phrs | 0.0001 | 0.0001 | 0.0001
Avr SubClaus sent | 0.7199 | 0.0001 | 0.0021
Avr SubClaus token | 0.0003 | 0.0001 | 0.0001
SubClaus to Noun phrs | 0.2813 | 0.0001 | 0.0001
SubClaus to Verb phrs | 0.0033 | 0.8824 | 0.0001
SubClaus to Prep phrs | 0.0169 | 0.2844 | 0.0001
SubClaus to Adj phrs | 0.0001 | 0.0001 | 0.0001
SubClaus to Adv phrs | 0.0001 | 0.0001 | 0.0001
Avr Prep phrs sent | 0.1307 | 0.0001 | 0.0001
Avr Prep phrs token | 0.2976 | 0.6734 | 0.0001
Prep phrs to Noun phrs | 0.1831 | 0.5031 | 0.0001
Prep phrs to Verb phrs | 0.0001 | 0.0001 | 0.0001
Prep phrs to SubClaus | 0.0001 | 0.0288 | 0.0014
Prep phrs to Adj phrs | 0.0001 | 0.0001 | 0.1481
Prep phrs to Adv phrs | 0.0001 | 0.0001 | 0.1013
Avr Adj phrs sent | 0.0001 | 0.0029 | 0.2439
Avr Adj phrs token | 0.0001 | 0.0001 | 0.0001
Adj phrs to Noun phrs | 0.0002 | 0.0075 | 0.0001
Adj phrs to Verb phrs | 0.3468 | 0.2768 | 0.4599
Adj phrs to SubClaus | 0.0001 | 0.0014 | 0.0001
Adj phrs to Prep phrs | 0.6291 | 0.7919 | 0.0001
Adj phrs to Adv phrs | 0.0001 | 0.0001 | 0.0001
Avr Adv phrs sent | 0.0001 | 0.0003 | 0.0001
Avr Adv phrs token | 0.0001 | 0.3127 | 0.0001
Adv phrs to Noun phrs | 0.0001 | 0.9194 | 0.4361
Adv phrs to Verb phrs | 0.0001 | 0.0001 | 0.0001
Adv phrs to SubClaus | 0.0001 | 0.7101 | 0.2759
Adv phrs to Prep phrs | 0.0001 | 0.0603 | 0.0001
Adv phrs to Adj phrs | 0.0001 | 0.0001 | 0.0001
Table 5: Phrasal based features.

###  4.6 Phrasal Features

Moving on to longer sequences of part-of-speech complexities, we also measure
phrase-level linguistic features. Table 5 shows 42 phrase-based features
centering on token and sentence ratios of grammatical components such as noun
phrases, verb phrases, adverbial phrases, adjectival phrases, subordinate
phrases and prepositional phrases. Overall, the human continuation obtained 30
phrasal-based features which significantly different with respect to the
prompt while 29 for the baseline GPT-2 and 35 for the finetuned GPT-2 model.
In contrast to the observation in word-level POS features, for phrasal-based
POS, the baseline GPT-2 model have more coincided non-significant features
with the prompt—6 compared to the finetuned GPT-2 model with only 3. Some of
these features are ratios of prepositional phrases to noun phrases and verbial
phrases to prepositional phrases. From this result, we posit that avoiding
finetuning of the GPT-2 model preserves the syntactic structure of the
sentence (upper portions of the tree) at phrase level but not at finegrained
word level as seen in POS features at Table 3. In addition, this result also
preempts why Transformer-based models are often finetuned for paraphrasing
task to trigger require number of lexical swaps and syntactic diversity
Witteveen and Andrews (2019); Krishna et al. (2020).

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Simpl Noun variation | 0.0004 | 0.0001 | 0.0001
Sqrd Noun variation | 0.0001 | 0.0001 | 0.0001
Corr Noun variation | 0.0001 | 0.0001 | 0.0001
Simpl Verb variation | 0.0001 | 0.0001 | 0.0001
Sqrd Verb variation | 0.0001 | 0.0001 | 0.0001
Corr Verb variation | 0.0001 | 0.1771 | 0.0001
Simp Adj variation | 0.0001 | 0.0001 | 0.0001
Sqrd Adj variation | 0.0001 | 0.0001 | 0.0001
Corr Adj variation | 0.0001 | 0.0001 | 0.0001
Simp Adv variation | 0.1610 | 0.0001 | 0.0001
Sqrd Adv variation | 0.0001 | 0.0001 | 0.0001
Corr Adv variation | 0.0001 | 0.0001 | 0.0001
Table 6: Lexical variation based features.

###  4.7 Syntax Tree Features

Following the results of parse tree-based features in Table 5, analyzing the
difference in parse tree depth is a unique and interesting way of measuring
readability complexity as done in Schwarm and Ostendorf (2005). Table 7
describes 4 syntax tree height-based features including average heights of
regular and flattened trees per token and sentence. From the result, there is
a pattern seen with the human-generated texts as it obtained non-significance
with average regular tree height and average feature tree height at a
sentence-level. This suggests that neural model-based generated texts do not
conform to the one property of syntax which is the parse tree height in
contrast to human continuations.

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Avr Tree height sent | 0.0326 | 0.0001 | 0.0002
Avr Tree height token | 0.0001 | 0.0001 | 0.0001
Avr FTree height sent | 0.1125 | 0.0001 | 0.0001
Avr Ftree height token | 0.0001 | 0.0001 | 0.0001
Table 7: Syntax tree based features.

###  4.8 Psycholinguistic Features

We also look at psycholinguistic variables in reading using external wordlists
such as the Age-of-Acquisition (AOA) database compiled by Kuperman et al.
(2012) which contains over 30,000 English content words and the SubtlexUS by
Brysbaert and New (2009) which is a compilation of over 74,000 word forms with
frequency values extracted from 8,000 general films and series. These special
databases contain words that are associated to various age levels where
children expected to learn when they reach the stage. The works of Vajjala and
Meurers (2016) and Chen and Meurers (2016) both have leveraged on these
predictors in readability assessment and text familiarity. Using the LingFeat
tool, we extract over 26 token, lemma, and sentence-based normalizations of
AOA and SubtlexUS variations. Results from Table 8 show that not a single
model has coincided with each other in terms on non-significant feature. The
baseline GPT-2 model’s continuations are significantly different from the
prompts with respect to psycholinguistic features used while the finetuned
GPT-2 model only obtained non-significance from the averages of token-based
features from SubltexUS. This may faintly suggest that the finetuned GPT-2
model simply reuse some of the words present from the prompt that are
recognized by or included in the SubtlexUS database.

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
AOA word sent | 0.0001 | 0.0001 | 0.0001
AOA word token | 0.0001 | 0.0001 | 0.0001
AOA lemma sent | 0.0001 | 0.0001 | 0.0001
AOA lemma token | 0.0001 | 0.0001 | 0.0001
AOA lemma Bird sent | 0.0001 | 0.0001 | 0.0001
AOA lemma Bird token | 0.0094 | 0.0001 | 0.0001
AOA Bristol sent | 0.0001 | 0.0042 | 0.0001
AOA Bristol token | 0.0001 | 0.0001 | 0.0001
AOA CortKhanna sent | 0.0001 | 0.0042 | 0.0001
AOA CortKhanna token | 0.0001 | 0.0001 | 0.0001
SubtlexUS sent | 0.0001 | 0.0001 | 0.0001
SubtlexUS token | 0.0001 | 0.0001 | 0.6334
SubtlexUS CD sent | 0.0002 | 0.0001 | 0.0001
SubtlexUS CD token | 0.0001 | 0.0001 | 0.0001
SubtlexUS FREQ sent | 0.0001 | 0.0001 | 0.0001
SubtlexUS FREQ token | 0.0001 | 0.0001 | 0.0182
SubtlexUS CDL sent | 0.0001 | 0.0001 | 0.0001
SubtlexUS CDL token | 0.0001 | 0.0001 | 0.0001
SubtlexUS SUBTL sent | 0.0001 | 0.0001 | 0.0001
SubtlexUS SUBTL token | 0.0001 | 0.0001 | 0.6334
SubtlexUS Lg10WF sent | 0.0001 | 0.0001 | 0.0001
SubtlexUS Lg10WF token | 0.8759 | 0.0001 | 0.0001
SubtlexUS SubLCD sent | 0.0002 | 0.0001 | 0.0001
SubtlexUS SubLCD token | 0.0001 | 0.0001 | 0.0001
SubtlexUS LgCD sent | 0.0001 | 0.0001 | 0.0001
SubtlexUS LgCD token | 0.0003 | 0.0001 | 0.0001
Table 8: Psycholinguistics based features.

###  4.9 Discourse Features

For the last linguistic feature set investigated, we look at discourse in the
form of averages of unique and non-unique entity presence that can affect
working memory load as well as local coherence distance measures which
captures distribution and transitions of entities in a passage Barzilay and
Lapata (2008); Guinaudeau and Strube (2013). Feng et al. (2009) previously
applied these cognitively-motivated features for assessing reading difficulty
in the case of adults with intellectual disabilities. Table 9 shows 10
discourse-level features extracted from the prompt-continuation pairs where
majority of the features have shown significant difference for all models.
This finding may hint that the generated continuations from the three models
have dissimilar levels of dependencies with respect to the prompts or vice
versa.

Feature | Human | GPT2 | GPT2-FT
---|---|---|---
Avr Entity sent | 0.0001 | 0.0881 | 0.0001
Avr Entity token | 0.0001 | 0.0001 | 0.0001
Avr Uniq Entity sent | 0.0001 | 0.0001 | 0.0001
Avr Uniq Entity token | 0.0001 | 0.0001 | 0.0001
Local Coherence PA | 0.0001 | 0.0001 | 0.0087
Local Coherence PW | 0.0001 | 0.0001 | 0.0087
Local Coherence PU | 0.0001 | 0.0001 | 0.0001
Local Coh Dist PA | 0.0001 | 0.0001 | 0.0107
Local Coh Dist PW | 0.0001 | 0.0001 | 0.0107
Local Coh Dist PU | 0.0001 | 0.0001 | 0.0001
Table 9: Discourse based features. GPT-2 | GPT-2 FT
---|---
SMOG | 0.188 | Avr Syll token | 0.155
Avr Syll token | 0.187 | AOA Bristol token | 0.102
SubtlexUS Lg10WF token | 0.177 | AOA CortKhanna token | 0.102
SubtlexUS CD token | 0.162 | AOA lemma token | 0.102
SubtlexUS SubLCD token | 0.162 | AOA word token | 0.097
SubtlexUS CDL token | 0.159 | SubtlexUS CD token | 0.093
SubtlexUS LgCD token | 0.154 | SubtlexUS SubLCD token | 0.093
Avr Verb POS token | 0.120 | SubtlexUS CDL token | 0.090
AOA lemma token | 0.119 | Avr Adv phrs token | 0.080
AOA word token | 0.116 | Adv phrs to Noun phrs | 0.079
Table 10: Top correlated complexity features of GPT-2 and finetuned GPT-2
model with human continuations.

##  5 Correlation in Human and GPT-2 Continuations

Aside from prompt-wise comparison, we also look at which linguistic complexity
feature from the GPT-2 models are correlated with the human continuations to
identify. This answers the question of which model is closer to produce more
human-like continuations in the lens of text complexity features. For this, we
use Pearson correlation to do the continuation-wise analysis and extract the
top correlated features described in Table 10. From the Table, both baseline
GPT-2 and finetuned GPT-2 model obtained 7 top complexity features all from
the psycholinguistics category referencing from Age-of-Acquisition and
SubtlexUS databases. This may suggest that even if there is a dissimilarity
with the prompt-continuation comparison for all models, they are nonetheless
correlated in the sense with the human continuation against other complexity
features. Likewise, for both baseline and finetuned GPT-2 models, the average
syllable count per token emerged as the top common (weakly) correlated
features. However, we cannot draw solid conclusions of human-likeness from
this specific result since all of the features are weakly correlated in the
general sense ($r<$0.50).

##  6 Potential Methods for Controlling Complexity

We put forward a list of possible methods and related works of similar vein
with substantial potential for enforcing UCTG in existing NLG systems which
can be explored by future researchers.

Building A Uniformly Complex Prompts- Continuation Pairs Dataset. Probably the
most naive approach for this task is to compile a dataset containing uniformly
complex texts prompts and their corresponding continuations which belong to
the same readability level or complexity spectrum. The dataset has to be large
enough to finetune GPT-2 or any large pretrained neural language model used in
NLG systems. This approach is similar to the work of Agrawal and Carpuat
(2019) where they compiled English and Spanish news articles of diverse grade
levels for sentence alignment in translation. Alternatively, one could also
split lengthy collections of literary texts (such as ones found in Project
Gutenberg333https://www.gutenberg.org/) assuming the resulting text splits
will retain the same readability level or with range when validated. The UCTG
evaluation as seen in this study can then be re-applied to the newly trained
model to probe if there are non-significance from the linguistic complexity
measures of continuation texts with respect to the input text prompts used.

Post-Editing with Specialized Vocabularies. Another relatively simple method
with potential is applying lexical substitution of complex words from the
generated continuations with words from specialized wordlists such as the Age-
of-Acquistion (AoA) by Kuperman et al. (2012) or the Academic Vocabulary List
(AVL) by Gardner and Davies (2014). These resources, often used in
developmental and literacy-related studies, contain words and their
corresponding normalized age values where they are typically learned by
children. The updated AoA database contains over 50,000 words distributed over
various categories such as nouns, verbs, and adjectives while the AVL contains
3,015 lemmas occurring in all academic domains, their word forms, inflected
forms, and ratio value per million words. However, one possible limitation of
this approach is that databases have limitation on the scope of the age done
during collection. The AoA database, for instance, currently only captures
words learnt until the age of 25. Generally, this may already suffice if ever
a study would only focus on texts for early to intermediate level users.

Controlling Generations with Auxiliary Models. More advanced techniques for
control is through anchoring the complexity features of text with existing
plug-and-play methods Dathathri et al. (2019); Pascual et al. (2021)
leveraging on attribute models. These attribute models, commonly in the form
of simple bag-of-words model or a linear classifier, operate by shifting the
distribution of the generated text continuation in a way that it will
gradually conform to the desired attribute by acting as a scorer of quality of
texts generated. For UCTG, the desired attribute model can be the target
readability level or complexity measure which is similar to the prompt. In
literature, attribute models can be stacked together to improve generation
results. Thus, a readability attribute model can be developed and combined
with attribute models of fluency, topic, and sentiment for better quality of
results. One can also rethink the way of gauging the complexity of texts for
generation tasks. Recently, August et al. (2022) explored controllable
generation of scientific definitions using a novel re-ranking feature. In this
work, a BART model was prompted to generated 100s of candidate definitions
which were ranked by quality using a linear SVM or BERT-based discriminator.

##  7 Moving Forward

We propose a new challenge towards making existing natural language generation
systems controllable by taking a perspective in text complexity. While
previous works have explored only a small portion of linguistic complexity
features Roemmele et al. (2017); See et al. (2019), we provide the bigger,
more in-depth picture by analyzing over a hundred linguistic complexity
features to show that even human continuations, often regarded as gold-
standard in literature, generally do not maintain uniformity with the
distributional linguistic properties of the prompts. In hindsight, for us
humans, we do not have the capability to measure text complexity values while
reading on-the-fly which makes our judgment unreliable and subjective Deutsch
et al. (2020). Thus, in the field of education where proper matching of
reading materials is crucial DuBay (2004), this raises the need for a tool
that can act as writing assistants for authors and educators towards making
children’s literature fit for various levels of audiences and conforms to
specially-defined metrics such as the Common European Framework of Reference
for Languages (CEFR). Similarly, the continuations from the GPT-2 models are
no better when compared to human results showing no significant correlation
with any of the linguistic features investigated. To this end, we highlight
possible future directions such exploration on compilation of uniformly
complex prompt-continuation pairs, post-editing with specialized wordlists,
and exploiting external models as discriminators. We foresee increased
accessibility of powerful NLG systems if we also consider the angle of
readability and text complexity whenever we build upon these resources for
public use.

##  8 Limitations

This study tests whether humans and GPT-2, often used as a baseline model for
NLG tasks, exhibit any solid and significant evidence of being able to
maintain stable linguistic complexity measures. From a wide range of
structured and non-structured NLG tasks, we decide to restrict our experiments
to a narrative generation task as this is more closely related to the story
writing process often done in the education field where materials are strictly
classified into various age or grade level categories.

##  9 Ethical Considerations

We foresee no serious ethical concerns for this evaluation study.

##  10 Acknowledgments

The authors would like to thank Alexandra DeLucia from the Johns Hopkins
University Center for Language and Speech Processing for their help in
providing the small version of the finetuned GPT-2 model. This project is
funded by the National University - Center for Research and the Google AI
Tensorflow Faculty Award of main proponent.

## References

  * Agrawal and Carpuat (2019) Sweta Agrawal and Marine Carpuat. 2019.  Controlling text complexity in neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1549–1564, Hong Kong, China. Association for Computational Linguistics.
  * Akoury et al. (2020) Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020.  STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6470–6484, Online. Association for Computational Linguistics.
  * August et al. (2022) Tal August, Katharina Reinecke, and Noah Smith. 2022.  Generating scientific definitions with controllable complexity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8298–8317, Dublin, Ireland. Association for Computational Linguistics.
  * Barzilay and Lapata (2008) Regina Barzilay and Mirella Lapata. 2008.  Modeling local coherence: An entity-based approach.  _Computational Linguistics_ , 34(1):1–34.
  * Beers and Nagy (2009) Scott F Beers and William E Nagy. 2009.  Syntactic complexity as a predictor of adolescent writing quality: Which measures? which genre?  _Reading and Writing_ , 22(2):185–200.
  * Brysbaert and New (2009) Marc Brysbaert and Boris New. 2009.  Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english.  _Behavior research methods_ , 41(4):977–990.
  * Carroll (1964) John B Carroll. 1964.  Language and thought.  _Foundations of Modern Psychology Series_.
  * Chang et al. (2021) Haw-Shiuan Chang, Jiaming Yuan, Mohit Iyyer, and Andrew McCallum. 2021.  Changing the mind of transformers for topically-controllable language generation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2601–2611, Online. Association for Computational Linguistics.
  * Chen and Meurers (2016) Xiaobin Chen and Detmar Meurers. 2016.  Characterizing text difficulty with word frequencies.  In _Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 84–94, San Diego, CA. Association for Computational Linguistics.
  * Coleman and Liau (1975) Meri Coleman and Ta Lin Liau. 1975.  A computer readability formula designed for machine scoring.  _Journal of Applied Psychology_ , 60(2):283.
  * Dathathri et al. (2019) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019.  Plug and play language models: A simple approach to controlled text generation.  _arXiv preprint arXiv:1912.02164_.
  * DeLucia et al. (2021) Alexandra DeLucia, Aaron Mueller, Xiang Lisa Li, and João Sedoc. 2021.  Decoding methods for neural narrative generation.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 166–185, Online. Association for Computational Linguistics.
  * Deutsch et al. (2020) Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020.  Linguistic features for readability assessment.  In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * DuBay (2004) William H DuBay. 2004.  The principles of readability.  _Online Submission_.
  * Dugast (1978) D Dugast. 1978.  _Sur quoi se fonde la notion d’étendue théoretique du vocabulaire?_ , volume 46.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
  * Feng et al. (2009) Lijun Feng, Noémie Elhadad, and Matt Huenerfauth. 2009.  Cognitively motivated features for readability assessment.  In _Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)_ , pages 229–237, Athens, Greece. Association for Computational Linguistics.
  * Flesch (1948) Rudolph Flesch. 1948.  A new readability yardstick.  _Journal of applied psychology_ , 32(3):221.
  * Fountas and Pinnell (1999) Irene C Fountas and Gay Su Pinnell. 1999.  _Matching Books to Readers: Using Leveled Books in Graded Reading, K-3._ ERIC.
  * François and Miltsakaki (2012) Thomas François and Eleni Miltsakaki. 2012.  Do NLP and machine learning improve traditional readability formulas? In _Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations_ , pages 49–57, Montréal, Canada. Association for Computational Linguistics.
  * Gardner and Davies (2014) Dee Gardner and Mark Davies. 2014.  A new academic vocabulary list.  _Applied Linguistics_ , 35(3):305–327.
  * Gickling and Armstrong (1978) Edward E Gickling and David L Armstrong. 1978.  Levels of instructional difficulty as related to on-task behavior, task completion, and comprehension.  _Journal of Learning Disabilities_ , 11(9):559–566.
  * Guevarra (2011) Rowena C. Guevarra. 2011.  Development of a Filipino text readability index.
  * Guinaudeau and Strube (2013) Camille Guinaudeau and Michael Strube. 2013.  Graph-based local coherence modeling.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 93–103, Sofia, Bulgaria. Association for Computational Linguistics.
  * Gunning et al. (1952) Robert Gunning et al. 1952.  Technique of clear writing.
  * Heilman et al. (2007) Michael Heilman, Kevyn Collins-Thompson, Jamie Callan, and Maxine Eskenazi. 2007\.  Combining lexical and grammatical features to improve readability measures for first and second language texts.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , pages 460–467, Rochester, New York. Association for Computational Linguistics.
  * Herdan (1960) Gustav Herdan. 1960.  _Type-token Mathematics_.  Mouton.
  * Imperial and Ong (2021) Joseph Marvin Imperial and Ethel Ong. 2021.  Diverse linguistic features for assessing reading difficulty of educational filipino texts.  _arXiv preprint arXiv:2108.00241_.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975\.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.  Technical report, Naval Technical Training Command Millington TN Research Branch.
  * Klare (1974) George R Klare. 1974.  Assessing readability.  _Reading Research Quarterly_ , pages 62–102.
  * Krishna et al. (2020) Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020.  Reformulating unsupervised style transfer as paraphrase generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 737–762, Online. Association for Computational Linguistics.
  * Kuperman et al. (2012) Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012.  Age-of-acquisition ratings for 30,000 english words.  _Behavior research methods_ , 44(4):978–990.
  * Lee et al. (2021) Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021.  Pushing on text readability assessment: A transformer meets handcrafted linguistic features.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10669–10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lu (2012) Xiaofei Lu. 2012.  The relationship of lexical richness to the quality of esl learners’ oral narratives.  _The Modern Language Journal_ , 96(2):190–208.
  * Mc Laughlin (1969) G Harry Mc Laughlin. 1969.  Smog grading-a new readability formula.  _Journal of reading_ , 12(8):639–646.
  * McNamara et al. (2010) Danielle S McNamara, Scott A Crossley, and Philip M McCarthy. 2010.  Linguistic features of writing quality.  _Written communication_ , 27(1):57–86.
  * O’Loughlin (1995) Kieran O’Loughlin. 1995.  Lexical density in candidate output on direct and semi-direct versions of an oral proficiency test.  _Language Testing_ , 12(2):217–237.
  * Pascual et al. (2021) Damian Pascual, Beni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer. 2021.  A plug-and-play method for controlled text generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3973–3997, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9.
  * Roemmele et al. (2017) Melissa Roemmele, Andrew S Gordon, and Reid Swanson. 2017.  Evaluating story generation systems using automated linguistic analyses.  In _SIGKDD 2017 Workshop on Machine Learning for Creativity_ , pages 13–17.
  * Schwarm and Ostendorf (2005) Sarah Schwarm and Mari Ostendorf. 2005.  Reading level assessment using support vector machines and statistical language models.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 523–530, Ann Arbor, Michigan. Association for Computational Linguistics.
  * See et al. (2019) Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019.  Do massively pretrained language models make better storytellers? In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 843–861, Hong Kong, China. Association for Computational Linguistics.
  * Senter and Smith (1967) RJ Senter and Edgar A Smith. 1967.  Automated readability index.  Technical report, Cincinnati Univ OH.
  * Sun et al. (2021) Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021.  Do long-range language models actually use long-range context? In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 807–822, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tweedie and Baayen (1998) Fiona J Tweedie and R Harald Baayen. 1998.  How variable may a constant be? measures of lexical richness in perspective.  _Computers and the Humanities_ , 32(5):323–352.
  * Vajjala and Meurers (2016) Sowmya Vajjala and Detmar Meurers. 2016.  Readability-based sentence ranking for evaluating text simplification.  _arXiv preprint arXiv:1603.06009_.
  * Welch (1947) Bernard L Welch. 1947.  The generalization of ‘student’s’problem when several different population varlances are involved.  _Biometrika_ , 34(1-2):28–35.
  * Witteveen and Andrews (2019) Sam Witteveen and Martin Andrews. 2019.  Paraphrasing with large language models.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 215–220, Hong Kong. Association for Computational Linguistics.
  * Xu et al. (2020) Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020.  MEGATRON-CNTRL: Controllable story generation with external knowledge using large-scale language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2831–2845, Online. Association for Computational Linguistics.