Received July 15, 2019, accepted August 16, 2019, date of publication August
26, 2019, date of current version September 12, 2019.
10.1109/ACCESS.2019.2937505

This work was partially supported by grants TEC2016-76465-C2-2-R (Mineco,
Spain), GRC-2018/53 and ED341D R2016/012 (Xunta de Galicia, Spain), and a
University of Vigo travel grant to visit the CLAN Research Group at the
University of Aberdeen in Scotland.

Corresponding author: Silvia García-Méndez (e-mail: sgarcia@gti.uvigo.es).

# A System for Automatic English Text Expansion

SILVIA GARCÍA-MÉNDEZ1  MILAGROS FERNÁNDEZ-GAVILANES2, ENRIQUE COSTA-
MONTENEGRO1, JONATHAN JUNCAL-MARTÍNEZ1, FRANCISCO J. GONZÁLEZ-CASTAÑO1, AND
EHUD REITER3  GTI Research Group, Telematics Engineering Department, School of
Telecommunications Engineering, University of Vigo, 36310 Vigo, Spain  Defense
University Center, 36920 Marín, Pontevedra, Spain  CLAN Research Group,
Department of Computing Science, Meston Building, University of Aberdeen,
Aberdeen AB24 3UE, Britain

###### Abstract

We present an automatic text expansion system to generate English sentences,
which performs automatic Natural Language Generation (nlg) by combining
linguistic rules with statistical approaches. Here, “automatic” means that the
system can generate coherent and correct sentences from a minimum set of
words. From its inception, the design is modular and adaptable to other
languages. This adaptability is one of its greatest advantages. For English,
we have created the highly precise aLexiE lexicon with wide coverage, which
represents a contribution on its own. We have evaluated the resulting nlg
library in an Augmentative and Alternative Communication (aac) proof of
concept, both directly (by regenerating corpus sentences) and manually (from
annotations) using a popular corpus in the nlg field. We performed a second
analysis by comparing the quality of text expansion in English to Spanish,
using an ad-hoc Spanish-English parallel corpus. The system might also be
applied to other domains such as report and news generation.

###### Index Terms:

Augmentative and Alternative Communication, Natural Language Generation,
sentence planning, surface realiser, text expansion.

=-15pt

##  I Introduction

Natural language generation (nlg) systems [1] take quantitative, visual and
linguistic data (words, sentences, texts) as input. In particular, we are
interested in text-to-text expansion, which generates complete sentences or
even texts from some meaningful words. A text expansion system must add
elements like conjunctions and prepositions to transform the input words into
linguistically correct outputs. For example, given the input words ‘she’,
‘look’, ‘picture’, ‘yesterday’, ‘not’, the output might be ‘She did not look
at the picture yesterday’. This is useful in Augmentative and Alternative
Communication (aac) [2], for instance. In this case, a user would select
pictograms corresponding to input words on a tablet to obtain the equivalent
linguistically correct written and oral output.

Our approach, which is modular and extensible to other languages, is based on
our previous work on automatic nlg systems for Spanish [3, 4]. We have
improved modularity that allows to easily replace domain-dependent components.

The rest of this paper is organised as follows. In Section II we review the
state-of-the-art, paying special attention to English nlg, automatic
generation and text expansion. In Section III we describe our contribution, a
system for English text expansion based on the aLexiE English lexicon (Section
III-A), a grammar (Section III-B), and a sentence planner and a surface
realiser (Section III-C). In Section IV we present our evaluation results,
based on a widely used corpus in the nlg field (Section IV-B1) and an ad-hoc
Spanish-English parallel corpus (Section IV-B2). Finally, Section V concludes
the paper.

##  II Related work

nlg tasks generally follow well-defined sub-tasks [1, 5]: content
determination, text structuring, sentence aggregation, lexicalisation
(expressing information with the right words), referring expression generation
(domain objects identification), and linguistic realisation (text
correctness).

Content determination filters irrelevant information. It is obviously context-
and application-dependent. For this sub-task, it has been proposed to apply
data-driven techniques [6].

Text structuring plans the discourse and sets the sentence presentation order.
Recent solutions are based on manual rules [7, 8] and Systemic Functional
Grammar (sfg) [9], Rhetorical Structure Theory (rst) [10], Meaning-Text Theory
(mtt) [11], Machine Learning [12, 13], and the Centred Theory [14], to cite
some.

Semantic- and syntactic-level sentence aggregation join data into single
fluent and readable sentences [15].

Lexicalisation transforms the result of sentence aggregation into natural
language (nl) but it must choose between different expression alternatives in
nl. Generally, results improve with more alternatives [16].

Referring Expression Generation (reg) selects the phrases or words that can
unambiguously describe domain entities. It selects the best properties to
distinguish elements, and it discards irrelevant information. reg algorithms
include the Full Brevity Algorithm [17], the Greedy Heuristic [18, 19], and
the Incremental Algorithm [20].

Linguistic realisation comprises generation of morphological forms as well as
insertion of auxiliary verbs, prepositions and punctuation marks. It may fall
into the generation gap, because input data are often incomplete (i.e. they
lack elements to be syntactically comprehensible) [21]. Templates avoid this
gap and ensure consistency. However, using templates to automatically
transform data into text (which are typically used in applications such as
weather, traffic, sports, and health reporting) only yields better results
than other approaches in small domains with little variation. Hand-coded
grammar-based systems outperform templates when detailed input is available as
in the case of kpml111Available at http://www.fb10.uni-bremen.de/anglistik
/langpro/kpml/README.html, July 2019. [9]. There are other alternatives like
statistical methods that produce probabilistic grammars from large corpora
increasing coverage at less effort [22]; such as the Head-driven Phrase
Structure Grammar (hpsg) [23]; the Lexical-Functional Grammar (lfg) [24]; the
Tree-Adjoining Grammar (tag) [25], and some Deep Learning approaches [26]. We
employ a hybrid system that combines the advantages of stochastic and grammar-
based systems with low nlg complexity since it uses only keywords to generate
complete sentences.

Intelligent nlg architectures may be modular, (roughly following the previous
stages of macro-planning and text structuring, micro-planning or sentence
aggregation, lexicalisation, referring expression generation, and linguistic
realisation with syntactic and morphological rules); planning-oriented (which
are less modular); or data-driven, based on statistical learning. Rule- (or
template-) based approaches [27, 28], however, are the most extended nowadays
in real applications.

According to types of output texts, these may be informative texts, summaries,
dialogues, recommendations, and persuasions or creative writings. Most
informative text systems generate routinary information from quantitative data
[29, 30, 31]. Summary generation [32] has applications in areas such as
medicine, sports, and finance. Persuasive texts are intended to shape user
behaviour [33]. Dialogue systems, of interest for call centres or gaming
interfaces, focus on human-machine communication [34, 35]. Creative text
generation is extremely difficult. It has been demonstrated that predefined
templates are too rigid for it [36]. Affective nlg systems have been able to
generate texts beyond factual information such as poetry [37], but the
operator has no control over the process. The quality of the results in nlg is
measured in terms of adequacy, fluency, readability, and variation [38]. There
is a trade-off between efficiency and output quality.

Next we review the most relevant existing systems. SimpleNLG [39] performs
surface realisation by following a knowledge-based approach. Originally for
English, it is now available for German [40], French [41], Brazilian
Portuguese [42], Italian [43], Spanish [44, 45], Dutch [46], Mandarin [47],
and Galician [48]. This library has had strong impact in nlg due to its simple
usage. Its main disadvantage from the perspective of our research is its
manual operation. The NaturalOWL [49] data-to-text manual tool imposes a
complex input format. It generates texts from the owl ontology. SumTime [50,
51], also for data-to-text generation, is highly sensitive to language
variations and, thus, it is only adequate for the language it was designed
for.

The system in [52] comprises a trainable sentence planner and a probabilistic
surface realiser. Its modular design is similar to ours, but it is only
available for English and it is not easily adaptable to other languages since
certain language-dependent resources are required to train the surface
realiser (a tag grammar, the source of the supertags to annotate the semi-
specified tag derivation tree, a treebank to obtain the tree stochastic model
driving the tree chooser and a corpus of sentences to train the language model
required for the linear precedence chooser). Our system also needs two
language-dependent elements, the lexicon and the grammar, but they can be
easily replaced and we provide enough information to create/adapt these
elements to any domain.

In the particular field of text expansion in English applied for aac, we must
mention the work by [53], in line with automatic generation. Their sentence
compansion technique [54, 55]222We thank Kathleen F. McCoy for informing us
about project compansion. takes a compressed message and expands it into a
well-formed sentence. In practice it is useful as a writing assistant or a
conversational aide in situations where grammatically correct output is
desired. Its bottleneck is its generation method, based on a unification-type
grammar that needs to explore many possibilities to deliver its output. Thus,
it is highly time consuming. Besides, most of its operation is based on
markers linked to a lexicon, as in the case of plural forms and possessives
(the system might interpret the noun ‘apple’ followed by a plural marker as
‘apples’; the pronoun ‘I’ followed by a possessive marker would be interpreted
as ‘my’). Input words must be present in the lexicon, since they are taken
from a word board and the user is not free to enter them. To avoid this
constraint we have designed an automatic procedure for lexicon acquisition. It
does not need markers, so that the entire generation process is much faster.

Summing up, most nlg systems are purpose-built and, as such, they are highly
sensitive to problem characterization. On the contrary, thanks to the
modularity of our system we can isolate domain-dependent modules (grammar and
lexica) from domain-independent ones (nlg surface realiser). It can be
tailored to different applications and domains using the corresponding
syntactic structures and vocabularies. It can be easily extended to other
languages as well.

As far as we know there are no other systems for automatic text-to-text
expansion in English with a hybrid architecture.

##  III Methodology and architecture of the proposed system

###  III-A aLexiE lexicon

This section describes the morphological part, that is, the lexicon providing
linguistic knowledge. This is our first contribution.

We pursue a fully automatic nlg system. It must select the grammar structure
for the input words and their inflection. Therefore we need an ample
vocabulary with linguistic data. The aLexiE lexicon serves this purpose. We
created it by interpreting input resources and automatically (without human
supervision) merging them with the two-step methodology in [56] followed by a
final verification step (similarly to [4]):

  1. 1.

Extraction of all possible entries and translation to a common format
(Algorithm 1).

  2. 2.

Automatic comparison and combination of existing lexica to create the new
resource (graph unification in [57] and [58]).

  3. 3.

Lexical verification of extracted and translated entries and their categories
against the Merriam-Webster Dictionary333Available at https://www.merriam-
webster.com, July 2019. (mwd) (Algorithm 2).

####  III-A1 Linguistic resources and creation of aLexiE

We built aLexiE from free English linguistic resources. We prioritized
correctness and coverage and selected the following:

  * •

The morphological and syntactic English lexicon from the Alexina
Project444Available at https://gforge.inria.fr/projects/alexina, July 2019.
(EnLex) [59].

  * •

The Specialist Lexicon555Available at
https://lsg3.nlm.nih.gov/LexSysGroup/Pro-
jects/lexicon/current/web/index.html, July 2019. (nih) of medical terms and
everyday words.

  * •

The Freeling English dictionary666Available at
http://nlp.lsi.upc.edu/freeling/index.php/ node/12, July 2019. (en-freeling),
automatically extracted from wsj (Wall Street Journal) and other corpora.

We performed extraction and mapping independently. Once the information was
taken from the selected resources, it was transformed to a common format.
Unlike for the Spanish version in [4], input resources were unrelated, and
thus we conducted independent extraction and translation stages for each
selected resource.

We first extracted entries from EnLex tagged as noun, pronoun, verb,
adjective, adverb, determiner, conjunction, and preposition, ignoring
interjections, numerals, and proper nouns. Each resulting EnLex word entry was
translated to the extensional Alexina format [60]. As an example, Listing 1
illustrates the EnLex entry for the English lemma ‘picture’. It can be
observed that this noun (cat=n), has two forms, ‘picture’ and ‘pictures’,
respectively for masculine singular and plural.

LISTING 1: Example of the English lemma ‘picture’ in EnLex.

⬇

picture N2 100;Lemma;N;;cat=N;

%default #dela+multext+init

<table name=”N2” rads=”.*”>

<form suffix=” tag=”s”/>

<form suffix=”s” tag=”p”/>

</table>

We did the same for the other two linguistic resources that we selected:
nih777We extracted adjectives, adverbs, conjunctions, determiners,
prepositions, and pronouns from this resource. There was not inflection
information in the entries. and the Freeling English Dictionary888From this
resource we extracted adjectives, adverbs, and verbs..

We kept the present, past, present participle, and past participle forms of
English verbs. This information allowed adjusting the verbal tense to context-
dependent semantic features. In the case of adjectives, we did not save the
comparative and superlative forms (we leave comparative and superlative
clauses to future work).

The merging process requires to handle the issue of the different formats and
tags of word entries in the selected resources. Algorithm 1 converts them to a
common format (note: $e$ is an entry in a lexicon).

Algorithm 1 : Extraction and conversion algorithm

function extraction_mapping({EnLex},{nih},{en-freeling})

for $e_{\mbox{\scriptsize\sc EnLex}}\in\\{\mbox{\sc EnLex}\\}$ do

lemma${}_{e_{\mbox{\scriptsize\sc EnLex}}}$ = $e_{\mbox{\scriptsize\sc
EnLex}}$.getLemma()

category${}_{e_{\mbox{\scriptsize\sc EnLex}}}$ = $e_{\mbox{\scriptsize\sc
EnLex}}$.getCategory()

end for

for $e_{\mbox{\scriptsize\sc nih}}\in\\{\mbox{\sc nih}\\}$ do

lemma${}_{e_{\mbox{\scriptsize\sc nih}}}$ = $e_{\mbox{\scriptsize\sc
nih}}$.getLemma()

category${}_{e_{\mbox{\scriptsize\sc nih}}}$ = $e_{\mbox{\scriptsize\sc
nih}}$.getCategory()

end for

for $e_{\mbox{\scriptsize\sc en-freeling}}\in\\{\mbox{\sc en-freeling}\\}$ do

lemma${}_{e_{\mbox{\scriptsize\sc en-freeling}}}$ = $e_{\mbox{\scriptsize\sc
en-freeling}}$.getLemma()

category${}_{e_{\mbox{\scriptsize\sc en-freeling}}}$ =
$e_{\mbox{\scriptsize\sc en-freeling}}$.getCategory()

end for

return({EnLex},{nih},{en-freeling})

end function

Verification in Algorithm 2 checks the quality of the word entries. It looks
for each lemma and its lexical categories in mwd (this dictionary has the
advantage that it allows more web queries than other freely available online
dictionaries).

Finally, collected entries are merged in a combination step that applies the
graph unification in [57] and [58]. This operation validates common
information by integrating data of different nature and discarding
inconsistent information. Specifically,

  1. 1.

It joins all entries with a common lemma (homography is only considered for
different lexical categories).

    1. (a)

For the entries that results from (1), feature structures are unified.

    2. (b)

Next, a new aLexiE entry is created with these structures. The entry comprises
all common information plus any particular data in the source entries.

  2. 2.

A new aLexiE entry is created for any lexical entry that cannot be generated
by combining entries from other lexica.

Algorithm 2 : Verification algorithm

function verification({set})

for $e_{\mbox{\scriptsize\sc set}}\in\\{\mbox{\sc set}\\}$ do

lemma${}_{e_{\mbox{\scriptsize\sc set}}}$ = $e_{\mbox{\scriptsize\sc
set}}$.getLemma()

category${}_{e_{\mbox{\scriptsize\sc set}}}$ = $e_{\mbox{\scriptsize\sc
set}}$.getCategory()

if (lemma${}_{e_{\mbox{\scriptsize\sc set}}}$.isNOTinmwd() or
category${}_{e_{\mbox{\scriptsize\sc set}}}$.isNOTinmwd())

then

{set}.delete(e${}_{\mbox{\scriptsize\sc set}})$

end if

end for

return({set})

end function

We remark that the common extraction and translation format avoids
inconsistencies in this merging procedure. Algorithm 3 combines all steps in
this section so far.

Algorithm 3 : Lexicon building algorithm

{EnLex}= LoadEnLex()

{nih}= LoadNIH()

{en-freeling}= LoadEn-Freeling()

extraction_mapping({EnLex},{nih},{en-freeling})

{aLexiE} = {EnLex} $\cup$ {nih} $\cup$ {en-freeling}

verification({aLexiE})

Therefore, aLexiE was built from inputs extracted from previously existing
resources, which were merged into a common format and finally verified.
Listing 2 shows the result for the lemma ‘picture’.

Note that, in Listing 2, this lemma is semantically tagged as an object. We
used the Multilingual Central Repository999A lexical database that integrates
the Spanish WordNet into the EuroWordNet framework, see
http://adimen.si.ehu.es/web /MCR, July 2019. (mcr) [61] to get the semantic
classification of nouns in aLexiE. Algorithm 4 summarises this procedure.

Due to its size, indexing aLexiE allows our system to conduct the whole nlg
process much more quickly.

####  III-A2 Automatic extension of the lexicon

Figure 1:  Syntax tree resulting from the grammar

nlg can be simplified by avoiding inputs with little meaning, such as
prepositions. Consequently, it is necessary to infer a priori which
preposition follows a particular verb. Indeed, a major challenge in text
expansion is inferring missing prepositions. We trained this process from the
text in the English Wikipedia, which was previously pos-tagged with Spacy
Tagger101010Available at https://spacy.io, July 2019.. The language model for
this training was based on trigrams centered around verbs, using syntactic and
semantic information.

LISTING 2: Example of the English lemma ‘picture’ in aLexiE.

⬇

<?xml version=”1.0” encoding=”UTF-8”

standalone=”no”?>

<lexicon>

<word>

<lemma>picture</lemma>

<category>noun</category>

<number>singular</number>

<plural>pictures</plural>

<semantic_tag>object</semantic_tag>

</word>

</lexicon>

Algorithm 4 : Adding syntactic and semantic data

function Add_Syntatic&Semantic_Data({alexie})

for $e_{\mbox{\scriptsize\sc alexie}}\in\\{\mbox{\sc alexie}\\}$ do

lemma${}_{e_{\mbox{\scriptsize\sc alexie}}}$ = $e_{\mbox{\scriptsize\sc
alexie}}$.getLemma()

category${}_{e_{\mbox{\scriptsize\sc alexie}}}$ = $e_{\mbox{\scriptsize\sc
alexie}}$.getCategory()

if (category${}_{e_{\mbox{\scriptsize\sc alexie}}}$.isNoun() and
lemma${}_{e_{\mbox{\scriptsize\sc alexie}}}$.isInmcr())

then

$e_{\mbox{\scriptsize\sc mcr}}$ =
searchInMCR(lemma${}_{e_{\mbox{\scriptsize\sc alexie}}}$)

{mcr}.add($e_{\mbox{\scriptsize\sc mcr}}$)

end if

end for

end function

As previously mentioned, we used mcr to get the semantic classification of the
entries tagged as nouns in aLexiE. In this case we established four semantic
categories to start working with: living things, foodstuff, places, and
objects. For each verb lemma in the training set we analysed if it was
followed by a preposition and a noun or a determiner and a noun. We computed
the probability by semantic category. Let us consider the entries in Table I.
Regarding the verb lemma ‘look’ and the semantic tag object, the preposition
‘at’ has the highest probability to go in between according to Table II.

TABLE I:  Automatic lexicon extension. Sentence examples Sentence | After processing
---|---
I look the business. | look + EMPTY + object
She looks at the picture. | look + at + object
You look at the car. | look + at + object
She looks like her mum. | look + like + living thing
TABLE II:  Automatic lexicon extension probabilities by lemma verb and semantic category After processing | Probability
---|---
look + EMPTY + object | 0.33
look + at + object | 0.66
look + like + living thing | 1

In this way, the language model along with the semantic classification allow
us to infer the most suitable preposition after a verb by applying semantic
knowledge rather than by only considering morphological forms.

Listing 3 shows the aLexiE entry for verb ‘look’. Note how the language model
has learned to add the preposition ‘at’ when the verb is followed by an object
(semantically speaking) or ‘for’ in the case of foodstuff or a place. If
‘look’ is followed by a living thing, the system will add the preposition
‘like’. In the running example in this section, since ‘picture’ is tagged as
an object in aLexiE, and given the syntactic and semantic data in the ‘look’
entry; the system will insert preposition ‘at’ between the two words if
provided as input to the system.

###  III-B Syntactic structure supported by a grammar

In this section we describe the syntactic stage of our system. It performs
syntactic structuring with the Definite-Clause Grammar (dcg) [62]

Syntactic structuring, also called parsing, creates the tree structure of the
desired target sentence. We infer this structure by checking the syntactic
trees from the grammar for the input words. Obviously diverse possible
syntactic structures may result, depending on the roles of the input words in
the sentence. Fig. 1 shows the syntax trees from the grammar for the input
words ‘she’, ‘look’, ‘picture’; ‘She looks at the picture’ and ‘She looks the
picture’. Context-Sensitive Languages (csl) are created from this type of
grammar. The system picks the most appropriate trees for the input words given
the different possibilities within a grammar.

For the English case, we adapted the simple grammar with wide range of basic
sentences described in [4], by adding all its linguistics features, such as
adjectives preceding nouns. The system can parse sentences regardless of their
complexity. Sentence types may be affirmative, negative, interrogative (in
positive or negative form) or imperative (in positive or negative form),
including some of the following features: a nominal syntagm subject; a
coordinated nominal syntagm subject (compound subject); a nominal syntagm
direct complement; a coordinated nominal syntagm direct complement; an
indirect complement, and other place or time complements.

LISTING 3: Example of the English lemma ‘look’ in aLexiE.

⬇

<?xml version=”1.0” encoding=”UTF-8”

standalone=”no”?>

<lexicon>

<word>

<lemma>look</lemma>

<category>verb</category>

<present3s>looks</present3s>

<past>looked</past>

<present_participle>

looking

</present_participle>

<past_participle>

looked

</past_participle>

<object>at</object>

<foodstuff>for</foodstuff>

<living>like</living>

<place>for</place>

</word>

</lexicon>

Figure 2: Our two-stage NLG architecture

In our notation, upper case corresponds to tree structures and lower case to
word components. Fig. 1 illustrates some linguistic rules taken from the
grammar.

####  III-B1 Rules for nominal/coordinated nominal syntagms

Nominal syntagms (ns) contain nouns, pronouns or proper nouns. Determiners may
precede nouns. A coordinated syntagm is a sentence with two nominal syntagms
connected by a conjunction.

####  III-B2 Rules for adjectival/adverbial/prepositional syntagms

In this case, an adjectival or adverbial syntagm, which consists of an adverb
followed by an adjective or vice versa, may precede or follow a noun. Noun-
Noun modifiers such as in ‘car door’ are not considered. In prepositional
syntagms (ps), unless empty, a preposition precedes a nominal syntagm.
Prepositional syntagms just follow (never precede) a nominal syntagm.

####  III-B3 Predicate rule

The sentence predicate contains a verb that may be followed by a nominal
syntagm (coordinated or not). The verb may be accompanied by an
adjectival/adverbial syntagm. Note that a verb can be followed by two nominal
syntagms (yet not coordinated ones, that is, without a conjunction in
between), as in sentence ‘She gave me a cookie yesterday’.

####  III-B4 Sentence rule

Sentences are composed either of a nominal or coordinated nominal syntagm
(subject) and a predicate, or of a single predicate (without subject). The
latter is quite common in imperative English clauses. Given the relations
among syntagms, the depth level of our system is limited to two iterations to
reduce computational load. For example, in case a nominal syntagm includes a
prepositional syntagm, the second nominal syntagm cannot contain another
prepositional syntagm (to avoid recursion).

###  III-C Proposed NLG library: Sentence Planner and Surface Realiser

The input words for our nlg library should be meaningful, such as adjectives,
nouns, and verbs. The library can automatically infer the determiners,
conjunctions, and prepositions that complement those input words in the output
sentence.

Fig. 2 shows our two-stage architecture, an automatic nlg processing pipeline.
The user introduces the words (plus symbol ? for interrogative sentences) in
[subject, verb, object] (svo) order, which is not limiting in practice in many
domains. The first Sentence Planner stage performs lexicalisation, which adds
words and configures sentences. The second Surface Realiser stage, our main
contribution, introduces any extra elements that may be necessary and applies
morphology inflections to produce grammatically correct and coherent
sentences. Fig. 3 represents an example of generation using the library. Fig.
4 shows the flowchart of our nlg library. The main tasks are the following
ones:

Figure 3: Sentence generation with our two-stage NLG library Figure 4:  Flow
chart of the NLG procedure

####  III-C1 Detection of the linguistic structure (affirmative, negative or
interrogative) of the sentence (Sentence Planner)

The sentence is considered negative if one of the input words is the negation
adverb ‘not’. It is treated as interrogative if the input words include a
question mark. If both elements are present, a negated question is generated
by the system. The sentence is considered affirmative in any other case. This
is the case in our example in Fig. 3. The library also adds extra elements
corresponding to any linguistic realisations in the grammar and the knowledge
in aLexiE.

####  III-C2 Subject insertion (Sentence Planner)

Imperative sentences and other sentences with elided subjects are quite common
in English, for example ‘Go to your room’. We want the nlg process to be
almost transparent to end users and, thus, if the user does not provide a
subject, the library takes the personal pronoun ‘I’ as such. Besides, it
generates a second option with elided subject in case the user wants to create
an imperative clause. In our example in Fig. 3 and Fig. 4, the library does
not include a subject because the user introduces ‘she’ as input. This is
detected thanks to the grammar.

####  III-C3 Inference of syntax structure (Sentence Planner)

The separation between subject and predicate simplifies the identification of
the best syntactic trees for the input words, since they are smaller. Once the
Sentence Planner decides the type of sentence, the library sets the boundary
between the subject and the predicate taking the main verb as a reference and
then searches for the best syntactic structure that matches them. For this
purpose we follow a Depth-First Search (dfs) [63] in our grammar departing
from the input words. In case that some of the input words are not in the
aLexiE lexicon, they will be treated as proper nouns. This means no
inflections will be applied to them.

In our example in Fig. 1, the system infers two possible syntax structures
(options 1 and 2).

####  III-C4 Inclusion of extra elements (Sentence Planner)

Once the syntactic structure is decided, some extra elements such as
determiners, prepositions, and conjunctions may be necessary. These elements
are inserted in the sentence if they are associated to feasible grammar
realisations. In our example in Figs. 3 and 4, the library adds to the output
the extra elements that were inferred in the previous stage from the grammar.

####  III-C5 Morphological inflections (Realiser)

This encompasses the inflections that are necessary to produce a grammatically
correct sentence, in which the subject dictates the morphological conjugation,
person, number, and gender inflections of the verb and other components.

The library distinguishes between subject and predicate before generating a
sentence. In this regard, it can apply linguistic features to adjust person,
gender, and number, to create sentences with coordinate subject. For example,
given the input words ‘caregiver’, ‘I’, ‘eat’, ‘apples’, the subject of the
resulting sentence ‘The caregiver and I eat apples’ is compound.

First, person, gender, and number features must be derived from the input
words. The subject (expected to be a nominal or coordinated nominal syntagm)
determines them. Continuing with the running example ‘The caregiver and I eat
apples’, the subject is a coordinated nominal syntagm. The first nominal
sytagm within is composed of the determiner ‘the’ and the singular noun
‘caregiver’. The second is the pronoun ‘I’. Consequently, the person and
number of the sentence are first person and plural, and the verb ‘eat’ is
inflected accordingly.

TABLE III:  Results of our NLG library Input words | Best output sentence/s
---|---
something, be, not, right | Something isn’t right.
where, my, glasses, be, ? | Where are my glasses?
dinner, be, good, last, night | Dinner was good last night.
appreciate, your, help, concern | I appreciate your help and concern.
live, yellow, house | I live in the yellow house.
how much, stamps, be, these, days, ? | How much are stamps these days?
final, grades, be, available, after, class, today, ? | Are final grades available after class today?
TABLE IV:  aLexiE lemma and form extraction statistics | Initial | Extracted
---|---|---
| Lemmas | Forms | Lemmas | %Lemmas
EnLex | 508,000 | 695,000 | 212,021 | 41.74%
NIH | 505,145 | 955,564 | 67,660 | 13.39%
English-Freeling | 37,000 | 68,000 | 14,368 | 38.83%

By default, the first time the user introduces input words, the library takes
masculine gender, singular number and first person. Then, using aLexiE, it
adapts these features with grammar rules. For instance, if the subject is a
coordinated nominal syntagm, the output sentence is plural. Regarding gender,
the output sentence is only feminine if all subject components are
so111111Some gender-specific words, such as waiter and waitress, remain in
modern English.. The following rules are applied in strict order to adjust the
person feature: (1) if the subject has an element that refers to the first
person, the output sentence is adjusted to that person; (2) if the sentence
has an element that refers to the second person with no relation with the
first person, the output sentence is adjusted to second person; (3) finally,
if the sentence has an element that refers to the third person with no
relation with the first and second persons, the sentence remains in third
person.

aLexiE contains the number and gender inflections of all lemmas and the person
features of verbs and pronouns. Once these features are decided, they are
applied to all word inputs. However, in case the subject is missing, default
features must be set (as previously mentioned).

The verbal tense of the output sentence is present unless a time adverb or a
time adverbial locution (e.g. last week) are provided. For example, for the
adverb ‘yesterday’, the tense of the output sentence is past. This linguistic
information can also be found in aLexiE.

The library handles contraction spellings as well. We implemented those from a
freely accessible list121212Available at
https://en.wikipedia.org/wiki/Wikipedia: List_of_English_contractions, July
2019..

In case a word is missing in the lexicon, no related features are available or
they cannot be inflected, our library treats the word as a proper noun.

When generating a sentence, it is necessary to create its syntagms and join
them while respecting their syntactic and semantic function. For example, to
create ‘She looks at the picture’ from the input words ‘she’, ‘look’ and
‘picture’, it is necessary to generate the nominal syntagm the ‘picture’ and
integrate it into a prepositional syntagm as ‘at the picture’. It is also
necessary to build the subject of the sentence ‘she’ and a predicate with
‘look’ as the main verb. Finally, it is necessary to integrate in the output
sentence the subject and the predicate with the prepositional complement (‘she
looks at the picture’). All these stages are automatic, even the inclusion of
the preposition, from the syntactic and semantic information in aLexiE.

Table III shows examples of automatically generated sentences with increasing
linguistic complexity. Alongside each example we indicate the input words.

##  IV Experimental results

First, we compared the aLexiE lexicon with other lexica from the state-of-the-
art (Section IV-A). Then, we evaluated automatic English text expansion with
our system (Section IV-B), both directly (by regenerating corpus sentences)
and manually (from annotations) using a widely used corpus in the nlg field
(Section IV-B1). To the best of our knowledge, there is no other system for
automatic text expansion. Therefore, we have created the English version of a
corpus (Section IV-B2), to compare automatic text expansion in a multilingual
scenario.

###  IV-A Lexicon

There are several resources for statistical natural language processing and
corpus-based computational linguistics131313E.g.
https://nlp.stanford.edu/links/statnlp.html, July 2019.. The main differences
between aLexiE and those freely available online resources are coverage,
correctness and completeness of linguistic information (morphology, syntax,
and semantics).

TABLE V:  Semantic sources for entries tagged as nouns in aLexiE | MCR | FrameNet | Total
---|---|---|---
Present | Absent | Present | Absent | Present | Absent
Nouns | 69,784 | 8,182 | 25 | 8,157 | 69,809 | 8,157

Table IV shows the information we combined from the selected resources to
create aLexiE. According to [64], EnLex contains 508,000 unique lemmas,
corresponding to 695,000 unique forms. We only extracted some entries,
yielding 212,021 unique lemmas that correspond to 41.74% of the extracted
entries. nih contains 505,145 unique lemmas and 955,564 inflected forms. In
this case we only extracted 67,660 unique lemmas that correspond to 13.39% of
the lexicon. Specifically, we only extracted adjective, adverb, conjunction,
determiner, preposition, and pronoun entries, since other entries had no
associated morphological information. Freeling for English contains 37,000
unique lemmas, corresponding to 68,000 unique forms. We only extracted
adjectives, adverbs, and verbs (for the same reason as for nih), producing
14,368 unique lemmas corresponding to 38.83% of the original set.

Table V shows the sources for the semantic classification of the nouns in
aLexiE. We only searched in FrameNet141414Available at
https://framenet.icsi.berkeley.edu/ fndrupal, July 2019. the lemmas of the
nouns that were missing in mcr. Table VI shows the amount of lemmas after
merging and verification. Table VII shows the lexical categories of the lemmas
and forms in aLexiE. Most were tagged as nouns (77,966), yielding over 141,000
aLexiE inflected forms. Determiners and pronouns were revised manually to
include plural forms for the lemmas ‘this’ and ‘that’.

TABLE VI:  Merged and verified lemmas in aLexiE by lexical category Category | Merged | Verified
---|---|---
Noun | 81,380 | 77,966
Pronoun | 86 | 86
Verb | 11,602 | 10,903
Adjective | 47,078 | 28,971
Adverb | 10,353 | 1,867
Conjunction | 99 | 70
Determiner | 98 | 98
Preposition | 140 | 140
Total | 150,836 | 120,101
TABLE VII:  aLexiE lemmas and forms by lexical category Category | Lemmas | Forms
---|---|---
Noun | 77,966 | 141,580
Pronoun | 86 | 92
Verb | 10,903 | 39,493
Adjective | 28,971 | 28,971
Adverb | 1,867 | 1,867
Conjunction | 70 | 70
Determiner | 98 | 100
Preposition | 140 | 140
Total | 120,101 | 212,313

###  IV-B English text expansion

As previously said, we evaluated our system by extracting keywords from
sentences from a widely used corpus in the nlg field (Section IV-B1) and an
ad-hoc Spanish-English parallel corpus (Section IV-B2). We evaluated output
quality in terms of completeness, correctness and similarity to the original
sentence.

We decided to discard some common state-of-the-art metrics such as rouge [65]
and bleu [66], because they weakly reflect human assessment of nlg, as
discussed in [67].

####  IV-B1 AAC corpus

Even though our approach may be used for general nlg scenarios, we chose an
aac corpus151515Available at http://aactext.org, July 2019. for our first
evaluation due to the interest of aac as a representative real application.
Some aac tools such as Talk Together161616Available at
https://acecentre.org.uk/resources/talk- together, July 2019. and LetMe
Talk171717Available at http://www.utac.cat/descarregues/cace- utac, July 2019.
have small vocabulary packages with rigid interactions. None of them generates
messages taking morphological, syntactic, and semantic information into
consideration. The interest of nlg for aac is illustrated by several previous
works [68, 69, 70]. First we selected sentences without commas or hyphens, to
ensure that there was a single sentence/idea in a clause (our system could
handle multiple ideas as separate sentences). Since our system performs nlg
automatically, we selected sentences in present, past, and future tense
because these can be inferred by time adverbial complements. We then filtered
the result to obtain the main words (adjectives, adverbs, nouns, pronouns,
proper nouns, and verbs). Next we lemmatised all those words but the nouns and
pronouns. This was because if we lemmatised the latter, the system would have
no clue to generate a sentence with a plural noun or pronoun since the
features of these particular words are independent from other components of
the sentence (conversely, adjectives depend on the noun they modify). For this
purpose we used the Spacy syntactic parser. The resulting dataset had 1,869
English sentences and their main words.

##### Annotation

We introduced the main words of a target sentence into our automatic nlg
system and we studied the output sentences. In case of a full match between
the target and generated sentences, automatic generation was considered
totally successful. This happened for 1,315 sentences, 70.25% of the total.
The remaining 554 were manually inspected.

Of these, 15 differed only in few capital letters. This was due to errors in
the target and missing words in aLexiE that were treated as proper nouns. Our
system correctly replaced words by proper nouns in eight sentences, such as ‘I
need a new harry potter book’, which was generated as ‘I need a new Harry
Potter book’. Even though the matches were inexact, we consider these
sentences success cases rather than failures. In five sentences the system
failed to detect the lexical category of some input words due to missing data
in aLexiE, as in ‘I’m itchy’, which our system generated as ‘I’m Itchy’ (words
like itchy are neither present in aLexiE nor in mwd). We did not consider
these sentences failures because they were due to missing words in the
dictionary. Finally, there were two sentences containing words without an
aLexiE entry that indeed existed in mwd: ‘Need a bigger size’ was generated by
our system as ‘Need a Bigger size’, and ‘It is 2 o’clock’ was generated as ‘It
is 2 O’clock’. These were indeed failures of the system.

There were 22 sentences containing spelling mistakes in the target such as ‘I
have an appoinment with the doctor today’ (‘appoinment’ instead of
‘appointment’).

Consequently, our system was able to generate 13501818181315 + 8 + 5 + 22 =
1350 correct sentences automatically, corresponding to 72.23% of the total.

Finally, it was only necessary to evaluate 517 sentences manually. They were
revised by five nlg researchers from atlanTTic, University of Vigo, with
English skills equivalent to C1 in the Common European Framework of Reference
for Languages (cefr) or 95 or above in the Test of English as a Foreign
Language (toefl). Table VIII shows the annotation options.

TABLE VIII:  Annotation tags Feature | Values
---|---
Error type | morphological (a), syntactic (b),
lexicon (c), grammar (d), target (e),
lemmatiser (f)
Evaluation | 0-5
Best generation | optional
Generation suggestion | optional

Manual evaluation considered six error possibilities: morphological error (a),
syntactic error (b), aLexiE error (c), grammar error (d), target error (e),
and lemmatiser error (f) (Table VIII). The annotators also rated the quality
of the generation from 0 (not generated) to 5 (full match between target and
output)1919190 and 5 ratings were automatically treated.. Moreover, when the
system presented different alternative outputs the annotator had to choose
one. We noticed that, in some errors, except for svo order the system would
have succeeded in generating the targets. The annotators were requested to
provide output suggestions in that situation.

The annotation task took two months. We handed instructions and examples to
the annotators in advance to guarantee the consistency of the resulting
corpus. The tests exploited various features of English grammar such as
sentence type and constructions with different word categories. The annotation
script returned an xml file. Listing 4 shows an annotated sentence example.

The final results can be summarised as follows. Firstly, we must distinguish
the cases when our nlg system generated a single possibility from those with
several output sentences. In the first case the error type was set by majority
vote between the annotators. If the annotators did not agree, the sentence was
tagged with no consensus about error type. The final rating of each output
sentence was computed as the arithmetic average of annotator ratings. In the
second case, first we checked if there was consensus in the best realisation
field. Otherwise, the sentence was tagged with no consensus about best
realisation. If the annotators provided suggestions of best realisations and
there was a consensus about them, we tagged and rated the best output
candidate as in the first case. Table IX shows the distribution of the
annotations.

LISTING 4: Annotation example

⬇

<?xml version=”1.0” encoding=”UTF-8”?>

<TAGGING>

<CLAUSE>

<TARGET>Dropped my change.</TARGET>

<Generated_Clauses>

<Clause>

Drop my change.

<Error>b</Error>

<Rating>1</Rating>

</Clause>

<Clause>

I drop my change.

<Error>a</Error>

<Rating>2</Rating>

</Clause>

</Generated_Clauses>

<Best_realisation>2</Best_realisation>

<Suggestion_for_Generation>

I dropped my change yesterday.

</Suggestion_for_Generation>

</CLAUSE>

</TAGGING>

TABLE IX:  Distribution of annotations of our dataset Annotation result | Sentence number
---|---
No consensus about best realisation | 0
No consensus about error type but | 45
consensus about best realisation
Total consensus | 472

When the annotators agreed about error and best realisation, their average
rating indicates that the information in the target could be understood from
the generated sentence. This also happened when the annotators agreed about
the best realisation but there was no consensus about error type. The
annotators suggested 367 different alternative outputs, of which our library
generated 160 correctly and automatically (43.597%). The remaining 207
sentences (56.403%) were not manually inspected. We suppose that many of these
generated sentences might be considered appropriate as well.

The main mistakes were due to verbal tense adjustment, since many sentences
were in past tense but did not have any time-related complement. Another
common failure was adding a different preposition instead of the one in the
target (in some cases, however, this change did not modify the meaning of the
output sentence).

##### Evaluation agreement

Manual evaluation was monitored with two recognized agreement metrics that
yield robust estimations of the differences between annotators: Alpha-
reliability [71, 72] and accuracy.

When the annotators perfectly agree, $Alpha=1$. When their agreement seems by
chance $Alpha=0$. Obviously both extremes should be avoided.

Our evaluation focused on nominal data because we measured error annotation
agreement between five observers. As previously said, we computed the
agreement in error type and obtained the average rate. The first step was to
build a 5-observers-by-523-sentences reliability data matrix containing
$5\times 523$ values.

Table X shows that our system generated 523 sentences for the 517 target
sentences in the corpus. This was because there were several generated
candidates for some targets.

TABLE X: Reliability data matrix of the AAC annotated dataset considering the error types in Table VIII | Sentences
---|---
| 1 | 2 | … | 125 | … | 523
Annotator 1 | b | b | … | a | … | a
Annotator 2 | b | b | … | d | … | a
Annotator 3 | b | d | … | d | … | a
Annotator 4 | b | b | … | b | … | a
Annotator 5 | b | b | … | b | … | a

Next we tabulated the coincidence matrix in Table XI, in units. Coincidence
matrices take into account the values in a reliability data matrix. They
differ from contingency matrices in that the latter consider units in two
dimensions, not values. Our coincidence matrix accounted for all pairable
errors from the five annotators into a 6-by-6 square matrix, omitting
references to annotators. This type of matrix is symmetric with respect to its
main diagonal, which holds all perfect matches. Note that the coincidences
were counted twice in the coincidence matrix. Disagreements (represented by
off-diagonal cells) were also counted twice, yet in different cells.

TABLE XI: Coincidence matrix of our annotated dataset considering the error types in Table VIII | a | b | c | d | e | f
---|---|---|---|---|---|---
a | 698 | 12.5 | 3 | 46.75 | 39.5 | 8.25
b | 12.5 | 378 | 7.5 | 91 | 17.75 | 7.25
c | 3 | 7.5 | 37 | 10.75 | 3.5 | 14.25
d | 46.75 | 91 | 10.75 | 635.5 | 83.25 | 39.75
e | 39.5 | 17.75 | 3.5 | 83.25 | 55.5 | 18.5
f | 8.25 | 7.25 | 14.25 | 39.75 | 18.5 | 4

We then estimated inter-agreement accuracy between pairs of annotators. This
simply averages the proportions given by the diagonal of the coincidence
matrix. Note that it neither accounts for fortuitous (dis)agreement nor for
value ordering. The results for $Alpha$ and accuracy in Table XII are
promising. Tables XIII and XIV represent inter-agreement between pairs of
annotators [73, 74, 75].

TABLE XII:  Overall inter-annotator agreement Agreement measure | Value |  |  |
---|---|---|---|---
$Alpha$ | 0.582 |  |  |
Accuracy | 0.691 |  |  |
TABLE XIII: $Alpha$ between annotator pairs | Annotator 1 | Annotator 2 | Annotator 3 | Annotator 4 | Annotator 5
---|---|---|---|---|---
Annotator 1 | - | 0.726 | 0.485 | 0.675 | 0.649
Annotator 2 | 0.726 | - | 0.451 | 0.614 | 0.646
Annotator 3 | 0.485 | 0.451 | - | 0.508 | 0.447
Annotator 4 | 0.675 | 0.614 | 0.508 | - | 0.624
Annotator 5 | 0.649 | 0.646 | 0.447 | 0.624 | -
TABLE XIV: Accuracy between annotator pairs | Annotator 1 | Annotator 2 | Annotator 3 | Annotator 4 | Annotator 5
---|---|---|---|---|---
Annotator 1 | - | 0.803 | 0.608 | 0.759 | 0.751
Annotator 2 | 0.803 | - | 0.585 | 0.717 | 0.753
Annotator 3 | 0.608 | 0.585 | - | 0.621 | 0.587
Annotator 4 | 0.759 | 0.717 | 0.621 | - | 0.728
Annotator 5 | 0.751 | 0.753 | 0.587 | 0.728 | -

####  IV-B2 Spanish-English parallel corpus

We are not aware of the existence of other systems for automatic text
expansion as we have defined it. Therefore, we decided to apply our automatic
nlg system to Spanish and English and compare the results.

For this purpose, we manually created the English version of the Spanish
corpus used in [4]202020Available at https://www.gti.uvigo.es/images/manual_
evaluation_EN.txt, July 2019..

The final parallel corpus is composed of 948 English/Spanish sentences
covering various grammar features such as different sentence types and
constructions with different word categories.

Table XV shows a comparison between the English and Spanish versions in terms
of automatic generation using the parallel corpus. First, our system generated
613 English sentences automatically. The remaining 123 sentences were
inspected manually. We noticed that the most relevant mistakes (in 106
sentences) were due to Wikipedia training, since the system failed to add a
certain preposition in the target. Two of the other 17 sentences were actually
correct, since they only differed in some capital letters, and four had
differences in determiners that did not affect their meaning.

Second, our system was able to automatically generate 72 English sentences out
of the 212 that the system in [4] failed to generate in Spanish. The remaining
140 sentences were manually inspected. The most common mistakes were related
to verbal tense adjustment (64 sentences) and wrong prepositions (34
sentences).

Next we compared the approach in this paper with the system in [4]. We
correctly generated 77.64% and 72.26% of the Spanish and English sentences in
the parallel corpus, respectively.

The most important difference between the two languages was the use of
prepositions. In Spanish there were few mistakes of this kind, but they were
the most common in English. In our opinion this was due to the difficulty to
detect phrasal verbs.

TABLE XV: Comparison between the Spanish (ES) and English (EN) systems using the parallel corpus (automatic evaluation) |  | ES version
---|---|---
|  | Automatic | Manual | Total
EN version | Automatic | 613 | 72 | 685
Manual | 123 | 140 | 263
Total | 736 | 212 | 948

##  V Conclusions

We have developed an automatic hybrid system for English text expansion.
Relying on the aLexiE lexicon and our grammar, our system is able to perform
fully automatic text expansion from few input words. The integration of new
lexical resources for any language is simple. The architecture separates
domain-independent from domain-dependent components, so that the latter can be
substituted. We remark that the aLexiE lexicon and the grammar we have
developed for English expansion are relevant results in themselves. They could
be useful to other nlg researchers.

As far as we know this is the first fully automatic hybrid system for English
text expansion, combining a knowledge base of vocabulary and grammar
realisations with a statistical language model for preposition inference. Our
system has a good success rate when generating coherent and grammatically
correct sentences from user-selected input words.

The surface realiser relies on aLexiE and our grammar to take its decisions.
For this reason, we provide a detailed description of the procedure to create
them. As future work we plan to predict the best grammar realisation for input
words without svo order.

We have offered an insightful analysis of semantic similarities and
differences between target and output texts, and we have assessed system
performance using state-of-the-art metrics. First, we evaluated the system by
regenerating texts from an aac corpus. The system succeeded 72.23% of the
time. As future work we will evaluate the system with aac users taking
advantage of the broad community of our Pictodroid suite212121Available at
http://www.accegal.org/en/pictodroid, July 2019.. Second, we conducted
automatic text expansion in English and Spanish and compared the results. Due
to the lack of a multilingual dataset, we had to create a parallel corpus for
this purpose. Even though generation was slightly better for Spanish, thanks
to its more predictable preposition usage, performances were comparable.

As another future research line we plan to create similar systems for new
languages such as French and Portuguese.

## References

  * [1] E. Reiter and R. Dale, _Building Natural Language Generation Systems_. New York, NY, USA: Cambridge University Press, 2000.
  * [2] Y. Elsahar, S. Hu, K. Bouazza-Marouf, D. Kerr, and A. Mansor, “Augmentative and Alternative Communication (AAC) advances: A review of configurations for individuals with a speech disability,” _Sensors_ , vol. 19, no. 8, p. 1911, 2019.
  * [3] S. García-Méndez, M. Fernández-Gavilanes, E. Costa-Montenegro, J. Juncal-Martínez, and F. J. González-Castaño, “Automatic Natural Language Generation Applied to Alternative and Augmentative Communication for Online Video Content Services using SimpleNLG for Spanish,” in _Proceedings of the 15th Web for All Conference on The Future of Accessible Work_. ACM, 2018.
  * [4] S. García Méndez, M. Fernández Gavilanes, E. Costa Montenegro, J. Juncal Martínez, and F. J. González Castaño, “A library for automatic natural language generation of Spanish texts,” _Expert Systems with Applications_ , vol. 120, pp. 372–386, 2019.
  * [5] E. Reiter and R. Dale, “Building applied natural language generation systems,” _Nat. Lang. Eng._ , vol. 3, no. 1, pp. 57–87, Mar. 1997. [Online]. Available: http://dx.doi.org/10.1017/S1351324997001502
  * [6] R. Kutlak, C. Mellish, and K. van Deemter, “Content selection challenge - University of Aberdeen entry,” in _Proceedings of the 14th European Workshop on Natural Language Generation_. Sofia, Bulgaria: Association for Computational Linguistics, August 2013, pp. 208–209. [Online]. Available: http://www.aclweb.org/anthology/W13-2133
  * [7] F. Mairesse and M. Walker, “Personage: Personality generation for dialogue,” in _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , 2007, pp. 496–503.
  * [8] O. Dušek and F. Jurcicek, “Training a natural language generator from unaligned data,” in _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_ , vol. 1, 2015, pp. 451–461.
  * [9] J. A. Bateman, “Enabling technology for multilingual natural language generation: The KPML development environment,” _Nat. Lang. Eng._ , vol. 3, no. 1, pp. 15–55, Mar. 1997. [Online]. Available: http://dx.doi.org/10.1017/S1351324997001514
  * [10] S. Williams and E. Reiter, “Generating basic skills reports for low-skilled readers,” _Natural Language Engineering_ , vol. 14, no. 4, pp. 495–525, 2008.
  * [11] L. Wanner, B. Bohnet, N. Bouayad-Agha, F. Lareau, and D. Nicklaß, “Marquis: Generation of user-tailored multilingual air quality bulletins,” _Applied Artificial Intelligence_ , vol. 24, no. 10, pp. 914–952, 2010.
  * [12] G. Lampouras and A. Vlachos, “Imitation learning for language generation from unaligned data,” in _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , 2016, pp. 1101–1112.
  * [13] H. Mei, M. Bansal, and M. R. Walter, “What to talk about and how? Selective generation using LSTMS with coarse-to-fine alignment,” in _Proceedings of NAACL-HLT_ , 2016.
  * [14] R. Barzilay and M. Lapata, “Modeling local coherence: An entity-based approach,” _Computational Linguistics_ , vol. 34, no. 1, pp. 1–34, 2008.
  * [15] M. A. Walker, A. Stent, F. Mairesse, and R. Prasad, “Individual and domain adaptation in sentence planning for dialogue,” _Journal of Artificial Intelligence Research_ , vol. 30, pp. 413–456, 2007.
  * [16] R. Hervás and P. Gervás, “Case retrieval nets for heuristic lexicalization in natural language generation,” in _Portuguese Conference on Artificial Intelligence_. Springer, 2005, pp. 55–66.
  * [17] R. Dale, “Cooking up referring expressions,” in _Proceedings of the 27th annual meeting on Association for Computational Linguistics_. Association for Computational Linguistics, 1989, pp. 68–75.
  * [18] D. Robert, _Generating Referring Expressions: Constructing Descriptions in a Domain of Objects and Processes. ACL-MIT Series in Natural Language Processing_. MIT Press, 1992.
  * [19] M. C. Frank, N. D. Goodman, and J. B. Tenenbaum, “Using speakers’ referential intentions to model early cross-situational word learning,” _Psychological Science_ , vol. 20, no. 5, pp. 578–585, 2009.
  * [20] R. Dale and E. Reiter, “Computational interpretations of the gricean maxims in the generation of referring expressions,” _Cognitive Science_ , vol. 19, no. 2, pp. 233–263, 1995.
  * [21] M. W. Meteer, “Bridging the generation gap between text planning and linguistic realization,” _Computational Intelligence_ , vol. 7, no. 4, pp. 296–304, 1991.
  * [22] I. Langkilde and K. Knight, “Halogen statistical sentence generator,” in _Proceedings of the Association for Computational Linguistics’02 Demonstrations Session, Philadelphia_ , 2002, pp. 102–103.
  * [23] H. Nakanishi, Y. Miyao, and J. Tsujii, “Probabilistic models for disambiguation of an HPSG-based chart generator,” in _Proceedings of the Ninth International Workshop on Parsing Technology_. Association for Computational Linguistics, 2005, pp. 93–102.
  * [24] A. Cahill and J. Van Genabith, “Robust PCFG-based generation using automatically acquired LFG approximations,” in _Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, 2006, pp. 1033–1040.
  * [25] C. Gardent and S. Narayan, “Multiple adjunction in feature-based Tree-Adjoining Grammar,” _Computational Linguistics_ , vol. 41, no. 1, pp. 41–70, 2015.
  * [26] T.-H. Wen, M. Gasic, N. Mrksic, P.-H. Su, D. Vandyke, and S. Young, “Semantically conditioned LSTM-based natural language generation for spoken dialogue systems,” _arXiv preprint arXiv:1508.01745_ , 2015.
  * [27] A. Cheyer and D. Guzzoni, “Method and apparatus for building an intelligent automated assistant,” May 3 2007, US Patent App. 11/518,292. [Online]. Available: https://www.google.com/patents/US20070100790
  * [28] D. Mirkovic and L. Cavedon, “Dialogue management using scripts,” Feb. 27 2008, EP Patent App. EP20,060,759,358. [Online]. Available: https://encrypted.google.com/patents/EP1891625A1?cl=und
  * [29] E. Reiter, C. Mellish, and J. Levine, “Automatic-generation of technical documentation,” _Applied Artificial Intelligence_ , vol. 9, no. 3, pp. 259–287, 1995.
  * [30] R. Dale, S. Geldof, J.-P. Prost _et al._ , “Using natural language generation in automatic route description,” _Journal of Research and practice in Information Technology_ , vol. 37, no. 1, p. 89, 2005.
  * [31] W. P. Liu, B. Georgescu, S. K. Zhou, and D. Comaniciu, “Automatic generation of radiology reports from images and automatic rule out of images without findings,” Nov. 23 2017, US Patent App. 15/158,375.
  * [32] I. Mani and M. Maybury, “Automatic summarization,” _John Benjamin’s Publishing Co_ , 2001.
  * [33] E. Reiter, R. Robertson, and L. Osman, “Lessons from a Failure: Generating Tailored Smoking Cessation Letters,” _Artificial Intelligence_ , vol. 144, pp. 41–58, 2003.
  * [34] A. Fiedler, “Natural language proof explanation,” in _Mechanizing Mathematical Reasoning, Essays in Honor of Jörg H. Siekmann on the Occasion of His 60th Birthday_ , 2005, pp. 342–363. [Online]. Available: https://doi.org/10.1007/978-3-540-32254-2_20
  * [35] T. W. Morris, “Conversational agents for game-like virtual environments,” in _AAAI 2002 spring symposium on artificial intelligence and interactive entertainment_ , 2002, pp. 82–86.
  * [36] F. Peinado, P. Gervás, and B. Díaz-Agudo, “A description logic ontology for fairy tale generation,” in _In Forth Int. Conf. on Language Resources and Evaluation: Workshop on Language Resources for Linguistic Creativity_ , 2004, pp. 56–61.
  * [37] P. Gervás, _An Expert System for the Composition of Formal Spanish Poetry_. London: Springer London, 2001, pp. 19–32. [Online]. Available: https://doi.org/10.1007/978-1-4471-0275-5_2
  * [38] A. Stent, M. Marge, and M. Singhai, _Evaluating Evaluation Methods for Generation in the Presence of Variation_. Berlin, Heidelberg: Springer, 2005, pp. 341–351. [Online]. Available: https://doi.org/10.1007/978-3-540-30586-6_38
  * [39] A. Gatt and E. Reiter, “SimpleNLG: A realisation engine for practical applications,” in _Proceedings of the 12th European Workshop on Natural Language Generation_. Stroudsburg, PA, USA: Association for Computational Linguistics, 2009, pp. 90–93. [Online]. Available: http://dl.acm.org/citation.cfm?id=1610195.1610208
  * [40] M. Bollmann, “Adapting SimpleNLG to German,” in _Proceedings of the 13th European Workshop on Natural Language Generation_. Association for Computational Linguistics, 2011.
  * [41] P.-L. Vaudry and G. Lapalme, “Adapting SimpleNLG for bilingual English-French realisation,” in _Proceedings of the 14th European Workshop on Natural Language Generation_. Sofia, Bulgaria: Association for Computational Linguistics, August 2013.
  * [42] R. De Oliveira and S. Sripada, “Adapting SimpleNLG for Brazilian Portuguese realisation,” in _Proceedings of the 8th International Natural Language Generation Conference (INLG)_ , 2014.
  * [43] A. Mazzei, C. Battaglino, and C. Bosco, “SimpleNLG-IT: adapting SimpleNLG to Italian,” in _The 9th International Natural Language Generation Conference_ , 2016.
  * [44] S. García Méndez, “Natural language generation library for Spanish,” Master’s thesis, Telecommunication Engineering School, University of Vigo, 2017.
  * [45] A. Ramos Soto, J. Janeiro Gallardo, and A. Bugarín Diz, “Adapting SimpleNLG to Spanish,” in _Proceedings of the 10th International Conference on Natural Language Generation_ , 2017, pp. 144–148.
  * [46] R. De Jong and M. Theune, “Going Dutch: Creating SimpleNLG-NL,” in _Proceedings of The 11th International Natural Language Generation Conference_ , E. Krahmer, A. Gatt, and M. Goudbeek, Eds. United States: Association for Computational Linguistics (ACL), 11 2018, pp. 73–78.
  * [47] G. Chen, K. Van Deemter, and C. Lin, “SimpleNLG-ZH: a linguistic realisation engine for Mandarin,” in _Proceedings of The 11th International Natural Language Generation Conference_. ACL Anthology, 2018, pp. 57–66.
  * [48] A. C. Fuentes, A. R. Soto, and A. B. Diz, “Adapting SimpleNLG to Galician language,” in _Proceedings of the 11th International Conference on Natural Language Generation_ , 2018, pp. 67–72.
  * [49] I. Androutsopoulos, G. Lampouras, and D. Galanis, “Generating natural language descriptions from OWL ontologies: the NaturalOWL system,” _CoRR_ , vol. abs/1405.6164, 2014. [Online]. Available: http://arxiv.org/abs/1405.6164
  * [50] E. Reiter, S. Sripada, J. Hunter, and I. Davy, “Choosing words in computer-generated weather forecasts,” _Artificial Intelligence_ , vol. 167, pp. 137–169, 2005.
  * [51] A. Belz, “Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models,” _Nat. Lang. Eng._ , vol. 14, no. 4, pp. 431–455, Oct. 2008. [Online]. Available: http://dx.doi.org/10.1017/S1351324907004664
  * [52] J. Chen, S. Bangalore, O. Rambow, and M. A. Walker, “Towards automatic generation of natural language generation systems,” in _Proceedings of the 19th international Conference on Computational Linguistics-Volume 1_. Association for Computational Linguistics, 2002, pp. 1–7.
  * [53] P. Demasco and K. McCoy, “Generating text from compressed input: An intelligent interface for people with severe motor impairments,” _Commun. ACM_ , vol. 35, no. 5, pp. 68–78, May 1992. [Online]. Available: http://doi.acm.org/10.1145/129875.129881
  * [54] P. Demasco, K. McCoy, Y. Gong, C. Pennington, and C. Rowe, “Towards more intelligent AAC interfaces: The use of natural language processing,” in _Proceedings of the 12th Annual Conference_ , 1989, pp. 141–142.
  * [55] K. McCoy, P. Demasco, M. Jones, C. Pennington, and C. Rowe, “Applying natural language processing techniques to augmentative communication systems,” in _Proceedings of the 13th Conference on Computational Linguistics-Volume 3_. Association for Computational Linguistics, 1990, pp. 413–415.
  * [56] D. Crouch and T. H. King, “Unifying lexical resources,” in _Proc. of Interdisciplinary Workshop on the Identification and Representation of Verb Features and Verb Classes_ , 2005.
  * [57] S. Necsulescu, N. Bel Rafecas, M. Padró, M. Marimon, and E. Revilla, “Towards the automatic merging of language resources,” in _First International Workshop on Lexical Resources: an ESSLLI 2011 Workshop; Aug. 1-5 2011; Ljubljana; p. 70-77_. ESSLLI, 2011.
  * [58] N. Bel, M. Padró, and S. Necsulescu, “A method towards the fully automatic merging of lexical resources,” in _Proc. of Workshop on Language Resources, Technology and Services in the Sharing Paradigm_. ACL, Nov 12 2011, pp. 8–15.
  * [59] B. Sagot, “The Lefff, a freely available and large-coverage morphological and syntactic lexicon for French,” in _Proc. of the 7th International Conference on LREC_ , N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, and D. Tapias, Eds. Valletta, Malta: ELRA, 2010.
  * [60] L. Danlos and B. Sagot, “Constructions pronominales dans DICOVALENCE et le Lexique-Grammaire–intégration dans le Lefff,” in _Proc. of the 27th Lexicon-Grammar Conference_ , L’Aquila, Italy, 2008. [Online]. Available: https://hal.inria.fr/inria-00524741
  * [61] A. González-Agirre, E. Laparra, and G. Rigau, “Multilingual central repository version 3.0: upgrading a very large lexical knowledge base,” in _6th Global WordNet Conference_ , Matsue, Japan, 2012.
  * [62] E. Maggiori, “Desarrollo de una gramática para aserciones simples en español y su implementación en Prolog,” 2013.
  * [63] R. Tarjan, “Depth-first search and linear graph algorithms,” in _Switching and Automata Theory, 1971, 12th Annual Symposium on_. IEEE, 1971, pp. 114–121.
  * [64] B. Sagot, “A multilingual collection of CoNLL-U-compatible morphological lexicons,” in _Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018.
  * [65] C.-Y. Lin and E. Hovy, “Automatic evaluation of summaries using n-gram co-occurrence statistics,” in _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1_. Association for Computational Linguistics, 2003, pp. 71–78.
  * [66] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting on Association for Computational Linguistics_. Association for Computational Linguistics, 2002, pp. 311–318.
  * [67] J. Novikova, O. Dušek, A. C. Curry, and V. Rieser, “Why we need new evaluation metrics for NLG,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 2017.
  * [68] A. Copestake, “Applying natural language processing techniques to speech prostheses,” Working Notes of the 1996 AAAI Fall Symposium on Developing Assistive Technology for People with Disabilities, Tech. Rep., 1996.
  * [69] P. Moreda, E. Lloret _et al._ , “The use of natural language processing techniques to support health literacy: an evidence-based review,” 2015.
  * [70] C. Nikolopoulos, D. Kuester, M. Sheehan, S. Ramteke, A. Karmarkar, S. Thota, J. Kearney, C. Boirum, S. Bojedla, and A. Lee, “Robotic agents used to help teach social skills to children with autism: the third generation,” in _RO-MAN, 2011 IEEE_. IEEE, 2011, pp. 253–258.
  * [71] K. Krippendorff, _Content analysis: An introduction to its methodology_. Sage, 2012.
  * [72] K. Klaus, _Computing Krippendorff’s alpha-reliability_ , 2011.
  * [73] H. Dorussen, H. Lenz, and S. Blavoukos, “Assessing the reliability and validity of expert interviews,” _European Union Politics_ , vol. 6, no. 3, pp. 315–337, 2005.
  * [74] M. Poesio and R. Artstein, “The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account,” in _Proceedings of the workshop on frontiers in corpus annotations ii: Pie in the sky_. Association for Computational Linguistics, 2005, pp. 76–83.
  * [75] J. P. Pestian, P. Matykiewicz, M. Linn-Gust, B. South, O. Uzuner, J. Wiebe, K. B. Cohen, J. Hurdle, and C. Brew, “Sentiment analysis of suicide notes: A shared task,” _Biomedical Informatics Insights_ , vol. 5, pp. BII–S9042, 2012.