# Text Normalization using Memory Augmented Neural Networks

Subhojeet Pramanik<EMAIL_ADDRESS>Aman Hussain
<EMAIL_ADDRESS>VIT University, Vandaloor-Kelambakkam Road,
Chennai, Tamil Nadu, India

###### Abstract

We perform text normalization, i.e. the transformation of words from the
written to the spoken form, using a memory augmented neural network. With the
addition of dynamic memory access and storage mechanism, we present a neural
architecture that will serve as a language-agnostic text normalization system
while avoiding the kind of unacceptable errors made by the LSTM-based
recurrent neural networks. By successfully reducing the frequency of such
mistakes, we show that this novel architecture is indeed a better alternative.
Our proposed system requires significantly lesser amounts of data, training
time and compute resources. Additionally, we perform data up-sampling,
circumventing the data sparsity problem in some semiotic classes, to show that
sufficient examples in any particular class can improve the performance of our
text normalization system. Although a few occurrences of these errors still
remain in certain semiotic classes, we demonstrate that memory augmented
networks with meta-learning capabilities can open many doors to a superior
text normalization system.

###### keywords:

text normalization , differentiable neural computer , deep learning

††journal: Speech Communication

##  1 Introduction

The field of natural language processing has seen significant improvements
with the application of deep learning. However, there are many unsolved
challenges in NLP yet to be solved by the prevalent deep neural networks. One
of the simple but interesting challenges lies in designing a flawless text
normalization solution for Text to Speech and Automatic Speech Recognition
systems. Unlike many other problems being solved by neural networks, the
tolerance for unacceptable or "silly" errors in text normalization systems is
very low. Prevailing neural architectures in Sproat and Jaitly [2017] produce
near perfect overall accuracy on such a problem. But there is a caveat.
Whereas deciding whether a word needs to be normalized or not turns out to be
an easier problem since it mostly falls back on classifying the semiotic class
of the token, the actual challenge lies in generating the normalized form of
the token. However, such instances which require normalization are usually
very sparse which might result in high overall accuracy. Hence, these specific
cases must get the highest attention when evaluating the performance of any
text normalization system. Yet the existing models are prone to making "silly"
mistakes which are extremely non-trivial and mission-critical to TTS (Text-to-
Speech) and ASR (Automatic Speech Recognition) systems. Finite-state filters,
which perform simple algorithmic steps on the normalized tokens, are then used
to correct such errors and "guide" the model. Although developing such FST
grammars is a lot simpler than constructing a fully fledged finite-state text
normalization system, they do require some human expertise and domain
knowledge of the language involved.

We, therefore, ask if there is a way to circumvent the requirement of human
involvement and language expertise, and instead design a system that is
language agnostic and learns on its own given enough data. Although the most
commonly used neural networks are adept at sequence learning and sensory
processing, they are very limited in their ability to represent data
structures and learn algorithms on its own. There have been recent
advancements in memory augmented neural network architectures such as the
Differentiable Neural Computer, with dynamic memory access and storage
capacity. Such architectures have shown the ability to learn algorithmic tasks
such as traversing a graph or finding relations in a family tree.
Normalization of semiotic classes of interest, particularly those containing
numbers and measurement units, can be performed using basic algorithmic steps.
While neural network architectures such as LSTM work sufficiently well in
machine translation tasks, they have shown to suffer in these semiotic classes
which require basic step-by-step transduction of the input tokens [Sproat and
Jaitly, 2017]. There is hope that neural network architectures with memory
augmentation will be able to learn the algorithmic steps (meta-learning),
similar to the Finite-state filters, but without any human intervention or the
external knowledge about the language and its grammar. With such memory
augmentation, the network should be able to learn to represent and reason
about the sequence of characters in the context of the text normalization
task.

We begin by defining the challenges of text normalization to try to understand
the reason behind these "silly" mistakes. Then, after a brief overview of the
prior work done on this topic, we describe the dataset released by Sproat and
Jaitly [2017] which has been used in a way so as to allow for an objective
comparative analysis. Subsequently, we delve into the theoretical and
implementation details of our proposed system. Thereafter, the results of our
experiments are discussed. Finally, we find that memory augmented neural
networks are indeed able to do a good job with far lesser amounts of data,
time and resources. To make our work reproducible, open to scrutiny and
further development, we have open sourced a demonstration of our system
implemented using Tensorflow [Abadi et al., 2015] at
https://github.com/cognibit/Text-Normalization-Demo.

##  2 Text Normalization

Text normalization is the canonicalization of text from one or more possible
forms of representation to a ’standard’ or ’canonical’ form. This
transformation is used as a preliminary step in Text-to-Speech (TTS) systems
to render the textual data as a standard representation that can be converted
into the audio form. In Automatic Speech Recognition (ASR) systems, raw
textual data is processed into language models using text normalization
techniques.

For example, a native English speaker would read the following sentence Please
forward this mail to 312, Park Street, Kolkata as Please forward this mail to
three one two Park Street Kolkata. However, The new model is priced at $312
would be read as The new model is priced at three hundred and twelve dollars.
This clearly demonstrates that contextual information is of particular
importance during the conversion of written text to spoken form.

Further, the instances of actual conversion are few and far between swathes of
words which do not undergo any transformation at all. On the English dataset
used in this paper, 92.5% of words remain the same. The inherent data sparsity
of this problem makes training machine learning models especially difficult.
The quality requirements of a TTS system is rather high given the nature of
the task at hand. A model will be highly penalized for making "silly" errors
such as when transforming measurements from 100 KG to Hundred Kilobytes or
when transforming dates from 1/10/2017 to first of January twenty seventeen.

###  2.1 Prior Work

One of the earliest work done on the problem of text normalization was used in
the MITalk TTS system [Allen et al., 1987]. Further, a unified weighted
finite-state transducers based approach was proposed by Sproat [1996]. The
model serves as the text-analysis module of the multilingual Bell Labs TTS
system. An advancement in this field was made by looking at it as a language
modeling problem by Sproat et al. [2001]. For ASR systems requiring inverse
text normalization, data-driven approaches have been proposed by Pusateri et
al. [2017]. The latest development in this problem space has been by Sproat
and Jaitly [2017]. Even though a recurrent neural network model trained on the
corpus results in very high overall accuracies, it remains prone to making
misleading predictions such as predicting completely inaccurate dates or
currencies. Such "silly" errors are unacceptable in a TTS system deployed in
production. However, a few of these errors were shown to be corrected by a FST
(Finite State Transducer) which employs a weak covering grammar to filter and
correct the misreadings.

###  2.2 Dataset

For the purposes of a comparative study and quantitative interpretation, we
have used the exact same English and Russian dataset as used in Sproat and
Jaitly [2017]. The English dataset consists of 1.1 billion words extracted
from Wikipedia and run through Google’s TTS system’s Kestrel text
normalization system to generate the target verbalizations. The dataset is
formatted into ’before’ or unprocessed tokens and ’after’ or normalized
tokens. Each token is labeled with its respective semiotic class 111ALL = all
cases; PLAIN = ordinary word (¡self¿); PUNCT = punctuation (sil); TRANS =
transliteration; LETTERS = letter sequence; CARDINAL = cardinal number;
VERBATIM = verbatim reading of character sequence; ORDINAL = ordinal number;
DECIMAL = decimal fraction; ELECTRONIC = electronic address; DIGIT = digit
sequence; MONEY = currency amount; FRACTION = non-decimal fraction; TIME =
time expression such as PUNCT for punctuations and PLAIN for ordinary words.
The Russian dataset consists of 290 million words from Wikipedia and is
formatted likewise. These datasets are available at
https://github.com/rwsproat/text-normalization-data.

Each of these datasets are split into 100 files. The base paper (Sproat and
Jaitly [2017]) uses 90 of these files as the training set, 5 files for the
validation set and 5 for the testing set. However, our proposed system only
uses the first two files of the English dataset (2.2%) and the first four
files of the Russian dataset (4.4%) for the training set. To keep the results
consistent and draw objective conclusions, we have defined the test set to be
precisely the same as the one used by the base paper. Hence, the first 100,002
and 100,007 lines are extracted from the 100th file output-00099-of-00100 of
the English and Russian dataset respectively.

##  3 Background: Memory Augmented Neural Networks

Traditional deep neural networks are great at fuzzy pattern matching, however
they do not generalize well on complex data structures such as graphs and
trees, and also perform poorly in learning representations over long
sequences. To tackle sequential forms of data, Recurrent Neural Networks were
proposed which have been known to capture temporal patterns in an input
sequence and also known to be Turing complete if wired properly [Siegelmann
and Sontag, 1995]. However, traditional RNNs suffer from what is known as the
vanishing gradients problem [Bengio et al., 1994]. A Long Short-Term Memory
architecture was proposed in [Hochreiter and Schmidhuber, 1997] capable of
learning over long sequences by storing representations of the input data as a
cell state vector. LSTM can be trained on variable length input-output
sequences by training two separate LSTM’s called the encoder and decoder
[Sutskever et al., 2014]. The Encoder LSTM is trained to map the input
sequence to a fixed length vector and Decoder LSTM generates output vectors
from the fixed length vector. This kind of sequence to sequence learning
approach have been known to outperform traditional DNN models in machine
translation and sequence classification tasks. Extra information can be
provided to the decoder by using attention mechanisms [Bahdanau et al., 2014]
and [Luong et al., 2015], which allows the decoder to concentrate on the parts
of the input that seem relevant at a particular decoding step. Such models are
widely used and have helped to achieve state of the art accuracy in machine
translation systems.

However, LSTM based sequence-to-sequence models are still not good at
representing complex data structures or learning to perform algorithmic tasks.
They also require a lot of training data to generalize well to long sequences.
An interesting approach by Joulin and Mikolov presents a recurrent
architecture with a differentiable stack, able to perform algorithmic tasks
such as counting and memorization of input sequences. Similar memory augmented
neural network [Grefenstette et al., 2015] have also shown to benefit in
natural language transduction problems by being able to learn the underlying
generating algorithms required for the transduction process. Further, a memory
augmented neural network architecture called the Neural Turing Machine was
introduced by Graves et al. that uses an external memory matrix with read and
write heads. A Controller network that works like an RNN is able read and
write information from the memory. The read and write heads use content and
location-based attention mechanisms to focus the attention on specific parts
of the memory. NTM has also shown promise in meta-learning [Santoro et al.,
2016] showing that memory augmented networks are able to generalize well to
even lesser training examples.

An improvement to this architecture was proposed by Graves et al. called the
Differentiable Neural Computer having even more memory access mechanisms and
dynamic storage capabilities. DNC, when trained in a supervised manner, was
able to store representations of input data as “variables” and then read those
representations from the memory to answer synthetic questions from the BaBI
dataset [Weston et al., 2015]. DNC was also able to solve algorithmic tasks
such as traversing a graph or inferring from a family tree, showing that it is
able to process structured data in a manner that is not possible in
traditional neural networks. Dynamic memory access allows DNC to process
longer sentences and moreover, extra memory can be added anytime without
retraining the whole network.

###  3.1 Differentiable Neural Computer

A basic DNC architecture consists of a controller network, which is usually a
recurrent network coupled with an external memory matrix
$M\in\mathbb{R}^{N\times W}$. At each timestep $t$, the controller network
takes as input a controller input vector
$\chi_{t}=[x_{t};r_{t-1}^{1},\dots,r_{t-1}^{R}]$, where
$x_{t}\in\mathbb{R}^{X}$ is the input vector for the time-step $t$ and
$r_{t-1}^{1},\dots,r_{t-1}^{R}$ is a set of $R$ read vectors from the previous
time step and outputs an output vector $v_{t}$ and interface vector
$\varepsilon_{t}\in\mathbb{R}^{(W\times R)+3W+5R+3}$. The controller network
is essentially a recurrent neural network such as the LSTM. The recurrent
operation of the controller network can be encapsulated as in Eqn 1:

$(v_{t},\varepsilon_{t})=\mathcal{N}([\chi_{1};\dots;\chi_{t};\theta])$ (1)

where $\mathcal{N}$ is a non-linear function, $\theta$ contains the all
trainable parameters in the controller network. The read vector $r$ is used to
perform read operation at every time step. The read vector $r$ defines a
weighted sum over all memory locations for a memory matrix $M$ by applying a
read weighting $w^{r}\in\Delta_{N}$ over memory $M$. $\Delta_{N}$ is the non
negative orthant of $\mathbb{R}^{N}$ with the unit simplex as a boundary.

$r=\sum_{i=1}^{N}{M[i,\cdot]w^{r}[i]}$ (2)

where the ‘·’ denotes all $j=1,\dots,W$. The interface vector
$\varepsilon_{t}$ is used to parameterize memory interactions for the next
time step. A write operation is also performed at each time-step using a write
weighting $w^{w}\in\Delta_{N}$ which first erases unused information from the
memory using an erase vector $e$ and writes relevant information to the memory
using the write vector $v_{t}$. The overall write operation can be formalized
as in Eqn 3.

$M_{t}=M_{t-1}\circ(E-w_{t}^{w}e_{t}^{\top})+w_{t}^{w}v_{t}^{\top}$ (3)

where $\circ$ denotes element-wise multiplication and $E$ is an $N\times W$
matrix of ones. The final output of the controller network
$y_{t}\in\mathbb{R}^{Y}$ is obtained by multiplying the concatenation of the
current read vector $r_{t}$ and output vector $v_{t}$ with a $RW\times Y$
dimensional weight matrix $W_{r}$.

$y_{t}=v_{t}+W_{r}[r_{t}^{1};\dots;r_{t}^{R}]$ (4)

The system uses a combination of different attention mechanisms to determine
where to read and write at every time-step. The attention mechanisms are all
parameterized by the interface vector $\varepsilon_{t}$. The write weighting
$w^{w}$, used to perform the write operation, is defined by a combination of
content-based addressing and dynamic memory allocation. The read weighting
$w^{r}$ is defined by a combination of content-based addressing and temporal
memory linkage. The entire system is end-to-end differentiable and can be
trained through backpropagation. For the purpose of this research, the
internal architecture of the Differentiable Neural Computer remains the same
as specified in the original paper [Graves et al., 2016]. The open-source
implementation of the DNC architecture used here is available at
https://github.com/deepmind/dnc.

###  3.2 Extreme Gradient Boosting

Boosting is an ensemble machine learning technique which attempts to pool the
expertise of several learning models to form a better learner. Adaptive
Boosting, or more commonly known as "AdaBoost", was the first successful
boosting algorithm invented by Freund and Schapire. [Breiman, 1998] and
[Breiman, 1999] went on to formulate the boosting algorithm of AdaBoost as a
kind of gradient descent with a special loss function. [Friedman et al.,
2000], [Friedman, 2001] further generalized AdaBoost to gradient boosting in
order to handle a variety of loss functions.

Gradient Boosting has proven to be a practical, powerful and effective machine
learning algorithm. Gradient boosting and deep neural networks are the two
learning models that are widely recognized and used by the competitive machine
learning community at Kaggle and elsewhere. Tree boosting as implemented by
XGBoost [Chen and Guestrin, 2016] gives state-of-the-art results on a wide
range of problems from different domains. The most defining factor for the
success of XGBoost is its scalability. It has been reported to run more than
ten times faster than existing solutions on a single node. The reason behind
using gradient boosting for classification first and a normalization model
later is two-fold. First, we solve the data sparsity problem by having the
deep neural network train only on the tokens which need normalization. Second,
we turn to the incredible scalability and efficiency of the XGBoost model to
condense huge amounts of data which makes it more amenable to quick training
and experimental iterations.

Now, we give a brief overview of the gradient boosting algorithm and establish
the intuition necessary to understand the proposed solution for text
normalization. Essentially, gradient boosting adds and fits weak learners in a
sequential manner to rectify the defects of the existing weak learners. In
adaptive boosting, these "defects" are defined by assigning higher penalty
weights to the misclassified data points in order to restrain the new learner
from making the previous mistakes again. Similarly, in gradient boosting, the
"defects" are defined by the error gradients.

The model is initiated with a weak learner $F(x_{i})$, which is a decision
stump i.e. a shallow decision tree. The subsequent steps keep adding a new
learner, $h(x)$, which is trained to predict the error residual of the
previous learner. Therefore, it aims to learn a sequence of models which
continuously tries to correct the residuals of the earlier model. The sum of
predictions is increasingly accurate and the ensemble model increasingly
complex.

To elucidate further, we consider a simple regression problem. Initially a
regression model $F(x_{1})$ is fitted to the original data points:
$(x_{1},y_{1}),(x_{2},y_{2}),\cdots,(x_{n},y_{n})$. The error in the model
prediction $y_{i}-F(x_{i})$ is called the error residual. Now, we fit a new
regression model $h(x)$ to data :
$(x_{1},y_{1}-F(x_{1})),(x_{2},y_{2}-F(x_{2})),...,(x_{n},y_{n}-F(x_{n}))$
where $F(x)$ is the earlier model and $h$ is the new model to be added to
$F(x)$ such that it corrects the error residuals $y_{i}-F(x_{i})$.

$F(x_{i}):=F(x_{i})+h(x_{i})$ (5) $F(x_{i}):=F(x_{i})+y_{i}-F(x_{i})$ (6)
$F(x_{i}):=F(x_{i})-1\frac{\delta J}{\delta F(x_{i})}$ (7)

This is quite similar to the method of gradient descent which tries to
minimize a function by moving in the opposite direction of the gradient:

$\theta_{i}:=\theta_{i}-\rho\frac{\delta J}{\delta\theta_{i}}$ (8)

where $\rho$ is the learning rate. However from a more applied perspective,
the model of choice for implementing the weak learners in the XGBoost library
[Chen and Guestrin, 2016] are decision tree ensembles which consist of a set
of classification and regression trees. Further implementation details are
discussed in the following section.

##  4 Proposed Architecture

We propose a two-step architecture for text normalization. For a given token
$w_{i}$ which is to be normalized, the token $w_{i}$ and some context words
$w_{i-k}$ to $w_{i+k}$ are first fed as characters into an XGBoost
classification model, where $k$ is the number of context words. The XGBoost
classification model is trained to predict whether a particular word is to be
normalized or not.

For those words which require normalization, a second model is used. We
propose a novel sequence-to-sequence architecture based on Differentiable
Neural Computer. This second model uses the Encoder Decoder architecture
[Sutskever et al., 2014] combined with Badhanau attention mechanism [Bahdanau
et al., 2014]. The model tries to maximize the conditional probability
$P(y\mathbin{|}x)$ where $y$ is the target sentence and $x$ is a sequence of
characters formed by the concatenation of the to-be-normalized token $w$ and
context words $w_{i-k}$ to $w_{i+k}$ surrounding the token.

The major intuition behind using two different models is that the instances of
actual conversion are few and far between. Training a single deep neural
network with such a heavily skewed dataset is incredibly difficult. Separating
the task of predicting whether a word needs to be normalized or not and
predicting the normalized sequence of words helps us increase the overall
accuracy of the normalization pipeline. Both models being independent of each
other are trained separately. The first model tries to maximize the
classification accuracy of predicting whether a word requires normalization.
The second model tries to minimize the softmax cross-entropy of the sequence
of predicted words averaged over all the time-steps.

###  4.1 XGBoost Classifier

An extreme gradient boosting machine (XGBoost) model is trained to classify
tokens into the following two classes: RemainSame and ToBeNormalized, to be
used in the later stage of the pipeline. Tokenization of the training data has
already been performed. Additional preprocessing of the tokens or words needs
to be done before the XGBoost model can be trained on it. Specifically, we
transform the individual tokens into numerical feature vectors to be fed into
the model. Each token is encoded as a vector comprised of the Unicode (UTF-8)
values of its individual characters. We limit the length of the vector at 30;
all characters beyond this are left out. In case of shorter words, the
remaining vector is filled or padded with $0$. Contextual information is
incorporated into the input feature vector by prepending the preceding token
and appending the succeeding token to the target input token. To demarcate the
boundaries between the consecutive tokens, we use the $-1$ integer value.
Since the starting position of the target input token can vary, this value
serves as the ’start of token’ identifier. An example can better elucidate the
process. For instance, the input vector for ’genus’ in the sentence :
Brillantaisia is a genus of plant in family Acanthaceae will be: [ -1, 97, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, -1, 103, 101, 110, 117, 115, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1, 111, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1].

After the data is preprocessed and ready, we perform a train-validation split
to help us tune the model. The performance metric we have used is AUC or area
under the curve. A top-down approach to hyperparameter tuning is employed. We
begin with a high learning rate and determine the best number of estimators or
trees which is the most important hyperparameter for the model along with the
learning rate. We find the best number of estimators to be 361. Then we go on
to tune the tree-specific parameters such as the maximum depth of the decision
stumps, the minimum child weight and the gamma value. Once we have a decent
model at hand, we start tuning the regularization parameters to get better or
similar performance at a reduced model complexity. Finally, we have an AUC
score of 0.999875 in the training set and a score of 0.998830 on the
validation set.

Figure 1: Top 10 important features for the English XGBoost model

The XGBoost package by [Chen and Guestrin, 2016] allows us to rank the
relative importance of the features for the classification task by looking at
the improvement in the accuracy brought about by any particular feature. On
generating the feature importance plot of the trained English XGBoost model in
Figure 1, we find out that the first six characters of the target token, i.e.
feature at the 32nd, 33rd, 34th, 35th, 36th and 37th position of the feature
vector, had the highest score. For a human classifying these tokens, the first
few characters of the word or token are indeed the best indicators to decide
whether it needs to be normalized or not. This assures us that the trained
model has in fact learned the right features for the task at hand. After these
first few features, the model also places high importance on other characters
belonging to the preceding and succeeding token. The high F1-score of the
model as reported in Table 3 confirms the overall effectiveness of the model.

The model could also be trained to classify the tokens into semiotic classes.
The semiotic classes which most confuses the DNC translator could be fed to a
separate sequence-to-sequence model which is exclusively trained on those
error-prone classes. Another direction to go from here would be to increase
the size of the context window during the data preprocessing stage to feed
even more contextual information into the model.

###  4.2 Sequence to Sequence DNC

The ToBeNormalized tokens, as classified by the XGBoost model, are then fed to
a recurrent model. For this end, we present an architecture called sequence-
to-sequence DNC that allows the DNC model to be adapted for sequence-to-
sequence translation purposes. Our underlying framework uses the RNN Encoder-
Decoder architecture [Sutskever et al., 2014]. We have also used attention
mechanisms to allow the decoder to concentrate on the various different output
states generated during the encoding phase. One major contribution of this
paper is to replace bidirectional LSTM used in a Neural Machine Translation
system with a single unidirectional DNC. During the encoding phase, the DNC
reads an input sequence of vectors $x=(x_{1},\dots,x_{T_{x}})$;
$x_{t}\in\mathbb{R}^{K_{x}}$, and outputs a sequence of annotation vectors
$h=(h_{1},\dots,h_{T_{x}})$; $h_{t}\in\mathbb{R}^{n}$, where $K_{x}$ is the
input vocabulary size and $T_{x}$ is the number of input tokens.

$h_{t}=g_{e}(x_{t},h_{t-1},s_{t})$ (9)

where function $g_{e}$ gives the output of the DNC network during the encoding
phase, and $s_{t}$ is the hidden state of the DNC. During the decoding phase
the DNC is trained to generate an output word $y_{t}\in\mathbb{R}^{K_{y}}$,
given a context vector $c_{t}\in\mathbb{R}^{n}$, where $K_{y}$ is the output
vocabulary size. The decoding phase uses Bahdanau attention mechanism
[Bahdanau et al., 2014] to generate a context vector $c_{t}$ by performing
soft attention over the annotation vectors $h$.

The Decoder defines a conditional probability $P$ of an output word $y_{t}$ at
time step $t$ given sequence of input vectors $x$ and previous predictions
$y_{1},\dots,y_{t-1}$.

$P(y_{t}\mathbin{|}y_{1},\dots,y_{t-1},x)=g_{d}(y_{t-1},s_{t},c_{t})$ (10)

where $g_{d}$ gives the output of the DNC network during the decoding phase,
and $s_{t}$ is the hidden state of the of the DNC at timestep $t$ computed by

$s_{t}=f(s_{t-1},y_{t-1},c_{t})$ (11)

where $f$ calculates the new state of the DNC network based on the previous
controller and memory states. During the decoding phase, the output of the DNC
is fed into a dense layer followed by a soft-max layer to generate word-by-
word predictions. We also used embedding layers to encode the input and output
tokens into fixed dimensional vectors during the encoding and decoding phases.

Figure 2: Sequence to sequence DNC, encoding phase. Figure 3: Sequence to
sequence DNC, decoding phase.

A DNC uses a $N\times W$ dimensional memory matrix for storing state
information compared to a single cell state in an LSTM. The presence of an
external memory allows the DNC to store representations of the input data in
its memory matrix using write heads and then read the representations from the
memory using read heads. Dynamic memory allocation helps the network to encode
large input sequences while retaining the inherent structure in those
sequences. The content and location based attention mechanisms give the
network more information about the input data during decoding. The ability to
read and write from memory helps the network in meta-learning to store richer
representations of the input data. We found out that network was able to
generalize faster compared to LSTM with a low number of training examples.
This is probably due to the fact that DNC is able to store complex quasi-
regular structure embedded in the input data sequences in its memory and then
later, it is able to infer from these representations.

We also found that feeding the context vector $c$ during the decoding stage
was necessary for convergence. Probably, the context vector provides the
network with more information about which locations to focus during each
decoding step. The annotations used for generating the context vector stores
information about the states of the DNC during the entire encoding phase.

For the purpose of text normalization, we feed the ToBeNormalized tokens in a
manner specified in Experiment 2 of [Sproat and Jaitly, 2017]. The
ToBeNormalized token is placed in between 3 context words to the left and
right, with a distinctive tag marking the to-be-normalized word. This is then
fed as a sequence of characters into the input embedding layer during the
Encoding stage. For the sentence The city is 15kms away from here, in order to
normalize the token 15km the input becomes

city is <norm> 15km </norm> away from

where <norm> and </norm> are tags that mark the beginning and end of the to-
be-normalized token. The output is always a sequence of words. During decoding
phase output tokens are first fed into an output embedding layer before
feeding it to the decoder. For the above example, the output becomes

fifteen kilometers

##  5 Experimental Results

The initial XGBoost classification layer gives an F1-score of 0.96 in for
English and 1.00 for Russian. The classifiers’ performance in terms of
precision and recall of the two classes is reported in Table 2 and 3. The DNC
model in the second layer was trained for 200k steps on a single GPU system
with a batch-size of 16 until the perplexity reached 1.02. The parameter
values used for training the second layer are given in Table 1 The overall
results of entire system are reported in Table 4 & 5. It can be seen that the
overall performance of the model in terms of the F1-score is quite good. The
Russian model reported an accuracy of $99.3\%$ whereas the English reported an
accuracy of $99.4\%$. As mentioned before the instances of actual conversion
are quite a few, which is the reason for the high overall accuracy. However,
when analyzing a text normalization system, it is more important to look into
the kind of errors it makes. A single metric such as accuracy or BLEU
(Bilingual Evaluation Understudy) score is not sufficient for comparison. It
is not much of a problem if a token from the DATE class: 2012 is normalized as
two thousand twelve, instead of twenty twelve. However, we certainly would not
want it to be translated to something like twenty thirteen. These ’silly’
errors are subjective by their very nature and thus, rely on a human reader.
This makes the analysis of these kinds of errors difficult but important. One
has to take a look at all the cases where the model produces completely
unacceptable predictions.

Table 1: sequence-to-sequence DNC experimental settings. $K_{x}$: input vocabulary size, $K_{y}$: output vocabulary size, $R$: number of read heads | $K_{x}$ | $K_{y}$ | Memory size ($N\times W$) | $R$ | Controller hidden units | Embedding size
---|---|---|---|---|---|---
English | 168 | 1781 | $256\times 64$ | 5 | 1024 | 32
Russian | 222 | 2578 | $256\times 64$ | 5 | 1024 | 32
Table 2: Classification Report for XGBoost Russian model | Precision | Recall | F1 score
---|---|---|---
RemainSelf | 1.00 | 1.00 | 1.00
ToBeNormalized | 0.99 | 1.00 | 1.00
Table 3: Classification Report for XGBoost English model | Precision | Recall | F1 score
---|---|---|---
RemainSelf | 1.00 | 1.00 | 1.00
ToBeNormalized | 0.94 | 0.99 | 0.96
Table 4: Comparison of accuracies over the various semiotic classes of interest on the English data-set. base accuracy: accuracy of the LSTM based sequence-to-sequence model proposed in [Sproat and Jaitly, 2017]. accuracy: accuracy of the proposed, XGBoost + sequence-to-sequence DNC model. | semiotic-class | base count | count | base accuracy | accuracy
---|---|---|---|---|---
0 | ALL | 92416 | 92451 | 0.997 | 0.994
1 | PLAIN | 68029 | 67894 | 0.998 | 0.994
2 | PUNCT | 17726 | 17746 | 1.000 | 0.999
3 | DATE | 2808 | 2832 | 0.999 | 0.997
4 | LETTERS | 1404 | 1409 | 0.971 | 0.971
5 | CARDINAL | 1067 | 1037 | 0.989 | 0.994
6 | VERBATIM | 894 | 1001 | 0.980 | 0.994
7 | MEASURE | 142 | 142 | 0.986 | 0.971
8 | ORDINAL | 103 | 103 | 0.971 | 0.980
9 | DECIMAL | 89 | 92 | 1.000 | 0.989
10 | DIGIT | 37 | 44 | 0.865 | 0.795
11 | MONEY | 36 | 37 | 0.972 | 0.973
12 | FRACTION | 13 | 16 | 0.923 | 0.688
13 | TIME | 8 | 8 | 0.750 | 0.750
Table 5: Comparison of accuracies over the various semiotic classes of interest on the Russian data-set. The headings are same as in Table 4. | semiotic-class | base count | count | base accuracy | accuracy
---|---|---|---|---|---
0 | ALL | 93184 | 93196 | 0.993 | 0.993
1 | PLAIN | 60747 | 64764 | 0.999 | 0.995
2 | PUNCT | 20263 | 20264 | 1.000 | 0.999
3 | DATE | 1495 | 1495 | 0.976 | 0.973
4 | LETTERS | 1839 | 1840 | 0.991 | 0.991
5 | CARDINAL | 2387 | 2388 | 0.940 | 0.942
6 | VERBATIM | 1298 | 1344 | 1.000 | 0.999
7 | MEASURE | 409 | 411 | 0.883 | 0.898
8 | ORDINAL | 427 | 427 | 0.956 | 0.946
9 | DECIMAL | 60 | 60 | 0.867 | 0.900
10 | DIGIT | 16 | 16 | 1.000 | 1.000
11 | MONEY | 19 | 19 | 0.842 | 0.894
12 | FRACTION | 23 | 23 | 0.826 | 0.609
13 | TIME | 8 | 8 | 0.750 | 0.750

It can be seen that the class-wise accuracies of the model are quite similar
to the base LSTM model. Upon analyzing the nature of error made in each class,
it was identified that the model performs quite well in classes: DIGIT, DATE,
ORDINAL and TIME. The errors reported in these classes are shown in Table 6.
Most of the errors in these classes are due to the fact that the DNC is
confused with the true context of the token. For example, the token 1968 in
DATE context is predicted as if in CARDINAL context. However, the DNC never
makes a completely unacceptable prediction in these classes for both English
and Russian data-sets as can be observed in Table 6. For readers unfamiliar
with the Russian language, we look at the issue with 22 июля. This error stems
from confusion in grammatical cases which do not exist in the English language
and are replaced with prepositions. However in the Russian language
prepositions are used along with grammatical cases, but may also be omitted in
many situations. Now, the 22nd is transformed to двадцать второго
(transliterated into Latin script as "dvadtsat vtorOGO") whereas when used
with the preposition of, such as of the 22nd, it is transformed to двадцать
второе (transliterated into Latin script as "dvadtsat vtorOE"). The baseline
LSTM based sequence-to-sequence architecture proposed in Experiment 2 of
Sproat and Jaitly [2017] showed completely unacceptable errors such as a DATE
11/10/2008 normalized as the tenth of october two thousand eight. On the other
hand the DNC network never makes these kind of ’silly’ errors in these
classes. This suggests that the DNC network can, in fact, be used as a text
normalization solution for these classes. This is an improvement over the
baseline LSTM model in terms of the quality of prediction.

The DNC network, however, suffers in some classes: MEASURE, FRACTION, MONEY
and CARDINAL, similar to the baseline LSTM network. The errors reported in
these classes are shown in Table 7. All the unacceptable mistakes in cardinals
occur in large numbers greater than a million. The DNC sometimes also
struggles with getting the measurement units and denominations right. For non-
Russian readers, we illustrate this with the MONEY token $1m where the
prediction is completely off; since  одиннадцать долларов сэ ш а means "eleven
US dollars" but один миллион долларов сэ шэ а means "1 million US dollars". In
terms of overall accuracy, the DNC performs slightly better in MEASURE, MONEY
and CARDINAL. The model, however, performs much worse in FRACTION compared to
the baseline model. The DNC network still makes unacceptable ’silly’ mistakes
in these classes such as predicting completely inaccurate digits and units,
which is not enough to make it a trustworthy system for these classes.

Table 6: Errors in which the DNC network is confused with the context of the token. | input | semiotic-class |  prediction |  truth |
---|---|---|---|---|---
0 | 2007 | DIGIT |  two thousand seven |  two o o seven |
1 | 1968 | DATE |  one thousand nine hundred sixty eight |  nineteen sixty eight |
2 | 0:02:01 | TIME |  zero hours two minutes and one seconds |  zero hours two minutes and one second |
3 | 22 июля | DATE |  двадцать второго июля |  двадцать второе июля |
4 | II | ORDINAL |  два |  второй |
Table 7: Errors in which the DNC network makes completely unacceptable predictions. | input | semiotic-class |  prediction |  truth |
---|---|---|---|---|---
0 | 14356007 | CARNINAL |  one million four hundred thirty five thousand six hundred seven |  fourteen million three hundred fifty six thousand seven |
1 | 0.001251 g/cm3 | MEASURE |  zero point o o one two five one sil g per hour |  zero point o o one two five one grams per c c |
2 | 88.5 million HRK | MONEY |  eighty eight point five million yen |  eighty eight point five million croatian kunas |
3 | 10/618,543 | FRACTION |  ten sixteenth sixty one thousand five hundred forty three |  ten six hundred eighteen thousand five hundred forty thirds |
4 | 15 м/с | MEASURE |  пятнадцать сантиметров в секунду |  пятнадцати метров в секунду |
5 | $1m | MONEY |  одиннадцать долларов сэ ш а |  один миллион долларов сэ ш а |

In order to understand why the model performs so well in some classes but
suffers in others, we proceeded to find the frequency of these specific tokens
in the English training dataset. The training set has 17,712 instances of
dates of the form $xx/yy/zzzz$. As reported in the earlier section, the model
made zero unacceptable mistakes in these DATE tokens. The baseline LSTM,
however, still reported unacceptable errors for dates of the similar form. On
the other hand, measurement units such as $mA$, $g/cm3$ and $ch$ occur less
than ten times in the training set. Compared to other measurement units, $kg$
and $cm$ are present more than $200$ times in the training set. CARDINAL has
$273,111$ tokens out of which only 1,941 are numbers which are larger than a
million. Besides, the error in MONEY for the English data-set was for the
denomination that occurred only once in the training set. The results in Table
7 clearly demonstrate that the model suffers only in the tokens for which a
sufficient number of examples are not available in the training set. The DNC
network never made any unacceptable prediction for examples that were
sufficiently present. This hints at the notion that the model performs poorly
particularly for units, cardinals and denominations which occur a lesser
number of times in the training set. Unlike the baseline LSTM, the model is
reasonable and durable to examples which are sufficiently present.

###  5.1 Ablation Study

Table 8: Class-wise accuracy comparison with and without DNC memory | semiotic-class | accuracy without memory | accuracy with memory
---|---|---|---
0 | ALL | 0.940704 | 0.994181
1 | CARDINAL | 0.345227 | 0.991321
2 | DATE | 0.186794 | 0.996469
3 | DECIMAL | 0.021739 | 1.000000
4 | DIGIT | 0.068182 | 0.818182
5 | FRACTION | 0.000000 | 0.687500
6 | LETTERS | 0.101490 | 0.971611
7 | MEASURE | 0.021127 | 0.985915
8 | MONEY | 0.054054 | 0.972973
9 | ORDINAL | 0.077670 | 0.980583
10 | PLAIN | 0.992444 | 0.993858
11 | PUNCT | 0.998873 | 0.998873
12 | TIME | 0.125000 | 0.750000
13 | VERBATIM | 0.812188 | 0.995005

To make the case for memory augmentation in neural networks to perform text
normalization, we conducted an ablation experiment to factor out its
contribution if any. We know that the DNC model consists of a controller
network, equipped with various memory access mechanisms to read and write from
a memory matrix. During the process of training, the controller network is
intended to learn to use the provided memory access mechanisms instead of just
relying on its internal LSTM state. This is important to make the most out of
the benefits that come from memory augmentation. The DNC controller network at
each time-step receives a set of $R$ read vectors as input. These read vectors
or memory activations are obtained by performing a read operation on the
memory matrix. Our ablation experiment intends to verify the contributions of
these memory activations for prediction. We use an existing model pre-trained
in the previous section and perform inference on the test data on two
conditions, with memory activations and without memory activations. The first
condition is just inference on the test data without any modification to the
DNC activations. For the second condition, we zero out the read vectors from
the memory at each time-step before providing it as input to the controller
network. If the DNC learns to use the provided memory access mechanisms during
the training process (i,e, the prediction is largely dependent on the value of
the read and write vectors), zeroing out the read vectors during inference
should have a significant impact in the performance of the model. However, if
the model is just learning to predict based on its internal LSTM controller
state, zeroing out the read vectors during inference should have a minute
impact in performance. As can be seen from the comparison in Table
LABEL:table:ablation_table, there is a significant drop in accuracy for most
semiotic classes when memory structures are inaccessible to the controller
network. Particularly, semiotic classes which require structured processing of
the input tokens such as, DATE, TIME, DIGIT, MEASURE, MONEY, and TELEPHONE,
show a significant reduction in performance. Upon analyzing the kind of errors
that the DNC network makes when memory structures are removed, it was found
that the DNC network gets the translation context correct in most cases. For
example, for the token 1984, the prediction of the DNC network without memory
is nineteen thousand two hundred eighty one. The model starts the translation
correctly, but however, it fails mid-way by predicting a completely incorrect
digit. This is indicative of our prior assumption that the model learns to
write the input tokens in its memory matrix during the encoding stage and
later reads from the memory during the decoding stage. If the memory
structures would not have been used by the network for translation, we should
have seen, essentially, no drop in performance on removing them. Apparently,
the read vectors have high feature importance in performing a successful
prediction during inference.

###  5.2 Results on up-sampled training set

Table 9: Predictions corrected after up-sampling for the English data-set. | token | semiotic-class |  prediction before up-sampling |  prediction after up-sampling |
---|---|---|---|---|---
0 | 295 ch | MEASURE |  two hundred ninety five hours |  two hundred ninety five chains |
1 | 2 mA | MEASURE |  two a m |  two milli amperes |
2 | 0.001251g/cm3 | MEASURE |  zero point o o one two five one sil g per hour |  zero point o o one two five one sil g per c c |
3 | 1/2 cc | MEASURE |  one _letter d_letter a_letter s_letter h_letter _letter t_letter w_letter o_letter _letter o_letter |  one half c c |
4 | 14356007 | CARDINAL |  one million four hundred thirty five thousand six hundred seven |  fourteen million three hundred fifty six thousand seven |

The initial results lead us to a follow-up question. Can our system perform
better given better quality data? Will a simple up-sampling procedure on the
rare kinds of tokens improve the model? To test our hypothesis of whether
sufficient examples can improve the performance in certain semiotic classes,
we up-sampled the distribution of those specific tokens which occurred less
than a particular threshold frequency. The up-sampling was done with
duplication only for MEASURE and CARDINAL on the English dataset. Sentences
which had measurement units which occurred less than 100 times were up-sampled
to have 100 instances each in the training set. Out of 253 measurement units,
229 of them occurred less than a 100 times in the entire training set.
Similarly, sentences with Cardinals with value larger than a million were up-
sampled to 10,000 instances. The final distribution of the training set was
59,439 for MEASURE and 299,694 for CARDINAL. The model was then retrained for
the same number of training iterations with the up-sampled data. The overall
accuracies and number of unacceptable errors for MEASURE & CARDINAL after up-
sampling are shown in Table 10. The comparison of the predictions is shown in
Table 9. Overall, it is very interesting to see that using simple data
augmentation techniques like up-sampling helped remove all the unacceptable
errors in MEASURE and reduced the number of unacceptable errors from three to
two in CARDINAL. Such an elementary technique even removed errors in rare
measurement units such as $ch$ and $g/cc$. However, the improvement observed
in CARDINAL was rather modest. And as expected, the number of unacceptable
errors in other classes were unaffected. This clearly provides evidence to the
initial assumption that our system improves, even if marginally, when a
sufficient number of examples are produced for any particular instance type.
Nonetheless, we can safely say that this system looks promising and worthy of
widespread adoption.

Table 10: Accuracies and no. of unacceptable errors before and after up-sampling for the English data-set. a1: accuracy before up-sampling, a2: accuracy after up-sampling, e1: no. of unacceptable errors before up-sampling, e2: no. of unacceptable errors after up-sampling. | semiotic-class | a1 | a2 | e1 | e2
---|---|---|---|---|---
0 | MEASURE | 0.971 | 0.986 | 4 | 0
1 | CARDINAL | 0.994 | 0.991 | 3 | 2

##  6 Discussion

Given the reduced number of unacceptable predictions in most semiotic classes
we can say that the quality of the predictions produced by DNC are much better
than the baseline LSTM model. The reason why DNC works better than LSTM might
be due to the presence of a memory matrix and read-write heads. The read-write
heads allows the DNC to store richer representations of the data in its
dynamic memory matrix. Research done by Santoro et al., shows memory augmented
neural networks have the ability to generalize well to less number of training
examples. The model never made any unacceptable prediction in some classes
(DIGIT, DATE, ORDINAL and TIME) for the same test set used by Sproat and
Jaitly. With a basic augmentation technique and minimal human requirement the
number of unacceptable errors in MEASURE was reduced to zero. The LSTM model
reported in [Sproat and Jaitly, 2017] reported unacceptable errors even when
sufficient examples are present. On the other hand, DNC is quite resistant to
errors when sufficient examples are present. However, the DNC is still prone
to making unacceptable predictions in some classes (FRACTION, MONEY and
CARDINAL) which makes it still risky as a standalone text normalization
system. There is still a lot of work to be done before a purely deep learning
based algorithm can be used a standalone component of a TTS system. We believe
the performance the model can be further improved by designing a more balanced
training set.

Apart from the domain of text normalization, we also provide evidence that a
sequence-to-sequence architecture made with DNC can be successfully trained
for tasks similar to machine translation systems. Until now DNC has only been
used for solving simple algorithmic tasks and have not been applied to real-
time production environments. The quality of the results produced by DNC in
text normalization demonstrates it is, in fact, a viable alternative to LSTM
based models. LSTM based architectures usually require large amounts of
training data. The results in Sproat and Jaitly [2017] show that the LSTM
based seq-to-seq models can sometimes produce a weird output even when
sufficient examples are present. For instance, LSTM’s did not work well in
predicting DATE, TIME and DIGIT, even though the training set had a lot of
examples from the category. DNC, on the other hand, is able to generalize well
and avoids these kinds of errors. We believe that the DNC architecture should
give good results in designing NMT models for languages which do not have a
lot of training data available. It is also important to note that a single
unidirectional DNC provides much better generalization compared to the stacked
bidirectional LSTM used by Sproat and Jaitly, proving that memory augmented
neural networks can provide much better results with significantly reduced
training times and fewer data points. The LSTM model reported in their paper
was trained on 8 parallel GPUs for about five and half days (460k steps). On
the contrary, our model was trained on a single GPU system for two days (200k
steps). Furthermore, our model used only 2.2% of the English data and 4.4% of
the Russian data for training.

##  7 Conclusion

Therefore, we can safely arrive at the conclusion that memory augmented neural
networks such as the DNC are in fact a promising alternative to LSTM based
models for a language agnostic text normalization system. Additionally, the
proposed system requires significantly lesser amounts of data, training
duration and compute resources. Our DNC model has reduced the number of
unacceptable errors to zero for some classes with basic up-sampling of rare
data points. However, there are still classes where the performance needs to
be improved before an exclusively deep learning based model can become the
text normalization component of a TTS system. Besides, we have also
demonstrated a system that can be used to train sequence-to-sequence models
using a DNC cell as the recurrent unit.

## Acknowledgements

We would like to show our gratitude to Richard Sproat (Senior Research
Scientist at Research & Machine Intelligence, Google, New York) for his
insights and comments that greatly improved the manuscript. We thank Kaggle
for hosting the Text Normalization Challenge by Richard Sproat and Kyle Gorman
which got us interested in this problem in the first place. We are also very
grateful to Google DeepMind for open sourcing their implementation of the
Differentiable Neural Computer which was a requirement for this research.

## References

## References

  * Abadi et al. [2015] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X., 2015.  TensorFlow: Large-scale machine learning on heterogeneous systems.  URL: https://www.tensorflow.org/. software available from tensorflow.org.
  * Allen et al. [1987] Allen, J., Hunnicutt, S.M., Klatt, D., 1987.  From Text to Speech: The MITalk System.  Cambridge University Press.
  * Bahdanau et al. [2014] Bahdanau, D., Cho, K., Bengio, Y., 2014.  Neural machine translation by jointly learning to align and translate.  CoRR abs/1409.0473.  URL: http://arxiv.org/abs/1409.0473, arXiv:1409.0473.
  * Bengio et al. [1994] Bengio, Y., Simard, P., Frasconi, P., 1994.  Learning long-term dependencies with gradient descent is difficult.  IEEE Transactions on Neural Networks 5, 157--166.  URL: http://dx.doi.org/10.1109/72.279181, doi:10.1109/72.279181.
  * Breiman [1998] Breiman, L., 1998.  Arcing classifier (with discussion and a rejoinder by the author).  Ann. Statist. 26, 801--849.  URL: https://doi.org/10.1214/aos/1024691079, doi:10.1214/aos/1024691079.
  * Breiman [1999] Breiman, L., 1999.  Prediction games and arcing algorithms.  Neural Computation 11, 1493--1517.  URL: http://dx.doi.org/10.1162/089976699300016106, doi:10.1162/089976699300016106.
  * Chen and Guestrin [2016] Chen, T., Guestrin, C., 2016\.  Xgboost: A scalable tree boosting system.  CoRR abs/1603.02754.  URL: http://arxiv.org/abs/1603.02754, arXiv:1603.02754.
  * Freund and Schapire [1999] Freund, Y., Schapire, R.E., 1999\.  A short introduction to boosting.
  * Friedman et al. [2000] Friedman, J., Hastie, T., Tibshirani, R., 2000.  Special invited paper. additive logistic regression: A statistical view of boosting.  The Annals of Statistics 28, 337--374.  URL: http://www.jstor.org/stable/2674028.
  * Friedman [2001] Friedman, J.H., 2001.  Greedy function approximation: A gradient boosting machine.  Ann. Statist. 29, 1189--1232.  URL: https://doi.org/10.1214/aos/1013203451, doi:10.1214/aos/1013203451.
  * Graves et al. [2014] Graves, A., Wayne, G., Danihelka, I., 2014.  Neural turing machines.  URL: http://arxiv.org/abs/1410.5401, arXiv:1410.5401.
  * Graves et al. [2016] Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Agapiou, J., Badia, A.P., Hermann, K.M., Zwols, Y., Ostrovski, G., Cain, A., King, H., Summerfield, C., Blunsom, P., Kavukcuoglu, K., Hassabis, D., 2016\.  Hybrid computing using a neural network with dynamic external memory.  Nature 538, 471+.  URL: http://dx.doi.org/10.1038/nature20101, doi:10.1038/nature20101.
  * Grefenstette et al. [2015] Grefenstette, E., Hermann, K.M., Suleyman, M., Blunsom, P., 2015\.  Learning to transduce with unbounded memory.  CoRR abs/1506.02516.  URL: http://arxiv.org/abs/1506.02516, arXiv:1506.02516.
  * Hochreiter and Schmidhuber [1997] Hochreiter, S., Schmidhuber, J., 1997\.  Long Short-Term memory.  Neural Comput. 9, 1735--1780.  URL: http://dx.doi.org/10.1162/neco.1997.9.8.1735, doi:10.1162/neco.1997.9.8.1735.
  * Joulin and Mikolov [2015] Joulin, A., Mikolov, T., 2015\.  Inferring algorithmic patterns with Stack-Augmented recurrent nets.  CoRR abs/1503.01007.  URL: http://arxiv.org/abs/1503.01007, arXiv:1503.01007.
  * Luong et al. [2015] Luong, M.T., Pham, H., Manning, C.D., 2015.  Effective approaches to attention-based neural machine translation.  URL: http://arxiv.org/abs/1508.04025, arXiv:1508.04025.
  * Pusateri et al. [2017] Pusateri, E., Ambati, B.R., Brooks, E., Platek, O., McAllaster, D., Nagesha, V., 2017\.  A mostly Data-Driven approach to inverse text normalization, in: Proc. Interspeech 2017, pp. 2784--2788.  URL: http://dx.doi.org/10.21437/Interspeech.2017-1274, doi:10.21437/Interspeech.2017-1274.
  * Santoro et al. [2016] Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., Lillicrap, T., 2016.  One-shot learning with Memory-Augmented neural networks.  URL: http://arxiv.org/abs/1605.06065, arXiv:1605.06065.
  * Siegelmann and Sontag [1995] Siegelmann, H.T., Sontag, E.D., 1995\.  On the computational power of neural nets.  J. Comput. Syst. Sci. 50, 132--150.  URL: http://dx.doi.org/10.1006/jcss.1995.1013, doi:10.1006/jcss.1995.1013.
  * Sproat [1996] Sproat, R., 1996.  Multilingual text analysis for text-to-speech synthesis, in: Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on.  URL: http://dx.doi.org/10.1109/ICSLP.1996.607867, doi:10.1109/ICSLP.1996.607867.
  * Sproat et al. [2001] Sproat, R., Black, A.W., Chen, S.F., Kumar, S., Ostendorf, M., Richards, C., 2001\.  Normalization of non-standard words.  Computer Speech & Language 15, 287--333.
  * Sproat and Jaitly [2017] Sproat, R., Jaitly, N., 2017\.  RNN approaches to text normalization: A challenge.  URL: http://arxiv.org/abs/1611.00068, arXiv:1611.00068.
  * Sutskever et al. [2014] Sutskever, I., Vinyals, O., Le, Q.V., 2014.  Sequence to sequence learning with neural networks, in: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems 27. Curran Associates, Inc., pp. 3104--3112.  URL: http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf.
  * Weston et al. [2015] Weston, J., Bordes, A., Chopra, S., Rush, A.M., van Merriënboer, B., Joulin, A., Mikolov, T., 2015.  Towards AI-complete question answering: A set of prerequisite toy tasks.  URL: http://arxiv.org/abs/1502.05698, arXiv:1502.05698.


# Why white noise is not enough?
On using radio front-end models while designing 6G PHY

Pawel Kryszkiewicz, Pawel Sroka, Marcin Hoffmann, Marcin Wachowiak

###### Abstract

From generation to generation there are increasing requirements for wireless
standards both in terms of spectral and energy efficiency. While up to now the
layered wireless transceiver architecture worked allowing for, e.g.,
separation of channel decoding algorithms from front-end design, this may need
reconsideration in the 6G era. Especially the hardware-originated distortions
have to be taken into account while designing other layer algorithms as the
high throughput and energy efficiency requirements will push these devices to
their limit revealing their nonlinear characteristics. This position paper
will shed some light on new degrees of freedom while cross-layer designing and
controlling multicarrier and multiantenna transceivers of 6G systems.

###### Index Terms:

6G, front-end nonlinearity, cross-layer design

##  I Introduction

Increasing number of users and applications results in more demanding
requirements on wireless networks. This can be met with the 5G network
employing simultaneous densification of base stations, spectral efficiency
improvement techniques, and an increase in the utilized bandwidth. However,
this comes at the cost of increased energy consumption of wireless
transceivers resulting, e.g., from a higher number of radio front-ends
(antennas) and more complex signal processing both at the transmitter and the
receiver side. Therefore, the next, 6G technology will need new design
paradigms to reflect the rising needs and new constraints. Up to now, the
effort was on obtaining maximal spectral efficiency for a ”perfect” radio
front-end. However, while pushing the transceiver design into its spectral and
energy consumption limits, new phenomena become significant and need to be
taken into account. This can be analyzed using a typical wireless transmitter
diagram in Fig. 1. In most cases each of the depicted blocks, e.g., modulator
or battery, is designed independently. The dashed line shows typical splits
between design domains. Most researchers, e.g., in the field of radio
resources management, assume perfect and linear modeling of electronic
components constituting the radio front-end, e.g., Power Amplifiers (PAs). In
reality, a linear increase of mean transmit power results in a linear increase
of consumed power and no signal distortion only when a high back-off
(difference between clipping level and mean transmit power) is used.
Unfortunately, this front-end model cannot be used for designing highly
energy-efficient systems. For this case of a wireless system operating close
to its energy efficiency maximum, a nonlinear front-end characteristic and its
influence on many system metrics have to be considered, e.g., for a given PA
the increased transmission power results in increased self-interference power
generated both in the transmission band and outside the transmission band,
widening the system bandwidth and possibly decreasing spectral efficiency[1].
At the same time, the transmitted signal, as a result of PA nonlinear
characteristics, has its distribution changed, resulting in a nonlinear
increase in power consumption[2]. These problems with nonlinear behavior can
be extended to some other components of the wireless transceiver, e.g.,
Analog-Digital Converters (ADCs) or batteries that power the front-end. As
such, nonlinear front-end models should be considered while designing specific
solutions for wireless transceivers and systems. This paper presents a couple
of new approaches that can be utilized in multiple components of a future 6G
system.

Figure 1: Diagram of a typical wireless transmitter. Dashed lines show typical
splits between design domains.

##  II Optimization of PA operating point for the OFDM signal

First, the operating point of a PA transmitting Orthogonal Frequency Division
Multiplexing (OFDM) waveform can be optimized in order to maximize the total
link spectral or energy efficiency[3]. Commonly, the operating point is
adjusted in order to achieve a given spectral emission mask (for out-of-band
frequency region) or Error Vector Magnitude (EVM) (in the in-band region) at
the transmitter output. However, if the operating point of the PA is adjusted
in order to maximize the link’s spectral efficiency, significant gains in
comparison to contemporary solutions are possible. Most importantly, this
requires precise modeling of the transmitted waveform (in most cases its
amplitude distribution is enough) and the front-end nonlinearity in addition
to the standard phenomena like multipath fading or interference observed at
the receiver. If the goal function changes to maximization of energy
efficiency, in addition, models of energy consumption are needed. This
obviously depends on a given implementation, e.g., PA class, or utilized
modules, e.g., what kind of coding is used. While energy efficiency is the
most important aspect for battery-powered devices, battery models are
typically omitted while designing communications systems. In reality, a
battery can reveal significant nonlinear characteristics, e.g., the so-called
”rate-capacity” effect causes two times higher instantaneous power required to
power the power amplifier, which results in a more than two-fold increase in
the energy drained from the battery. These all models are to be combined
together to find the optimal operating point of the PA.

An example solution, based on mathematical framework provided in [3], is shown
in Fig. 2. It is assumed that the transmitter uses soft-limiter, class B PA.
The operating point of the PA, i.e., IBO, is optimized in order to achieve
maximal spectral efficiency (SE) or energy efficiency (EE). Moreover, two
different battery models are considered. The maximal unclipped PA output is
assumed to be 100 mW and the constant power consumption of other electronic
components in the transmitter is set to 10 mW. The $SNR_{\mathrm{SAT}}$ is
maximal signal-to-noise ration achievable over given link, i.e., when
transmitting a single carrier of power 100 mW. As expected the SE maximization
achieves the highest Signal to Noise and Distortion Ratio (SNDR) in the whole
SNR range. Most importantly, the optimal IBO value changes significantly
depending on the channel conditions. If the EE is to be optimized there is no
point in achieving so high SNDR (as it results in higher energy consumption)
resulting both in different SNDR and optimal IBO curves. Finally, it is
visible that a battery model should be considered while optimizing EE of a
transmitter as it has influence both on SNDR and optimal IBO values.

Figure 2: Optimal values of Signal to Noise and Distortion Ratio (SNDR) and
Input Back-Off (IBO) as a function of saturation Signal-to-Noise Ratio (SNR)
while Spectral Efficiency (SE) or Energy Efficiency (EE) maximization for two
battery models.

##  III Front-end characteristics enabling optimization of non-Gaussian
waveform

While in most OFDM applications a large number of occupied subcarriers is
considered central limit theorem can be used to model instantaneous OFDM
signal amplitude using Rayleigh distribution, e.g., in the use case mentioned
above. However, if a small number of subcarriers is used, e.g., in the uplink
from some sensor devices, the above-mentioned distribution is not valid
anymore. Such a system, relying on orthogonal frequency-division multiple
access (OFDMA) is considered in future 5G and 6G systems in the scope of
massive machine-type communications (mMTC), where large numbers of simple
devices can connect to a base station to transmit small amounts of data. The
optimization of the PA operating point together with the number of utilized
subcarriers, influencing complex envelope power variation, constitutes
additional degrees of freedom in such a sensor network optimization. The more
subcarriers are used the higher instantaneous power variations are expected,
resulting in a higher amount of nonlinear distortion. At the same time, a
wider spectrum allows typically for higher throughput. As such optimization of
the operating point for such a scenario is not a trivial task. Moreover, the
frequency-domain characteristic of nonlinear distortion can be used while
designing subcarriers allocation procedures. While the OFDM nature guarantees
that subcarriers are orthogonal, the spectral regrowth caused by nonlinear PA
results in users utilizing adjacent frequency channels causing distortion to
one another. This noise-like effect will appear contributing to the multi-user
interference, with the interference level depending on the distance in
frequency from the subcarriers used by the interferer. From the resource
allocation point of view, such additional interference will play a significant
role in the allocation of subcarriers to different users, as it may
significantly degrade the quality of transmission in adjacent sets of
resources. Thus, alternatives in the form of reducing the number of allocated
subcarriers, which typically results in lower distortion level or improved
link budget, or introducing some null subcarriers in the form of guard bands,
should be considered.

As an example we show in Fig. 3 the optimal, i.e., maximizing data rate, Input
Back Off (IBO) of the PA as a function of link’s Signal to Noise Ratio (SNR)
and number of utilized subcarriers (from 1 to 12). The clipper/soft-limiter PA
model is used. IBO is a ratio between the maximal unclipped input power of the
amplifier and the mean input signal power. For the single carrier transmission
it is reasonable to transmit signal with maximal possible power without
introducing any distortion, i.e., 0 dB IBO as the signal has constant
envelope. For higher number of subcarriers it is visible that in general
optimal IBO is the higher the higher is the SNR. Usage of low IBO in this case
will result in the higher wanted signal power but significantly increased
distortion power at the receiver. Most importantly, the curves differ between
different number of subcarriers showing there is a new degree of freedom in
optimization of IoT devices.

Figure 3: Optimal operation point of the PA (in terms of data rate
maximization) vs. the expected SNR with low number of subcarriers used in OFDM
transmission.

##  IV Front-end aware Massive MIMO system

Massive MIMO (mMIMO) is currently at the heart of the fifth generation of
wireless networks and is envisioned to be employed also in future generations.
While it is commonly combined with OFDM, it inherits a similar nonlinearity
response. However, now, in addition to spectral regrowth, the unequal spatial
distribution of the nonlinear distortion is revealed. The nature of mMIMO
causes the signals transmitted from each array’s antenna to add, e.g., in-
phase boosting the received power in a given location or in a counter-phase
causing signals fading, depending on the utilized precoding. Now, when the
nonlinearities, generated and transmitted by each component front-end, come
into play, their addition has to be considered as well. While there was a
belief that nonlinear distortions will distribute omnidirectionally [4], there
are some cases, e.g., with line-of-sight propagation or a low number of
layers, when the nonlinear distortions have the same spatial distribution as
the wanted signal [5]. An example is shown in Fig. 4. This depicts normalized
radiation pattern of an OFDM waveform radiated from 128 transmitters. In each
transmitter the same PA model is used, i.e., soft-limiter of 3 dB IBO. The
Maximal Ratio Transmission (MRT) precoding for the azimuth of $45^{\circ}$ is
assumed. It is visible that both the desired signal and distortion have
similar spatial characteristics.

Figure 4: Normalized radiation pattern of desired and distortion signal
components in regard to azimuth angle for two-path channel model, MRT
precoding for azimuth $45^{\circ}$, 128 antennas and IBO = 3 dB.

In other words, in this case, the increase in the number of antennas does not
influence the Signal-to-Distortion Ratio (SDR). Obviously, this depends on the
wireless channel properties. The SDR value at the scheduled receiver for 3
different number of antennas, 2 types of channels[6] and varying IBO is
depicted in Fig. 2. As expected in all the cases the SDR rises with IBO. Most
important for the Massive MIMO case is the array gain, i.e., if full array
gain is available only for wanted signal it is expected that SDR rises by
around 12 dB by changing 1 antenna to 16. However, while it reaches around 9
dB for NLOS channel, it is smaller than 5 dB for LOS case. This shows that
increasing the number of antennas will not solve the Massive MIMO nonlinear
front-end problem.

Figure 5: SDR as a function of IBO for 3GPP channel models, and varying number
of antennas. MRT precoding for $45^{\circ}$ azimuth angle applied.

Even though massive MIMO can combat, e.g., high path loss or multi-user
interference, nonlinear distortion remains a valid problem that has to be
addressed. Obviously, this can be done using some OFDM-typical methods, such
as Peak to Average Power Ratio (PAPR) reduction algorithms [7] or front-end
predistortion. However, it is also possible to utilize some mMIMO-specific
solutions, e.g., precoding that reduces nonlinearity[8]. While the method in
[8] is designed for a single carrier system, mMIMO precoding aware of OFDM
processing is still missing. Obviously, the nonlinearity problem can be
addressed also at the other link side, i.e., by utilizing advanced receivers.
It has been shown in [9] for a single antenna OFDM system, that nonlinear
distortion can be treated as a kind of redundancy coding, distributing
information from a given subcarrier to other subcarriers, thus increasing
frequency diversity. The extension of this work to an mMIMO system is an open
topic as well.

##  V Front-end modeling for ML-based network optimization

The utilization of front-end characteristics can go far beyond the link-level
optimization, providing new degrees of freedom. The existence of nonlinearity
should be considered while analyzing the 5G and beyond wireless systems on a
system level. In this context, especially if this happens in a complex system
composed of multiple transmit antennas, multiple base stations that interfere,
and utilizing multi-stage processing, e.g., some proprietary traffic steering
or scheduling algorithms, control and optimization of such a system is a
complex task. In such a case analytical models do not exist or are
significantly simplified, and we cannot rely on standard optimization methods.
Though the 6G systems are expected to move toward the utilization of Machine
Learning (ML) and Artificial Intelligence (AI) in order to optimize network
parameters based on the analysis of real-data [10]. In this context, the most
straightforward approach is to optimize the operating point of the PAs used at
base stations. This parameter can be used to find a balance between the
transmitted power, power amplifier efficiency, and nonlinear distortion level.
Obviously, various goal functions can be used, e.g., to maximize spectral
efficiency, improve the cell-edge users throughput, or maximize energy
efficiency. As a reference, a classic method is proposed in [11], where the
optimization of PA operation point is considered, aiming at achieving maximal
signal-to-noise and distortion power ratio in a single OFDM link. While the
authors proposed an analytical solution, the considered system model was
relatively simple, e.g., only flat fading was considered. This is one of the
reasons why this solution can be not considered optimal in many situations.
Therefore, we propose to utilize some Reinforced Learning technique, where,
depending on the network state (which can be modeled in various ways,
initially we resort to users’ path-losses), various PA operating points are
considered, and via the network’s throughput observation the optimal operating
point is selected.

We have evaluated the proposed solution in terms of computer simulations. In
the simulation scenario, we have considered downlink in a single mMIMO cell
with one BS equipped with a rectangular antenna array of 128 elements and
operating at a center frequency of $3.6$ GHz with a bandwidth of $25$ MHz. Due
to the low complexity of both hardware and signal processing, we have
considered analog beamforming with a so-called Equal Gain Transmission (EGT)
precoder [12]. This type of precoder ensures that equal power is being
allocated per BS’s antenna. The PA installed at the BS along with its non-
linear effects is modeled as a soft limiter [1]. We used an Wireless Insite®
Ray-Tracing software to generate accurate radio channel coefficients for 150
sets of 20 UE distributed uniformly over the cell area. The single set is
named step, while all sets constitute epoch. Throughout the multiple epochs,
the RL agent was applying different values of IBO (constant over the whole
epoch), and related user throughputs were observed as rewards. The related
results are depicted in Fig. 6

Figure 6: Throughput of cell-edge (10th percentile), median, and the best
(90th percentiles) users obtained over the training epoch for different values
of IBO.

In the figure, it is clearly visible that there exists optimal value of
IBO$=9$ dB for cell-edge users (10th percentile), median users, and the best
users (90th percentile). This value provides the best balance between the
users’ received power and non-linear distortion. Choosing IBO, which is too
close to the PA saturation point ($0$ dB, $3$ dB, $6$ dB) reduces the user
throughput due to the high non-linear distortion level. On the other hand, for
high values of IBO, the received power is too small. This phenomenon is
especially visible for the cell edge (10th percentile) of users. From this
perspective ML models can be further trained to adjust IBO in a context-
dependent manner, i.e., based on the users’ path-loss distribution, or
possible location information.

##  VI Conclusions

The above discussion sheds some light on a typically overlooked problem of
nonlinearities present in wireless transceivers. The typical mathematical
models resorting to white noise and multipath fading channels cannot be used
if our aim is to design an ultimately efficient 6G system. However, this
requires a significant effort in realistic modeling and optimization of the
wireless front-end while designing higher-layer algorithms. This will open new
degrees of freedom, at the cost of increased complexity. One possibility is to
support a classic convex optimization solution with machine learning tools.

## Acknowledgements

This research was funded by the Polish National Science Centre, project no.
2021/41/B/ST7/00136. For the purpose of Open Access, the author has applied a
CC-BY public copyright license to any Author Accepted Manuscript (AAM) version
arising from this submission. https://doi.org/10.26636/jtit.2023.170523

## References

  * [1] K. M. Gharaibeh, _Nonlinear distortion in wireless systems: Modeling and simulation with MATLAB_. John Wiley & Sons, 2011.
  * [2] H. Ochiai, “An analysis of band-limited communication systems from amplifier efficiency and distortion perspective,” _IEEE Transactions on Communications_ , vol. 61, no. 4, pp. 1460–1472, 2013.
  * [3] P. Kryszkiewicz, “Efficiency maximization for battery-powered ofdm transmitter via amplifier operating point adjustment,” _Sensors_ , vol. 23, no. 1, 2023\. [Online]. Available: https://www.mdpi.com/1424-8220/23/1/474
  * [4] E. Bjornson, J. Hoydis, M. Kountouris, and M. Debbah, “Massive mimo systems with non-ideal hardware: Energy efficiency, estimation, and capacity limits,” _IEEE Transactions on Information Theory_ , vol. 60, no. 11, pp. 7112–7139, 2014.
  * [5] C. Mollén, U. Gustavsson, T. Eriksson, and E. G. Larsson, “Spatial characteristics of distortion radiated from antenna arrays with transceiver nonlinearities,” _IEEE Transactions on Wireless Communications_ , vol. 17, no. 10, pp. 6663–6679, 2018.
  * [6] 3GPP, “Study on channel model for frequencies from 0.5 to 100 GHz,” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 38.901, 03 2022, version 17.0.0 Release 17.
  * [7] P. Kryszkiewicz, “Amplifier-coupled tone reservation for minimization of ofdm nonlinear distortion,” _IEEE Transactions on Vehicular Technology_ , vol. 67, no. 5, pp. 4316–4324, 2018.
  * [8] R. Zayani, H. Shaïek, and D. Roviras, “Efficient precoding for massive mimo downlink under pa nonlinearities,” _IEEE communications letters_ , vol. 23, no. 9, pp. 1611–1615, 2019.
  * [9] S. V. Zhidkov and R. Dinis, “Belief propagation receivers for near-optimal detection of nonlinearly distorted ofdm signals,” in _2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring)_ , 2019, pp. 1–6.
  * [10] H. Yang, A. Alphones, Z. Xiong, D. Niyato, J. Zhao, and K. Wu, “Artificial-intelligence-enabled intelligent 6g networks,” _IEEE Network_ , vol. 34, no. 6, pp. 272–280, 2020.
  * [11] C. H. Azolini Tavares, J. C. Marinello Filho, C. M. Panazio, and T. Abrão, “Input back-off optimization in ofdm systems under ideal pre-distorters,” _IEEE Wireless Communications Letters_ , vol. 5, no. 5, pp. 464–467, 2016\.
  * [12] S. Zhang, R. Zhang, and T. J. Lim, “Massive mimo with per-antenna power constraint,” in _2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP)_ , 2014, pp. 642–646.