11institutetext:  University of Notre Dame, Notre Dame IN 46556, USA
11email: {dprijate, sgrieggs<EMAIL_ADDRESS>22institutetext: PAR
Government, 421 Ridge St, Rome NY 13440, USA
22email: {eric_robertson<EMAIL_ADDRESS>
#  Handwriting Recognition with Novelty: Supplemental Material††thanks: This
research was sponsored by the Defense Advanced Research Projects Agency
(DARPA) and the Army Research Office (ARO) under multiple contracts/agreements
including HR001120C0055, W911NF-20-2-0005,W911NF-20-2- 0004,HQ0034-19-D-0001,
W911NF2020009. The views contained in this document are those of the authors
and should not be interpreted as representing the official policies, either
expressed or implied, of the DARPA or ARO, or the U.S. Government.

Derek S. Prijatelj 11 0000-0002-0529-9190 Samuel Grieggs 11
0000-0002-2433-5257 Futoshi Yumoto 22 0000-0001-5354-8254 Eric Robertson 22
0000-0002-4942-4619 Walter J. Scheirer 11 0000-0001-9649-8074

##  1 Supplemental Material

This is the supplemental material for the main paper intended to provide
complete details of the Handwriting Recognition (HWR) domain and agent-centric
approach to it for others interested in working on this challenge problem.
Additional notes are provided for the domain formalization and the evaluation
protocol. A large selection of supplemental experiments is provided to explore
different aspects of the HWR domain with novelty in more detail.

##  2 Formalization of HWR with Novelty

This section consists of further details on the formalization of the HWR
domain. Specifically, we provide more discussion on the HWR novelty theory,
oracle definition, and ontology here.

###  2.1 A Theory Novelty for HWR: Additional Details

The theory of novelty for the HWR domain in the main paper notably excludes
two pieces from Boult et. al [1]. One of those components is the agent’s
($\alpha$) action space $\cal A$ that contains all possible actions
$a_{t}\in{\cal A}$ that the agent may take. The other part is the state
recognition function $f_{t}(x_{t},s_{t}):\mathbb{R}^{d}\times{\cal
S}\mapsto{\cal S}\times{\cal A}$, where $x_{t}$ is an observation-space input,
and $s_{t}\in{\cal S}$ is the agent’s state at the current time-step $t$ for
all possible states $\cal S$ of the agent. In traditional image classification
tasks, these two components do not come into play as the agent is neither
stateful nor does it take actions in the world beyond outputting the predicted
class for a sample. In image classification, as well as the HWR domain with
novelty as defined in the main paper, the state recognition function could
simply be the predicted class for a given sample.

However, these two components could be a part of an agent for the HWR domain
with novelty. For example, the state of the agent could update if the agent
were to include incremental learning to solve one of the HWR tasks. The
incremental learning would then be a state change in the learning for the
agent, as would any “learning” that were to occur over time, such as the
continued training of an artificial neural network. An extreme example of
incremental learning would be an agent that is pre-trained on available
handwritten documents, but is given a new set of documents in a different
language, thus with new glyphs, characters, words, language, etc., to be
learned and is only given these new documents over time as they are
discovered.

If the open world HWR agent proposed in the main text were to be extended and
given the ability to act in the world, then the possible actions $\cal A$
would come into play. An example of this would be an automated robot
performing the entire handwritten document transcription process itself, e.g.,
opening the physical books carefully, gently turning the pages, and possibly
changing the perceptual operator by zooming in and out of the images or
changing sensors.

####  2.1.1 2.1.1 Caveats of Defining the Oracle

The oracle mainly consists of the datasets used for evaluation. However, the
oracle also determines world dissimilarity and regret functions given the
data. For datasets that consist of ground truth labels typically used in
supervised learning, the datasets may be enough. However, there may be
additional domain knowledge that is not available in the datasets, either
implicitly or explicitly defined by the oracle. An example is the case of
label frequency within the dataset mismatching the current domain knowledge of
the world. In this case, the oracle provides label weightings to adjust the
dataset’s samples to better represent the current domain knowledge of the
universal population of those labels. The information about these weightings
may or may not be provided to the agent as an extended part of the world
space.

Another example case is when the dataset has missing labels, either partially
or completely, the oracle is expected to provide the information that defines
the task given the data. A complete lack of labels for a certain type of
information is a rather common occurrence in domains where novelty is present
and cannot be labeled in any capacity, and where labeled or unlabeled data may
be used in training and evaluation. The datasets do not necessarily define the
oracle’s information in its entirety. This is a challenge for evaluation
design that needs to be accounted for when assessing agents that must manage
novelty. A sampling problem, such as HWR, thus defines the oracle and task
through the world space, world dissimilarity, world regret, and the
information used from datasets and domain knowledge.

###  2.2 HWR Ontological Specification

An ontological specification serves to characterize the novelty space and
provide a basis for measuring the difficulty of detecting novelty within that
space. Novelty in HWR is organized by three categories: writing style, pen
selection and background novelties, which are the novelties that are not
specific to the content of the text. The intent of specification is to
describe all observable novelties (from an oracle’s view of the world)
including environmental novelties such as water damage to the writing medium,
temporal and locale novelties such as date and time representations and
document structures, and text-related novelties such as copyedit marks.

In the development of the ontology-based knowledge graph, we first start with
the characterization of writing style. Writing style is made up of the style
attributes slant angle, word space, character size and pen pressure. Each
style attribute is described by a continuous function, defined in Table 1,
which is applied to images of words present in each writing sample [3]. The
results from the functions for all samples are binned to form discrete style
descriptors, which are used to construct style attribute nodes in the
knowledge graph. The number of bins is chosen to provide adequate separation
of each writing style. In our initial assessment, we used four bins for slant
angle and three bins for the rest of the style attributes. The style
attributes are collected for all writing sample images. The most frequent
style attribute value is assigned to each writer. The result is a set of
associations between each writing sample, the style, and the writer, as shown
in the knowledge graph in Fig. 1. We apply the same approach for background
and pen novelties. The non-style measurement functions for these novelties are
described in Table 2.

Style | Function
---|---
Pen Pressure |  ${\left(\sum_{i=1}^{\cal N}pixel[i]\right)/{\cal N}}$ where ${\cal N}$ is the number pixels in the written text, and ${pixel[i]}$ is the intensity of a pixel ${i}$.
Slant Angle |  ${\max\limits_{A^{i}}~{}S(A^{i})}$ where ${A^{i}}$ is the set of angles [-45,-30,-20,-15,-5,0,5,15,20,30,45], and ${S(A^{(i)})}$ is a shear estimate [9].
Word Spacing |  Average number of horizontal pixels between words where a space is a vertical slice with fewer than 30% quantile of vertical pixels for a line image.
Character Size |  Average number of pixels over all vertical slices of the image excluding those slices labeled as a space.
Table 1: Style Measurement Functions Novelty Type | Function
---|---
Background |  Entropy of the grey level background (without text)
Pen |  Entropy of the grey level pixel intensities in the written text.
Table 2: Non-Style Measurement Functions.

A correct knowledge graph consists of each writing sample associated to a
single writer via a two step path through the four style attribute nodes. The
writing style measurement functions provide a gross measure of writing style.
Combined with the inaccuracies introduced with binning, not all writing
samples from the same writer are associated with the same set of bins across
all style attributes. Since the same writer’s style is an aggregate value over
a set of writing samples, some binned measures for a writing sample form an
association to a style attribute not associated with the writer of the sample.
This is highlighted in the sample graph in Fig. 1 via a red edge. This
suggests an optimization strategy for style binning and association to
maximize the number of writing samples associated with a writer through the
four style attribute nodes.

Figure 1:  Illustrative Knowledge graph of Writing Style for four style
attributes associated writing samples a05-022-07 and r06-143-01 and five
selected writers. The red edge on the bottom represents the writing style of a
sample not associated with the sample’s author.

###  2.3 Ontological Specification for Novelty Characterization

Characterization was achieved through groups of clusters over writer samples
created by the agent. Each group explains a single characterization of novelty
as it occurs in each text image. Groups included in our initial study are:

  * •

Up to 3 clusters for pen pressure, character size and word spacing,

  * •

Up to 4 clusters for slant angle,

  * •

Up to 3 clusters for category of novelty: writer novelty, background and pen
novelties.

A single ‘writer novelty’ cluster occurs in the novelty category cluster group
when novelty does not occur — all non-novel examples cluster together. Fig. 2
illustrates this approach with two cluster groups.

For performance evaluation of characterization, we use Normalized Mutual
Information (NMI) to measure the quality of the clusters. We first separate
the agent characterizations of writing samples with no novelty and the three
categories of novelty: writing style, pen and background. We interpret
characterizations in the non-novel subgroup as a base measurement of the
agent’s dependence on the cluster-represented attributes to describe novelty.

Novelty | PP | CS | WS | SA | NC
---|---|---|---|---|---
Style | ${PP_{s}}$ | ${CS_{s}}$ | ${WS_{s}}$ | ${SA_{s}}$ | ${NC_{s}}$
Background | ${PP_{b}}$ | ${CS_{b}}$ | ${WS_{b}}$ | ${SA_{b}}$ | ${NC_{b}}$
Pen | ${PP_{p}}$ | ${CS_{p}}$ | ${WS_{p}}$ | ${SA_{p}}$ | ${NC_{p}}$
No Novelty | ${PP_{n}}$ | ${CS_{n}}$ | ${WS_{n}}$ | ${SA_{n}}$ | ${NC_{n}}$
Table 3: Characterization cluster groups are Pen Pressure (PP), Character Size
(CS), Word Spacing (WS), Slant Angle (SA), and Novelty Category (NC).

The characterization promotes better understanding of an agent’s performance
in the HWR domain with novelty. We first establish a baseline cluster quality
using non-novel writing samples. In the baseline, cluster groups organize
samples by similar styles and backgrounds. As different types of novelty are
introduced, new cluster centers are formed to isolate those samples perceived
as having the group’s representative novelty.

We partition and evaluate the characterization clusters by novelty category,
shown in each row of Table 3, to highlight the interactions between different
categories of novelty and the novel style attributes. Applicable measures to
this structure include NMI and cluster purity. This structure for analysis
aids in understanding agent response to mixed novelties, such as style and
background changes. The No Novelty row serves as a baseline characterization
of writing samples without novelty. The Style row measures characterization
clusters of samples with novel writing styles. In terms of a mapping to
empirical observations, low performance for cell $PP_{p}$ (Pen, Pen Pressure)
in comparison to baseline No Novelty was observed, indicating that this
paper’s open world agent is unable to discern pen changes from pen pressure
novelty. We do not expect to see much variation from the non-novel examples in
cells ${CS_{p}}$, ${WS_{p}}$ and ${SA_{p}}$, since pen pressure novelties do
not significantly affect character size, slant angle and word spacing. We also
expect matched performance to the baseline No Novelty conditions for separable
novelty categories, such as all Style cells in the Background row ($PP_{b}$,
$CS_{b}$, $WS_{b}$, $SA_{b}$). For example, an agent is expected to separate
novel from non-novel backgrounds, but may fail to adequately separate groups
of samples using two different novel backgrounds.

Figure 2: Example Clusters for two cluster groups, Style and Background, under
three novelty types: (1) No Novelty; (2) Novel Style; and (3) Novel
Background.

The Novelty Category (NC) cluster group serves to characterize the core types
of novelty. NC cells $NC_{c}$, $NC_{b}$ and $NC_{p}$ are meant to be
measurements of an agent’s ability to distinguish different samples within the
same category of novelty. For example, $NC_{b}$ measures an agent’s ability to
distinguish examples with blue backgrounds from those with red backgrounds.

For an initial assessment, we characterized the last 32 test images selected
from each test prior to evaluating characterization of the novelty. We provide
the sample set of measurements using Cluster Purity in Table 4.

$\text{Purity}=\frac{1}{N}\sum_{i=1}^{k}max_{j}|c_{i}\cap t_{j}|$ (1)

where ${N}$ = number of writing samples, ${k}$ = number of clusters, ${c_{i}}$
is a cluster in ${C}$, and ${t_{j}}$ is a ground truth novelty label.

In this sample, we see evidence of confounding variables when characterizing
pen pressure with pen changes. Characterization of slant angle, when compared
to non-novel cases, is weakly affected by pen changes. Word spacing was
significantly affected in all three novelty cases. Style changes were
correctly separated from background and pen novelties, as indicated in the
Novelty Category cluster group.

Novelty | PP | LS | WS | SA | NC
---|---|---|---|---|---
Style | 0.88 | 0.85 | 0.55 | 0.53 | 1.00
Background | 0.83 | 0.89 | 0.45 | 0.61 | 0.89
Pen | 0.75 | 0.71 | 0.57 | 0.77 | 1.00
Non-Novel | 0.84 | 0.75 | 0.80 | 0.80 | 1.00
Table 4: Cluster Purity Characterization Results based on novelty type.
Characterization cluster groups here are Pen Pressure (PP), Letter Size (LS),
Word Spacing (WS), Slant Angle (SA) and Novelty Category (NC).

##  3 Additional Information on HWR Agents

The details of the baseline open world HWR agent generally defined in the main
paper are included here, along with an additional agent that is not designed
to handle novelty.

###  3.1 Baseline Open World HWR Agent

This paper’s proposed HWR novelty detecting agent is further specified in this
section. This baseline open world agent for HWR is built upon the
Convolutional Recurrent Neural Network (CRNN) architecture which is commonly
used for closed set HWR tasks. The IAM dataset [4], a very commonly used
handwritten text dataset, contains a number of writing errors that were
introduced when the dataset was created. These are mostly in the form of
crossed out misspelled words. The ground truth provided with the dataset
represents these errors, with the “#” character. This is treated as the
baseline agent’s exposure to novel characters, serving as the known unknown
class. This known unknown class was further expanded upon by introducing a
subset of novel characters from the RIMES dataset, which contains numerous
characters with diacritics that are not present in IAM.

For all experiments using a CRNN, the model was structured as follows and is
shown in Table 6. Five convolutional layers feed into five bidirectional LSTM
layers, each with a kernel size of 3 and a stride and padding of 1. The 5 LSTM
layers have a hidden size of 256. Input Images were resized to 64 pixels tall.
The CRNNs were trained until they did not improve for 80 epochs using the
RMSprop optimizer with an initial learning rate of $3*10^{-4}$. On average the
models would train for around $300$ epochs, using a batch size of $8$.
Training proceeded indiscriminately on a selection of Nvidia Titan X, Titan
Xp, 1080ti, 2080ti and RTX 6000 GPUs, with each epoch taking about 5-10
minutes depending on the GPU used. Inference averaged about 33 milliseconds
per sample on a 2080ti.

The CRNN serves as both the feature extractor and transcript predictor as seen
in the main paper’s Fig. 2. In the supplemental Figure 3, the CRNN portion of
the agent is depicted in isolation to indicate its merging of feature
extraction and transcript predicting during its training in a supervised
learning fashion. To be used as a feature extraction for the EVMs for writer
identification and ODAI, the penultimate layer of the CRNN (specifically the
last layer of its RNN portion), was used as the encoding of the line images.
Given that the sequence differs with the length of the input line image, the
encodings were padded with zeros to the maximum time-step size (656 with the
IAM and RIMES data) and then an incremental principal component analysis [6]
method was used with 1000 components to obtain a memory-managable encoding for
the EVMs. And to expedite the process given both memory and time constraints
we used only one fourth of the data after running through the CRNN. Due to
using 1000 components to fit in memory, a lot of useful information was lost
and probably significantly affected the performance of the MEVMs that used the
CRNN-PCA as their feature space.

Figure 3:  The CRNN in isolation depicting its joint use as both the feature
extractor and transcript predictor. The feature extraction occurs in joint
with its supervised learning of the transcription task. The penultimate layer
of the CRNN is used as one of the examined feature spaces for training the
style task EVMs, after using PCA on the zero padded sequential output. Due to
memory constraints, 1,000 components were used with from an incremental
implementation of PCA on 25% of the training data.

The Extreme Value Machine (EVM) is an open set classifier designed to handle
novel classes [7]. The EVM has various hyperparameters, including tail size,
cover threshold, distance measure, and distance multiplier. The tail size used
was 1000, the cover threshold was 0.5, the distance measure used was Cosine
similarity, and the distance multiplier was 0.5. These hyperparameters were
the same for the two separate EVMs trained on their respective style tasks of
writer identification and ODAI. The implementation of the EVM used is a
pytorch version with GPU support111This will be made publicly available after
the publication of this paper.. EVM training took approximately 2 hours per
training fold on both data sets, thus equating to 20 hours total of EVM
training given the two experiments with 5-fold cross validation. Prediction
time was approximately one half hour for each evaluation fold using the EVM.
Hardware used for training and inference of the EVM matched that of the CRNN
described above.

The EVM’s output a probability vector of size $K+1$ for $K$ known labels. The
extra label serves as the general novel class label (referred to as the
unknown class in the EVM documentation). To obtain this probability vector,
the EVM outputs all of the probabilities for the $K$ classes in its
implementation. To obtain the probability of the novel class, the probability
of the maximum probable known class $k_{m}$ is taken with $1-k_{m}$ calculated
as the probability of the novel class. The rest of the known probabilities are
scaled by $k_{m}$ and the probability of novelty is appended to the end of the
probability vector.

###  3.2 Baseline Closed World HWR Agents

Two additional closed world agents were evaluated as comparison points to the
open world agent. One agent performs the writer identification style subtask,
while the other performs the text transcription task. They do not pass
information between each other, and have no specific abilities to manage
novelty.

####  3.2.1 Baseline Closed World Writer Identification Agent

A baseline closed world agent for just writer identification was created for
comparison to the open world agent, described in the main text, under novel
conditions. The closed world baseline agent predicts, for each sample, one of
the 50 known writers in the training set by applying the softmax function to
the output of the dense layer of a CNN. The baseline model serves to
demonstrate limited utility only in a closed world, over-fit to known writers,
with considerable degradation in performance when exposed to novel conditions.

The baseline writer identification model is a neural network consisting of
three groups of 2-D convolution layers with RELU activation and max pooling,
followed by two groups dense connected layers with RELU activation and 50%
drop out, ending with a dense softmax activated layer over the 50 known
writers [5].

zero_padding2d | (115, 115, 1)
---|---
Conv-2D | (58, 58, 32)
MaxPooling2D | (29, 29, 32)
Conv-2D | (29, 29, 64)
MaxPooling2D | (14, 14, 64)
Conv-2D | (14, 14, 128)
MaxPooling2D | (7, 7, 128)
Flatten | 6,272
DropOut | 6,272
Dense | 512
DropOut | 512
Dense | 256
DropOut | 256
Dense | 50
Table 5: Baseline Closed World Writer-Predictor Agent Model. Conv2d | (16, 64, x)
---|---
BatchNorm2d | (16, 64, x)
LeakyReLU | (16, 64, x)
MaxPool2d | (16, 32, .5x)
Conv2d | (32, 32, .5x)
BatchNorm2d | (32, 32, .5x)
LeakyReLU | (32, 32, .5x)
MaxPool2d | (32, 16, .25x)
Conv2d | (48, 16, .25x)
BatchNorm2d | (48, 16, .25x)
LeakyReLU | (48, 16, .25x)
Dropout2d | (48, 16, .25x)
Conv2d | (48, 16, .25x)
BatchNorm2d | (48, 16, .25x)
LeakyReLU | (48, 16, .25x)
Dropout2d | (48, 16, .25x)
Conv2d | (64, 16, .25x)
BatchNorm2d | ((64, 16, .25x)
LeakyReLU | (64, 16, .25x)
Conv2d | (80, 16, .25x)
BatchNorm2d | (80, 16, .25x)
LeakyReLU | (80, 16, .25x)
Flatten Interior | (1280, .25x)
Reshape | (.25x, b, 1280)
5 x bidirectional LSTM | (.25x, b, 512)
Linear | (.25x, b, 80)
LogSoftmax | (.25x, b, 80)
Table 6:  Convolutional Recurrent Neural Network used for Handwriting
Recognition in the Baseline Open World Agent. For experiments in which a CRNN
embedding is used, the embedding is extracted at the double line. Note that
‘x’ is the input image width and ‘b’ is the batch size, which is only shown
when it is not in the first position.

IAM Writer Identification Distribution for
---
Basic Feature and Transcription Evaluation
Datasplit | Total | Total Writers | Total Intersecting
Type | Writers | in Split | Between Pairs
|  | train | val | test | train & val | train & test | val &test
IAM Aachen | 431 | 373 | 93 | 170 | 65 | 135 | 57
5-Fold CV | 431 |  $\sim$354 |  $\sim$251 |  $\sim$259 |  $\sim$216 |  $\sim$216 |  $\sim$216

Table 7:  The mean 5-fold cross validation experiment’s distribution for IAM
writer identification. The version of the IAM data is the Aachen version,
which standardizes some of the character transcriptions and handles errors.
RIMES was excluded from this due to all RIMES documents being treated as a
single unknown writer. The used 5-fold cross validation indicates the
approximate distribution of writers between each data split for a single round
of training and evaluation of a fold.

####  3.2.2 Baseline Closed World Text Transcription Agent

A baseline closed world agent for just text transcription was created for
comparison to the open world agent described in the main text under novel
conditions. This baseline agent produces text for each writing sample, based
on what it knows from the 50 known writers in the training set, by applying
log-softmax to the output of deep recurrent layers [8]. It serves to
demonstrate limited utility only in a closed world, over-fit to known writers,
with considerable degradation performance when exposed to novel conditions.

##  4 Basic Feature and Transcription Evaluation: Additional Protocol
Information and Detailed Analysis

The data splits for the cross validation were obtained by first halving the
unique writers into two equal groups of 216 each. Then, one half was randomly
shuffled and split into 5 folds in a traditional 5-fold cross validation
manner, stratified by the writer identifiers for the best representation of
all 216 writers in each fold’s samples. The other half was then further split
into 5 groups of unique writers with no intersection. This second split
ensures that for every fold, there is a set of novel writers in the test
dataset. Each half’s 5 folds were then aligned randomly such that typical
5-fold cross validation may occur. The training set, consisting of 4 folds,
for every round of cross validation was then split in the exact same way, such
that the validation set also had novel writers at evaluation time. This method
of obtaining the 5-fold cross validation folds results in the approximate
distribution of writers in training, validation, and testing sets as seen in
Table 7. This is approximate, because due to the imbalanced number of samples
per writer, where some writers had less than 5 samples, some folds had more
writers than others.

###  4.1 Novel Characters in Transcription

In general, most HWR models will have some sort of “background” class to
represent spurious marks or mistakes on the page. For the purposes of this
experiment we trained on a combined IAM and RIMES dataset in which the RIMES
transcriptions were modified to include the characters that are not a part of
the IAM dataset as background. RIMES and IAM were broken into folds such that
Zipf’s law gives us a variety of known unknown and unknown unknown characters
in each fold, in the terminology of open world recognition. In terms of
novelty, due to Zipf’s law, novel characters unseen at training time occurred
naturally in both the validation and testings sets for all folds. The addition
of RIMES, and thus all of its French specific characters not included in the
IAM dataset, included more characters whose labels were never known to the
agent. However, some were included by design in the training set as unknown
characters seen during training (known unknowns).

###  4.2 Novelty in Overall Image Appearance

In order to simulate novelty in the ODAI style recognition subtask, we
augmented the IAM and RIMES datasets by randomly modifying the backgrounds of
the images. The data was split into three different representation classes for
training, Noise, which added Gaussian background noise as well as over the
foreground, Antique, which adds a background similar to that of a historical
document, and the Original White background from a clean document scan. The
Original White background with black foreground is the unaltered background of
each image found in IAM and RIMES, and is the typical ideal clean scan of
handwritten documents. Additionally there are two known unknown classes that
are seen at training time, Reflect_0, which is flipping the text image over
the horizontal axis, and Blur, which adds Gaussian blur to the image. There is
another augmentation only included in the validation and test sets, Reflect_1,
which reflects the image over the vertical axis. Finally, the test set
includes another novel image appearance class where the image has inverted
color, the InvertColor class.

The Antique class used a set of free-to-use background images totaling in 16
different background images all accessed as of the data 2020-12-30. Nine of
these images were taken from commons.wikimedia.org that were categorized as
“old paper”, “vintage paper”, or as “parchment”:

  * •

El siglo de las tinieblas, o memorias de un inquisidor; novela histórica
origina:  https://commons.wikimedia.org/wiki/File:El_siglo_de_las_tinieblas,
_o_memorias_de_un_inquisidor;_novela_hist%C3%B3rica_original_(1868)
_(14590934239).jpg

  * •

Old paper 1: https://commons.wikimedia.org/wiki/File:Old_paper1.jpg

  * •

Old paper 3: https://commons.wikimedia.org/wiki/File:Old_paper3.jpg

  * •

Old paper 4: https://commons.wikimedia.org/wiki/File:Old_paper4.jpg

  * •

Old paper 6: https://commons.wikimedia.org/wiki/File:Old_paper6.jpg

  * •

Old paper 7: https://commons.wikimedia.org/wiki/File:Old_paper7.jpg

  * •

Vinatage Paper Texture: https://commons.wikimedia.org/wiki/File:
Vintage_Paper_Texture_(9789792113).jpg

  * •

Blank page, brown paper texture:
https://commons.wikimedia.org/wiki/File:Blank_page,_brown_paper_texture_(14802136533).jpg

  * •

Parchment 00: https://commons.wikimedia.org/wiki/File:Parchment.00.jpg

Besides these more authentic paper backgrounds, some artists’ free-to-use
interpretations of antique paper were also used:

  * •

Pixabay empty brown canvas on Pexels: https://www.pexels.com/photo/abstract-
ancient-antique-art-235985/

  * •

HD paper texture by Imrooniel on Deviant Art:
https://www.deviantart.com/imrooniel/art/HD-paper-texture-298160595

  * •

5 paper textures by MarshmellowHeaven that were stained with coffee and tea:

    * –

https://www.deviantart.com/marshmellowheaven/art/Texture-Paper-1-195235719

    * –

https://www.deviantart.com/marshmellowheaven/art/Texture-Paper-2-195236191

    * –

https://www.deviantart.com/marshmellowheaven/art/Texture-Paper-3-195236939

    * –

https://www.deviantart.com/marshmellowheaven/art/Texture-Paper-4-195237220

    * –

https://www.deviantart.com/marshmellowheaven/art/Texture-Paper-5-195237843

These background images were randomly selected for every handwriting line
image that was chosen to be of the Antique class. All backgrounds were turned
to grayscale, while remaining in RGB color space to match IAM and RIMES
formats. Given the chosen background image, a cropping of that background that
fit the size of the handwriting line image was selected and the handwriting
line was laid over that cropped background. The exact procedure is available
in the provided code, which will be publicly available upon publication. While
the Large-Scale evaluation did not assess ODAI as the 5-fold CV experiments
did in the main paper, it did use the same code with different backgrounds to
assess how the baseline agents performed when the background changed.

Training Set Mean Measures of 5-fold Cross Validation
---
Task | Multi-class Classif. | Binary Novelty
| with Novel Class | Detection
Model | NMI | Acc. | NMI | Acc.
Writer ID |  |  |  |
Mean HOG EVM | 0.8198 $\pm$ 2.09e-3 | 0.8444 $\pm$ 8.58e-4 | 0.9557 $\pm$ 1.12e-3 | 0.9889 $\pm$ 3.38e-4
10-Mean HOG EVM | 0.9871 $\pm$ 9.55e-4 | 0.9921 $\pm$ 4.37e-3 | 0.9586 $\pm$ 3.35e-3 | 0.9900 $\pm$ 9.38e-4
ResNet50 EVM | 1.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0 | 1.0 $\pm$ 0.0
CRNN-PCA EVM | 0.9996 $\pm$ 9.90e-5 | 0.9998 $\pm$ 4.07e-3 | 0.9990 $\pm$ 3.94e-4 | 0.9998 $\pm$ 6.18e-5
Appearances (ODAI) |  |  |  |
Mean HOG EVM | 0.6611 $\pm$ 2.47e-3 | 0.8559 $\pm$ 1.46e-3 | 0.4817 $\pm$ 4.04e-3 | 0.7654 $\pm$ 2.78e-3
10-Mean HOG EVM | 0.7124 $\pm$ 1.33e-3 | 0.8973 $\pm$ 5.74e-4 | 0.6076 $\pm$ 3.02e-3 | 0.8348 $\pm$ 1.74e-3
ResNet50 EVM | 0.8327 $\pm$ 5.44e-6 | 0.8000 $\pm$ 4.61e-6 | 0.4326 $\pm$ 6.93e-6 | 0.6667 $\pm$ 6.40e-6
CRNN-PCA EVM | 0.4455 $\pm$ 3.19e-3 | 0.6546 $\pm$ 5.27e-3 | 0.0840 $\pm$ 6.78e-3 | 0.3089 $\pm$ 1.31e-2
| Character Acc. | Word Acc. | NMI | Acc.
Transcription |  |  |  |
CRNN | 0.9904 $\pm$ 5.83e-4 | 0.9660 $\pm$ 1.98e-3 | 0.9601 $\pm$ 3.17e-3 | 0.9913 $\pm$ 8.07e-4

Table 8:  The mean 5-fold results with standard error for the train split of
all three experiments. “NMI” stands for Normalized Mutual Information. All
measures reported here are found after selecting the maximum probable class as
predicted by the classifier after thresholding the maximum probability to
determine if novel.

Validation Set Mean Measures of 5-fold Cross Validation
---
Task | Multi-class Classif. | Binary Novelty
| with Novel Class | Detection
Model | NMI | Acc. | NMI | Acc.
Writer ID |  |  |  |
Mean HOG EVM | 0.7394 $\pm$ 9.28e-3 | 0.7123 $\pm$ 4.93e-3 | 0.7754 $\pm$ 1.66e-2 | 0.9265 $\pm$ 7.19e-3
10-Mean HOG EVM | 0.6497 $\pm$ 7.90e-3 | 0.7852 $\pm$ 3.83e-3 | 0.3857 $\pm$ 8.17e-3 | 0.6246 $\pm$ 6.13e-3
ResNet50 EVM | 0.6403 $\pm$ 8.56e-3 | 0.7876 $\pm$ 3.23e-3 | 0.3793 $\pm$ 6.18e-3 | 0.6126 $\pm$ 6.18e-3
CRNN-PCA EVM | 0.6513 $\pm$ 7.95e-3 | 0.8074 $\pm$ 3.54e-3 | 0.3949 $\pm$ 6.96e-3 | 0.6266 $\pm$ 6.77e-3
Appearances (ODAI) |  |  |  |
Mean HOG EVM | 0.5809 $\pm$ 2.54e-3 | 0.7886 $\pm$ 1.96e-3 | 0.3358 $\pm$ 3.86e-3 | 0.6464 $\pm$ 3.69e-3
10-Mean HOG EVM | 0.4948 $\pm$ 4.14e-3 | 0.7525 $\pm$ 1.96e-3 | 0.2894 $\pm$ 2.75e-3 | 0.5799 $\pm$ 2.49e-3
ResNet50 EVM | 0.0272 $\pm$ 1.18e-3 | 0.5097 $\pm$ 4.11e-4 | 0.0181 $\pm$ 7.61e-4 | 0.0989 $\pm$ 2.21e-3
CRNN-PCA EVM | 0.0177 $\pm$ 1.87e-3 | 0.4315 $\pm$ 5.96e-3 | 0.0027 $\pm$ 1.42e-2 | 0.4848 $\pm$ 6.28e-3
| Character Acc. | Word Acc. | NMI | Acc.
Transcription |  |  |  |
CRNN | 0.9516 $\pm$ 3.53e-3 | 0.8861 $\pm$ 2.61e-3 | 0.8787 $\pm$ 7.03e-3 | 0.9664 $\pm$ 2.44e-3

Table 9:  The mean 5-fold results with standard error for the validation split
of all three experiments. “NMI” stands for Normalized Mutual Information. All
measures reported here are found after selecting the maximum probable class as
predicted by the classifier after thresholding the maximum probability to
determine if novel.

##  5 Large-Scale 55K Test Evaluation: Additional Protocol Information and
Detailed Analysis

The Mean HOG configuration of the baseline open world HWR agent was evaluated
with 55,000 tests. We generate 5,500 tests based on experimental conditions.
For each generated test, we create nine additional tests, re-ordering the test
samples to average-out sample variations while retaining the same conditions
of the test. Tests were constructed and grouped by types of novelty. The tests
were constructed to evaluate both single writing sample novelty detection and
world change detection indicated by data distribution change from a non-
novelty phase to a novelty phase of the test. In addition to establishing a
foundation for novelty detection and characterization in HWR, in this
evaluation, we establish some initial metrics for novelty difficulty,
identifying factors impacting the performance of detecting novelty and
transcribing handwritten text.

###  5.1 Protocol: Modified IAM Off-Line Handwriting Data.

The roughly 55,000 novel writing samples used in evaluation were constructed
from modified samples of the IAM Offline Handwriting Dataset [4]. The training
data will be publicly released after this paper’s publication. A
representative portion of the tests will be released as well.

Training and evaluation data, in the form of individual lines, was selected
from IAM. Prior to training, lines were denoised, removing shadow boxes around
the letters of each word. Features were then extracted from the clean lines of
written text to capture writing characteristics including pen pressure, letter
slant, word spacing and character size [2]. A distance matrix was formed
between by the sum of absolute differences between each writer’s mean style
across all example words from each writer. The distance matrix served as a
writer similarity measurement.

The training set was made up lines of text from 50 selected writers
representing a subset of the writer style descriptor values, leaving one or
two bins for each feature excluded for use in the novelty evaluation set.
Lines of text did not contain any additional effects, using a white
background. Sample lines from six additional writers were chosen to compose an
unknown writer training set. The set was supplemented with samples from the
RIMES dataset and samples from the same 50 writers with background effects
including salt and pepper noise, antique paper, and faded impressions of
shaded boxes around the words in each line of text.

The evaluation set was made up of the remaining writers and writing sample
manipulations to alter characteristics of both the writing style and the
background. Letter style manipulations included thinning or widening,
brightness, resizing and slant adjustments to each line of text. The
background was composed from Creative Commons licensed images of textured
paper. Pen manipulations were similarly constructed by merging in textures and
colors, weighted by the pixel strength (i.e., pen pressure).

The difficulty associated with each test is determined by the novelty type.
The difficulty for novel writers and novel letter manipulations was determined
by the ontological separation of four writing style features: pen pressure,
letter slant, character size, and word spacing. Grouping novel writers with
non-novel writers with similar styles is intended to make detection more
difficult. The difficulty of novel pen and backgrounds was measured by the
inverse intensity of the background (since the letters are black).

Most novel examples were constructed with a single type of novelty. Background
and pen novelties were applied to sample text lines from the 50 known writers.
The number of text lines per test varied based on availability of data
targeting the specific novelty: 512, 768, or 1,024. In total, each test
selected from 1,696 non-novel examples of the 50 known writers and
approximately 50,000 novelties. Tests were composed of writing samples
selected and organized by six independent discrete variables defining the
experimental conditions of each test to explore the performance regime in
novelty detection, resulting in 3,888 unique combinations. Using several
subtypes (e.g., different backgrounds) of novelties by type and difficulty, we
constructed approximately 5,500 tests, each reordered nine times to average
out sample variations.

Difficulty and novelty type (Table 10) affect writer prediction and
transcription accuracy. Variables (Table 11) associated with distribution and
placement of novelty in stream of data, such as introduction point, density of
novel to non-novel samples and distribution type are varied to measure impact
on novelty detection.

Novelty Type | Count
---|---
Background | 17,662
Letter | 11,868
Pen | 11,289
Writer | 8,427
No Novelty | 1,696
Table 10: Number of writing samples for each type of novelty. Independent Variables | Values
---|---
Mean Novelty | 0.4, 0.5, 0.6, 0.7, 0.8, 0.9
Introduction Pt. |
Density of Novelty | 6 different densities
Novelty Type | Writer, Letter, Background
Difficulty | Easy, Medium, Hard
Distribution Type | High (positive skew),
| Low (negative skew),
| Mid (normal), Flat (uniform)
Test Length | 512, 768 and 1,024
Table 11: Independent Variables forming the experimental conditions of each
Novelty Test

###  5.2 Supplemental Results for the Large-Scale 55K Evaluation

This section provides a more fine-grained analysis over the 55,000 tests
presented to the closed world agents and the novelty detecting open world
agent described in the main text of the paper. The agent configuration used
for the open world agent experiments utilizes HOG features for all style
tasks. For this analysis, we present general novelty detection, text
transcription and writer identification performance across all tests based on
types of novelty.

####  5.2.1 Closed World Agents: Transcription and Writer Identification

As expected, novelty negatively impacted the writer identification and sample
transcription accuracy. Results are shown in Table 12. Mean Character
Transcription Accuracy is reported as
${1-L(G_{s},A_{s})/\max(|{G_{s}}|,|{A_{s}}|)}$ where ${L}$ is Levenshtein Edit
Distance, ${G_{s}}$ is ground truth text for writing sample ${s}$, and
${A_{s}}$ is the agent’s predicted transcription for writing sample ${s}$,
averaged over the ten variations of each test. Writer Identification Accuracy
is reported as mean accuracy of the top-1 and top-3 predictions out of $K$+1
writers, where $K$ = 50 for all tests, and the additional class is for novel
writers.

| Mean | Writer ID | Writer ID
---|---|---|---
Is Novel? | Char. Acc. | Top-3 Acc. | Top-1 Acc.
False | 0.85 | 0.99 | 0.99
True | 0.47 | 0.40 | 0.24
Table 12: Baseline closed world agent mean character transcription accuracy,
top-3 writer identification accuracy, and top-1 writer identification accuracy
in response to non-novel and novel writing samples.

####  5.2.2 Open World Agent: Novel vs. Non-Novel Predictions

Again as expected, novelty negatively impacted both text transcription and
writer identification accuracy. However, the open world agent is significantly
better at the text transcription task. Results are shown in Table 13.
Transcription performance is reported as mean character accuracy computed
using the ground truth and the agent provided transcriptions for all tests.
Writer identification accuracy is reported as mean accuracy of the Top-1 and
Top-3 predictions out of $K$+1 writers, where $K$ = 50 for all tests.

| Mean | Writer ID | Writer ID
---|---|---|---
Is Novel? | Char. Acc. | Top-3 Acc. | Top-1 Acc.
False | 0.82 | 0.942 | 0.719
True | 0.62 | 0.479 | 0.220
Table 13: Baseline open world agent mean character transcription accuracy,
top-3 writer identification accuracy, and top-1 writer identification accuracy
in response to non-novel and novel writing samples.

####  5.2.3 Open World Agent: Novel Style Manipulations

Style manipulations include manipulations to the characters. These
manipulations had a measurable impact on writer identification performance.
Results are shown in Table 14. Dilating the letters did not affect
performance, down-weighting pen width as a major factor of a writer’s style.
More extreme character manipulations such as large slants and slants coupled
with dilation were more easily detected as being novel, as expected. Inverting
pixel values for written text did not adversely affect writer identification
performance. The novelty detector did not equate letter inversion as novelty.
Each novelty type was represented by 1,696 sample images.

Four different summary statistics are computed for the novel style
manipulations. Novelty Detection Accuracy is mean accuracy of all of the
detection decisions. Mean Character Transcription Accuracy is defined as

${1-L(G_{s},A_{s})/\max(|{G_{s}}|,|{A_{s}}|)}$ (2)

where ${L}$ is Levenshtein Edit Distance, ${G_{s}}$ is ground truth text for
writing sample ${s}$, ${A_{s}}$ is the agent’s predicted transcription for
writing sample ${s}$, averaged over the ten variations of each test. NMI
represents normalized mutual information between the actual writer of the
sample and the top-1 predicted writer. Writer Identification Accuracy is mean
accuracy of the top-3 predictions out of $K$+1 writers, where $K$ = 50 across
all tests, and the additional class is for novel writers. These summary
statistics are also used for the novel pens and novel backgrounds assessments,
which are described below.

| Novelty | Mean |  | Writer
---|---|---|---|---
Novelty | Detection | Char. |  | ID
Type | Acc. | Acc. | NMI | Acc.
Dilate | 0.99 | 0.70 | 0.01 | 0.57
Erode | 0.79 | 0.77 | 0.35 | 0.68
Increase Size | 0.99 | 0.33 | 0.01 | 0.02
Big Right Slant | 0.79 | 0.62 | 0.21 | 0.09
Slant w/ Dilate | 0.99 | 0.46 | 0.04 | 0.00
Big Left Slant | 1.00 | 0.55 | 0.01 | 0.02
Small Slant | 0.86 | 0.52 | 0.23 | 0.04
Inverted | 0.33 | 0.71 | 0.79 | 0.94
Table 14: Novelty detection accuracy, mean character transcription accuracy,
top-1 writer identification mean normalized mutual information, and top-3
writer identification accuracy given pen novelties grouped by novel style
changes.

####  5.2.4 Open World Agent: Novel Pens

Novel Pens include manipulations to written text, replacing the pixels with
textures and colors, weighted by the intensity of the pen as described by pen
pressure. Results are shown in Table 15. Pen manipulations had minimal impact
on writer identification performance. Each novelty type was represented by
1,696 sample images.

| Novelty | Mean |  | Writer
---|---|---|---|---
Novelty | Detection | Char. |  | ID
Type | Acc. | Acc. | NMI | Acc.
Blue Color | 0.98 | 0.78 | 0.09 | 0.53
Brown Texture | 0.74 | 0.75 | 0.43 | 0.79
Gold Texture | 0.92 | 0.69 | 0.22 | 0.73
Rainbow | 0.98 | 0.71 | 0.10 | 0.57
Red Color | 0.98 | 0.70 | 0.10 | 0.53
Table 15: Novelty detection accuracy, mean character transcription accuracy,
top-1 writer identification mean normalized mutual information, and top-3
writer identification accuracy grouped by novel pens.

####  5.2.5 Open World Agent: Novel Backgrounds

Background manipulation had a more diverse impact on writer prediction
performance than style manipulations. Results are shown in Table 16. NMI
represents normalized mutual information between actual sample writer and
top-1 predicted writer. Novel types of shadow boxes (from the uncleaned lines
extracted from IAM) had the highest writer identification accuracy, perhaps
due to similar associations made with these types of artificial irregularities
in the training set. As with pen manipulations, more extreme manipulations
resulted in higher detection accuracy. Increased texture interfered with the
agent’s ability to identify the writer.

| Novelty | Mean |  | Writer
---|---|---|---|---
Novelty | Detection | Char. |  | ID
Type | Acc. | Acc. | NMI | Acc.
Antique | 0.40 | 0.80 | 0.47 | 0.39
Blue Fabric | 0.98 | 0.40 | 0.10 | 0.42
Blue Color | 0.98 | 0.69 | 0.01 | 0.54
Blue Wall | 0.99 | 0.33 | 0.01 | 0.10
Brown Fabric | 1.00 | 0.60 | 0.01 | 0.11
Crinked Paper | 1.00 | 0.71 | 0.02 | 0.16
Gaussian Noise | 1.00 | 0.14 | 0.00 | 0.16
Gold Wall | 0.68 | 0.51 | 0.34 | 0.37
Rainbow Paper | 0.98 | 0.25 | 0.12 | 0.54
Shadow Boxes | 0.59 | 0.88 | 0.62 | 0.91
Table 16: Novelty detection, mean character transcription accuracy, top-1
writer identification mean normalized mutual information, and top-3 novel
writer identification accuracy grouped by writing style.

####  5.2.6 Open World Agent: Writer Similarity in Novel Writer Discovery

Each test is composed of sample writing from known and unknown writers. Here
we find the minimum distance of an unknown writer across all known writers. We
hypothesize that the greater the distance of writer style attributes of
unknown writers with known writers, as captured in the ontological
specification, the easier it is to detect a novel writer.

Surprisingly, the results did not show a strong correlation as expected. Table
17 shows the Pearson’s correlation of each style attribute with detection and
top-1 novel writer identification accuracy. We believe this due to two key
factors: not enough variability in writing styles in the unknown population
and the chosen set of attributes insufficiently capturing all of the essential
characteristics of writing style. Pen pressure had the highest correlation of
the four ontological specified factors. Collectively, a weak positive
correlation did support the hypothesize. The proposed benchmark can be
augmented with additional attributes, as the challenge problem evolves.

Style |  Novelty Det. Corr. |  Writer ID Corr.
---|---|---
Slant Angle | 0.06 | 0.01
Skew Angle | 0.02 | 0.04
Word Spacing | -0.02 | -0.05
Pen Pressure | 0.14 | 0.09
Character Size | 0.03 | 0.04
Summed | 0.12 | 0.18
Table 17: Novelty detection and novel writer identification correlation
grouped by writing style.

####  5.2.7 Open World Agent: Factors in Novelty Detection

A critical factor in the 55K tests is the density and location of novelty
introduction — the switch between pre-novelty and post-novelty phases of the
test given a stream of writing samples. This approach treats novelty as
perceived world changing events rather than outliers, where confidence of
novelty predictions increases as more novel examples are encountered in the
data stream, increasing the body of evidence. With this approach, the level of
false positives, those misidentified non-novel examples that fall in the pre-
novelty phase of the test, can be substantially reduced. Fig. 4 shows the
false positive count by the proportion of novelty. The variability and amount
of false positives decreases as the proportion of novel samples to non-novel
samples increases.

Figure 4: False positive count by the proportion of novelty present.

We conducted ANOVA to identify factors affecting the false positive rate (see
Table 18). Along with the proportion of novelty, distribution type had a
significant impact on the false positive rate. A positively skewed
distribution, where novel samples densely occur at the start of the novelty
phase of the test, is associated with lower false positive rate when compared
to other distribution types such as a negatively skewed distribution. Novelty
difficulty had a weak association to the false positive rate.

Factor | Sum of Squares | $df$ | F | $p$
---|---|---|---|---
Distribution |  |  |  |
Type | 1.058e+06 | 3 | 129.300 | 0.000
Level of |  |  |  |
Difficulty | 7.456e+03 | 2 | 1.365 | 0.255
Location of |  |  |  |
Novelty | 2.646e+06 | 1 | 969.301 | 0.000
Proportion of |  |  |  |
Novelty | 5.848e+05 | 1 | 214.240 | 0.000
Residual | 1.205e+00 | 44170 |  |
Table 18: ANOVA analysis of statistical influence given several test
generating independent variables identified in Table 11 on false positive
rate.

## References

  * [1] Boult, T.E., Grabowicz, P.A., Prijatelj, D.S., Stern, R., Holder, L., Alspector, J., Jafarzadeh, M., Ahmad, T., Dhamija, A.R., Li, C., Cruz, S., Shrivastava, A., Vondrick, C., Scheirer, W.J.: A unifying framework for formal theories of novelty:framework, examples and discussion. In: AAAI (2021)
  * [2] Joshi, P.M., Agarwal, A., Dhavale, A., Suryavanshi, R., Kodolikar, S.: Handwriting analysis for detection of personality traits using machine learning approach. International Journal of Computer Applications 130, 40–45 (11 2015). https://doi.org/10.5120/ijca2015907189
  * [3] Malemnganba, M.: ml-graphology. https://github.com/Malemm/ml-graphology (2018)
  * [4] Marti, U.V., Bunke, H.: The IAM-database: an English sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition 5(1), 39–46 (2002)
  * [5] Reddy, T.: Offline handwriting recognition cnn. https://github.com/TejasReddy9/handwriting_cnn (2018)
  * [6] Ross, D.A., Lim, J., Lin, R.S., Yang, M.H.: Incremental Learning for Robust Visual Tracking. International Journal of Computer Vision 77(1), 125–141 (May 2008). https://doi.org/10.1007/s11263-007-0075-7, https://doi.org/10.1007/s11263-007-0075-7
  * [7] Rudd, E.M., Jain, L.P., Scheirer, W.J., Boult, T.E.: The extreme value machine. IEEE T-PAMI 40(3), 762–768 (2017)
  * [8] Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition (2015)
  * [9] Vinciarelli, A., Luettin, J.: A new normalization technique for cursive handwritten words. Pattern Recognit. Lett. 22(9), 1043–1050 (2001), http://dblp.uni-trier.de/db/journals/prl/prl22.html#VinciarelliL01


# Network of Recurrent Neural Networks

Chao-Ming Wang
School of Software, Beijing Jiaotong University, Beijing, China
<EMAIL_ADDRESS>
###### Abstract

We describe a class of systems theory based neural networks called “Network Of
Recurrent neural networks” (NOR), which introduces a new structure level to
RNN related models. In NOR, RNNs are viewed as the high-level neurons and are
used to build the high-level layers. More specifically, we propose several
methodologies to design different NOR topologies according to the theory of
system evolution. Then we carry experiments on three different tasks to
evaluate our implementations. Experimental results show our models outperform
simple RNN remarkably under the same number of parameters, and sometimes
achieve even better results than GRU and LSTM.

## Introduction

In recent years, Recurrent Neural Networks (RNNs) (?) have been widely used in
Natural Language Processing (NLP). Traditionally, RNNs are directly used to
build the final models. In this paper, we propose a novel idea called “Network
Of Recurrent neural networks” (NOR) which utilizes existing basic RNN layers
to make the structure design of the higher-level layers. From a standpoint of
systems theory (?; ?), a recurrent neural network is a group or an
organization made up of a number of interacting parts, and it actually is
viewed as a complex system, or a complexity. Dialectically, every system is
relative: it is not only the system of its parts, but also the part of a
larger system. In NOR structures, RNN is viewed as the high-level neuron, and
several high-level neurons are used to build the high-level layers rather than
directly used to construct the whole models.

Conventionally, there are three levels of structure in Deep Neural Networks
(DNNs): neurons, layers and whole nets (or called models). From a perspective
of systems theory, at each level of such increasing complexity, novel features
that do not exist at lower levels emerge (?). For example, at the neurons
level, single neuron is simple and its generalization capability is very poor.
But when a certain number of such neurons are accumulated into a certain
elaborate structure by certain ingenious combinations, the layers at the
higher level begin to get the unprecedented ability of classification and
feature learning. More importantly, such new gained capability or property is
deducible from but not reducible to constituent neurons of lower levels. It’s
not a property of the simple superposition of all constituent neurons, and the
whole is greater than the sum of the parts. In systems theory, such kind of
phenomenon is known as whole emergence (?). Whole emergence often comes from
the evolution of the system (?), in which a system develops from the lower
level to the higher level, from simplicity to complexity. In this paper, the
motivation of NOR structures is to introduce a new structure level to RNN-
related networks by transferring traditional RNN from the system to the agent
and from the outer dimension to the inner dimension (?).

In 1993, W. Brian Arthur (?) has identified three mechanisms by which
complexity tends to grow as systems evolve:

  * •

Mechanism 1: increase in co-evolutionary diversity. The agent in the system
seem to be a new instance of agent class, type or species. As a result, the
system seems to have new external agent types or capabilities.

  * •

Mechanism 2: increase in structural sophistication. The individual system
steadily accumulates increasing numbers of new systems or parts. Thus, newly
formed system seems to have new internal subsystems or capabilities.

  * •

Mechanism 3: increase by “capturing software”. The system capture simpler
elements and learns to “program” these as “software” to be used as its own
ends.

In this paper, with the guidance of first two mechanisms, we introduce two
methodologies to NOR structures design, which are named as aggregation and
specialization. Aggregation and specialization are natural operations for
increasing complexity in complex systems (?). The former is related to
Arthur’s second mechanism, in which traditional RNNs are aggregated and
accumulated into a high-level layer in accordance with a specific structure,
and the latter is related to Arthur’s first mechanism, in which the RNN agent
in a high-level layer is specialized as the RNN agent that performs a specific
function.

We make several implementations and carry out experiments on three different
tasks, including sentiment classification, question type classification and
named entity recognition. Experimental results show that our models outperform
constitute simple RNN remarkably with the same number of parameters and
achieve even better results than GRU and LSTM sometimes.

## Background

#### Systems Theory

Systems Theory was originally proposed by biologist Ludwig Von Bertalanffy (?;
?) for biological phenomena. In biology systems, there are several different
levels which begin with the smallest units of life and reach to the largest
and most extensive category: molecule, cell, tissue, organ, organ system,
organization etc. Traditionally, a system could be decomposed into its
individual components so that each component could be analyzed as an
independent entity, and components could be added in a linear fashion to
describe the totality of the system (?). However, Von Bertalanffy argued that
we cannot fully comprehend a phenomenon by simply breaking it down into
elementary parts and then reforming it. We instead need to apply a global and
systematic perspective to underline its functionality (?), because a system is
characterized by the interactions of its components and the nonlinearity of
those interactions (?).

#### Whole Emergence

In systems theory, the phenomenon (i.e., the whole is irreducible to its
parts) is known as emergence or whole emergence (?). Emergence can be
qualitatively described as: the whole is greater than the sum of the parts
(?). Or it can also be quantitatively expressed as:

$\displaystyle W>\sum_{i}^{n}p_{i}$ (1)

where $W$ is the whole of the system and consists of $n$ parts, and
$\\{p_{i}\\}_{i=1\cdots n}$ is the $i$-th part. In 1972, Philip W. Anderson
highlighted the idea of emergence in has article “More is Different” (?) in
which he stated that a change of scale very often causes a qualitative change
in the behavior of the system. For example, in human brains, when one examines
a single neuron, there is nothing that suggests conscious. But a collection of
millions of neurons is clearly able to produce wonderful consciousness.

The mechanisms behind the emergence of complexity can be used to design neural
network structures. One of the widely accepted reasons is the repeated
application and combination of two complementary forces or operations:
stretching and folding (in Physics term (?)), splitting and merging (in
Computer Science term (?)), or specialization and cooperation (in Sociology
term). Merging or aggregating of agents means generally a number of
(sub-)agents is aggregated or conglomerated into a single agent. Splitting or
specializing means the agents are clearly separated from each other and each
agent is constrained to a certain class or role (?).

#### Recurrent Neural Networks at the Edge of Chaos

Recurrent Neural Networks (RNNs) (?; ?) are a class of deep neural networks
that possess internal short-term memory due to recurrent feed-back connections
between units, which makes them be able to process arbitrary sequences of
inputs. Formally, given a sequence of vectors $\\{x_{t}\\}_{t=1\cdots T}$, the
equation of Simple RNN (?) is:

$\displaystyle h_{t}=f(Wx_{t}+Uh_{t-1})$ (2)

where $W$ and $U$ are parameter matrices, and $f$ denotes a nonlinearity
function such as tanh or ReLU. For simplicity, the neuron biases are omitted
from the equation.

Actually, RNNs can behave chaotically. There have been some works analysing
RNNs theoretically or experimentally from the perspective of systems theory.
(?) provided an exposition of research regarding system-theoretic aspects of
RNNs with sigmoid activation functions. (?) analyzed the computation at the
edge of chaos in RNNs and calculated the critical boundary in parameter space
where the transition from ordered to chaotic dynamics takes place. (?)
employed a dynamical systems perspective to understand the exploding gradients
and vanishing gradients problems in RNNs. In this paper, we obtain
methodologies from systems theory to conduct structure designs of RNN related
models.

## Network of Recurrent Neural Networks

### Overall Architecture

Figure 1: Overview of NOR structure.

(a) MA-NOR layer.

(b) Another MA-NOR layer.

(c) MS-NOR layer.

(d) SS-NOR layer.

Figure 2: The sectional views of NOR layers at one time-step. “I” means the
component I, “O” means the component O, and “R” means RNN neuron.

As the high-level illustration shown in Figure 1, NOR architecture is a three-
dimensional spatial-temporal structure. We summarize NOR architecture as four
components: I, M, S and O, in which component I (input) and O (output) control
the head and tail of NOR layer, component S (subnetworks) is in charge of the
spatial extension and component M (memories) is responsible for the temporal
extension of the whole structure. We describe each component as follows:

Component I: Component I controls the head of NOR architecture. It does data
preprocessing tasks and distributes processed input data to subnetworks. At
each time-step $t$, the form of upcoming input data $x_{t}$ may be various,
such as one single vector, or several vectors with the multi-granularity
information, even the feature vectors with noise. One single vector may be the
simplest situation, and the common solution is copying this vector into $n$
duplicates and feed each of them into one single subnetwork in the component
S. In this paper, the copying method meets our needs and we formalize it as:

$\displaystyle\\{x_{t}^{i}\\}_{i=1\cdots n}=C(x_{t})$ (3)

in which $C$ means copy function, and $x_{t}^{i}$ will be fed into $i$-th
subnetwork.

Component M: Component M manages all memories over the whole layer, not only
internal but also external memories (?). But in this paper, component M only
considers internal memory and do not apply any extra processing to the
individual memory of each RNN neuron. That is:

$\displaystyle m_{t}^{j}=I(o_{t-1}^{j})$ (4)

where $I$ means identity function, the superscript $j$ is the identifier of
$j$-th RNN neuron 111The notation of the “subnetwork” is different from the
“neuron”, for one subnetwork may be composed of several neurons. We use
superscript $i$ as the identifier of the subnetwork. So, the input data of
$i$-th subnetwork at time-step $t$ is denoted as $x_{t}^{i}$. , $m_{t}^{j}$ is
the memory of $j$-th RNN neuron at time-step $t$ and $o_{t-1}^{j}$ is the
transformation output of $j$-th RNN neuron at time-step $t-1$222In this paper,
we just use simple RNN (?) applied with ReLU activation as our basic RNN
neuron. Thus the memory at this time-step is just the output at last time-
step..

Component S: Component S is made up of several different or same subnetworks.
Interaction may exist in these subnetworks. The responsibility of component S
is to manage the logic of each subnetwork and handle the interaction between
them. Suppose component S has $n$ in-degree and $m$ out-degree, i.e.,
component S receives $n$ inputs and produces $m$ outputs, the $k$-th output is
generated by necessary inputs and memories:

$\displaystyle s_{t}^{k}=f([X,M])$ (5)

where $s_{t}^{k}$ is the $k$-th output at time-step $t$, $X$ and $M$ are
needed inputs and memories, and $f$ is the nonlinear function which can be
one-layer RNN or two-layer RNN etc.

Component O: To form a layer we need a certain amount of neurons. So one of
the NOR properties is multiple RNNs. A natural approach to integrate multiple
RNN neurons’ signals is collecting all outputs first and then using a MLP
layer to measure the weights of each outputs. Traditional neuron outputs a
single real value, so the collection method is directly arranging them into a
vector. But RNN neurons is different, for each of them outputs a vector not a
value. A simple method is concatenating all vectors and then connecting the
concatenated vector to the next MLP. Another is pooling each RNN output vector
into a real value, then arranging all these real values into a vector, which
seems same as traditional neurons. In this paper, the former solution is used
and formalized as:

$\displaystyle s_{t}=[s_{t}^{1};s_{t}^{2};\cdots;s_{t}^{m}]$ (6)
$\displaystyle o_{t}=r(W_{MLP}*s_{t})$ (7)

where $s_{t}$ is the concatenated vector, $W_{MLP}$ is the weight of MLP and
$r$ means the ReLU activation function of MLP.

### Methodology I: Aggregation

Any operation with changing a boundary can cause a emergence of complexity.
The natural boundary is the agent itself, and sudden emergence of complexity
is possible at this boundary if complexity is transfered from the agent to the
system or vice versa from the system to the agent. There are two basic
operations, aggregation and specialization, that can be used to transfer
complexity between different dimensions (?).

According to Arthur’s second mechanism, internal complexity can be increased
by aggregation and composition of sub-agents, which means a number of RNN
agents is conglomerated into a single big system. In this way, aggregation and
composition transfer traditional RNN from the outer to the inner dimension,
from the system to the agent, for the selected RNNs are accumulated to become
a part of a larger group.

For a concrete NOR layer, suppose it is composed of $n$ subnetworks, and
$i$-th subnetwork is made up of $k^{i}$ RNN neurons. Then, at the time-step
$t$, given the input $x_{t}$, the operation flow is as follows:

  1. 1.

Component I: copy $x_{t}$ into $n$ duplications using equation (3), then we
get $x_{t}^{1},x_{t}^{2},\cdots,x_{t}^{n}$.

  2. 2.

Component M: deliver the memory of each RNN neuron from the last time-step to
the current time-step using equation (4), then we get memories
$m_{t}^{1,1},\cdots,m_{t}^{1,k^{1}}$ in first subnetwork and memories
$m_{t}^{2,1},\cdots,m_{t}^{2,k^{2}}$ in second subnetwork, etc.

  3. 3.

Component S: for each subnetwork $i$, take advantage of the input $x_{t}^{i}$
and memories $m_{t}^{i,1},\cdots,m_{t}^{i,k^{i}}$ to get the nonlinear
transformation output:

$\displaystyle s_{t}^{i}=f([x_{t}^{i},m_{t}^{i,1},\cdots,m_{t}^{i,k^{i}}])$
(8)

then, we get $s_{t}^{1},s_{t}^{2},\cdots,s_{t}^{n}$.

  4. 4.

Component O: concatenate all outputs by equation (6) and use a MLP function to
determine how much signals in each subnetwork to flow through the component O
by equation (7).

Obviously, the number, the type and the interaction of the aggregated RNNs
determine the internal structure or inner complexity of the newly formed layer
system. Thus, we propose three kinds of topologies of NOR aggregation method.

#### Multi-Agent

In systems theory, the natural description of complex system is the multi-
agent system created by replication and adaptation. “Replication” means to
copy and reproduce a new RNN agent, and “adaptation” means they are not
totally same and some changes on weights or somewhere else by variation can
increase the diversity of the system.

As shown in Figure 2(a), there is a NOR layer (called MA-NOR) composed of four
parallel RNNs. Figure 3 shows this layer being unrolled into a full network.
Each subnetwork of MA-NOR layer is a one-tier RNN, thus at time-step $t$, the
$i$-th subnetwork of component S in MA-NOR is calculated as:

$\displaystyle s_{t}^{i}=o_{t}^{i}=r(W_{i}x_{t}^{i}+U_{i}m_{t}^{i})$ (9)

where $r$ means ReLU activation function, $W_{i}$ and $U_{i}$ are parameters
of corresponding RNN neuron, $o_{t}^{i}$ is the nonlinear transformation
output and will be delivered to next time-step to be used as $m_{t+1}^{i}$,
and $s_{t}^{i}$ is the output of $i$-th subnetwork which is equal to
$o_{t}^{i}$.

Figure 3: The unfolding of MA-NOR in three time-steps.

The nonlinear function in equation (5) of each subnetwork may be more complex.
For example, Figure 2(b) shows a NOR layer made up of three two-tier RNNs. At
time-step $t$, the $i$-th subnetwork in component S is calculated as

$\displaystyle o_{t}^{i,1}=r(W_{t}^{i,1}x_{t}^{i}+U_{t}^{i,1}m_{t}^{i,1})$
(10) $\displaystyle s_{t}^{i}=$ $\displaystyle
o_{t}^{i,2}=r(W_{t}^{i,2}x_{t}^{i}+U_{t}^{i,2}m_{t}^{i,2})$ (11)

#### Multi-Scale

Figure 4: The unfolding of MS-NOR in three time-steps.

The combination of multiple RNNs in a Multi-Agent NOR layer makes it somewhat
like an ensemble. And empirically, diversity among the members of a group of
agents is deemed to be a key issue in ensemble structure (?). One way to
increase the diversity is to use the Multi-Scale topology which introduce new
agent type to the system and can learn sequence dependencies in different
timescales.

Figure 2(c) shows a NOR layer made up of four subnetworks, in which two of
them are one-tier RNNs and the others are two-tier RNNs. Two kinds of
timescale dependencies are learned in component S, which are formalized as
follows:

$\displaystyle s_{t}^{1}=$ $\displaystyle
o_{t}^{1}=r(W_{t}^{1}x_{t}^{1}+U_{t}^{1}m_{t}^{1})$ (12) $\displaystyle
s_{t}^{2}=$ $\displaystyle o_{t}^{2}=r(W_{t}^{2}x_{t}^{2}+U_{t}^{2}m_{t}^{2})$
(13) $\displaystyle
o_{t}^{3,1}=r(W_{t}^{3,1}x_{t}^{3}+U_{t}^{3,1}m_{t}^{3,1})$ (14)
$\displaystyle s_{t}^{3}=$ $\displaystyle
o_{t}^{3,2}=r(W_{t}^{3,2}o_{t}^{3,1}+U_{t}^{3,2}m_{t}^{3,2})$ (15)
$\displaystyle o_{t}^{4,1}=r(W_{t}^{4,1}x_{t}^{4}+U_{t}^{4,1}m_{t}^{4,1})$
(16) $\displaystyle s_{t}^{4}=$ $\displaystyle
o_{t}^{4,2}=r(W_{t}^{4,2}o_{t}^{4,1}+U_{t}^{4,2}m_{t}^{4,2})$ (17)

#### Self-Similarity

The above mentioned aggregation and composition operation lead to big RNN
agent-groups. While in turn, they can also be combined to form even bigger
group. Such repeated aggregation and high accumulation makes the fractal and
self-similar structure come into being.

Figure 5: The unfolding of SS-NOR in three time-steps.

As shown in Figure 2(d), we also use three paths. But after each path first
learns its own intermediate representation, the second layers gather all
intermediate representations of three paths to learn high-level abstract
features. In this way, different paths do not learn and train independently.
The connections among each other helps the model easy to share informations.
Thus, it becomes possible that the whole model learns and trains to be an
organic rather than parallel independent structure. We formalize the
cooperation of component S as follows:

$\displaystyle o_{t}^{1,1}=r(W_{t}^{1,1}x_{t}^{1}+U_{t}^{1,1}m_{t}^{1,1})$
(18) $\displaystyle
o_{t}^{2,1}=r(W_{t}^{2,1}x_{t}^{2}+U_{t}^{2,1}m_{t}^{2,1})$ (19)
$\displaystyle o_{t}^{3,1}=r(W_{t}^{3,1}x_{t}^{3}+U_{t}^{3,1}m_{t}^{3,1})$
(20) $\displaystyle s_{t}^{1}=$ $\displaystyle
o_{t}^{1,2}=r(W_{t}^{1,2}[o_{t}^{1,1},o_{t}^{2,1},o_{t}^{3,1}]+U_{t}^{1,2}m_{t}^{1,2})$
(21) $\displaystyle s_{t}^{2}=$ $\displaystyle
o_{t}^{2,2}=r(W_{t}^{2,2}[o_{t}^{1,1},o_{t}^{2,1},o_{t}^{3,1}]+U_{t}^{2,2}m_{t}^{2,2})$
(22) $\displaystyle s_{t}^{3}=$ $\displaystyle
o_{t}^{3,2}=r(W_{t}^{3,2}[o_{t}^{1,1},o_{t}^{2,1},o_{t}^{3,1}]+U_{t}^{3,2}m_{t}^{3,2})$
(23)

### Methodology II: Specialization

We have mentioned that the emergence of complexity is usually connected to a
transfer of complexity, a transfer at the boundary of the system. Aggregation
and composition transfer complexity from the system to the agent, and from the
outer dimension to the inner dimension. Another way to be used to cross the
agent boundary is the specialization or inheritance, which transfer complexity
from the agent to the system, and from the inner dimension to the outer
dimension (?). Specialization is related to Arthur’s first mechanism. It
increases structural sophistication outside of the agent by adding new agent
forms. Through inheritance and specialization objects become objects of a
certain class and agents become agents of a certain type, and the more such an
agent becomes a particular class or type, the more it needs to delegate
special tasks that it can not handle alone to other agents (?).

The effect of specialization is the emergence of delegation and division of
labor in the newly formed groups. Thus, the formalization of $k$-th output in
Component S can be rewritten as the following:

$\displaystyle s_{t}^{k}=g(f_{1}([X,M]),f_{2}([X,M]),\cdots,f_{L}([X,M]))$
(24)

where $f_{l}$ is the $l$-th specialized agent function, $g$ means the
cooperation of all specialized agents, and $L$ is the number of specialized
agents. Equation (24) denotes the function $f$ in equation (5) is implemented
by the separated operations $f_{1},f_{2},\cdots,f_{L}$ and $g$.

#### Gate-Specialization

Figure 6: Gate Specialization. Figure 7: The sectional views of Gate-NOR layer
at one time-step.

We see gate mechanism is one of the specialization methods. As shown in Figure
6, a general RNN agent is separated into two specialized RNN agents, one is
for gate duty and the other is for generalization duty. A concrete Gate-NOR is
shown in Figure7. In the original Multi-Agent NOR layer, each RNN agent is
specialized as one generalization specific RNN and one gate specific RNN. We
formalize it as:

$\displaystyle o_{t}^{1}=\sigma(W_{t}^{1}x_{t}^{1}+U_{t}^{1}m_{t}^{1})$ (25)
$\displaystyle o_{t}^{2}=r(W_{t}^{2}x_{t}^{2}+U_{t}^{2}m_{t}^{2})$ (26)
$\displaystyle s_{t}^{1}=o_{t}^{1}\odot o_{t}^{2}$ (27) $\displaystyle
o_{t}^{3}=\sigma(W_{t}^{3}x_{t}^{3}+U_{t}^{3}m_{t}^{3})$ (28) $\displaystyle
o_{t}^{4}=r(W_{t}^{4}x_{t}^{4}+U_{t}^{4}m_{t}^{4})$ (29) $\displaystyle
s_{t}^{2}=o_{t}^{3}\odot o_{t}^{4}$ (30)

where $\sigma$ denotes the sigmoid activation and $\odot$ denotes element-wise
multiplication.

Task | # of Params. | IRNN | GRU | LSTM | MA-NOR | MS-NOR | SS-NOR | Gate-NOR
---|---|---|---|---|---|---|---|---
Sentiment Classification | 200 k | 212 | 107 | 88 | 89 | 66 | 61 | 61
400 k | 320 | 166 | 139 | 136 | 100 | 90 | 97
800 k | 468 | 252 | 213 | 203 | 149 | 132 | 149
Question classification | 100 k | 198 | 86 | 68 | 74 | 54 | 53 | 45
200 k | 319 | 148 | 119 | 122 | 88 | 83 | 79
400 k | 497 | 244 | 199 | 193 | 139 | 126 | 133
Named Entity Recognition | 200 k | 197 | 86 | 67 | 74 | 54 | 53 | 45
400 k | 319 | 148 | 119 | 122 | 88 | 83 | 79
800 k | 497 | 244 | 199 | 193 | 139 | 126 | 133
Table 1: Number of hidden neurons for RNN, GRU, LSTM MA-NOR, MS-NOR, SS-NOR
and Gate-NOR for each network size specified in terms of the number of
parameters (weights).

#### Relationship with LSTM and GRU

We see Long Short-term Memory (LSTM) (?) and Gated Recurrent Unit (GRU) (?) as
two special cases of Network of Recurrent neural networks. Take LSTM for
example, at time-step $t$, given input $x_{t}$ and previous memory cell
$c_{t-1}$ and hidden state $h_{t-1}$, the transition equations of standard
LSTM can be expressed as the following:

$\displaystyle i=\sigma(W_{i}x_{t}+U_{i}h_{t-1})$ (31) $\displaystyle
f=\sigma(W_{i}x_{t}+U_{f}h_{t-1})$ (32) $\displaystyle
o=\sigma(W_{o}x_{t}+U_{o}h_{t-1})$ (33) $\displaystyle
g=tanh(W_{g}x_{t}+U_{g}h_{t-1})$ (34) $\displaystyle c_{t}=c_{t-1}\odot
f+g\odot i$ (35) $\displaystyle s_{t}=tanh(c_{t})\odot o$ (36)

From the perspective of NOR (Network of Recurrent Neural Networks), LSTM is
made up of four RNNs, in which three of four (i.e., $i$, $f$, $o$ RNNs) are
specialized for gate tasks to control how much of informations let through in
different parts. Moreover, there is only a shared memory $h_{t-1}$ which can
be accessed by each RNN cell in LSTM.

While in turn, LSTM and GRU can also be combined to form even bigger group.

## Experiments

In order to evaluate the performance of the presented model structures, we
design experiments on the following tasks: sentiment classification, question
type classification and named entity recognition. We compare all models under
the comparable parameter numbers to validate the capacity of better utilizing
the parametric space. In order to verify the effectiveness and universality of
the experiments, we conduct three comparative tests under total parameters of
different orders of magnitude, see Table 1. Every experiment is repeated 20
times with different random initializations and then we report the mean
results. It’s worthy noting that our aim here is to compare the model
performance under the same hyper-parameter settings, not to achieve best
performance for one single model.

(?) showed that when initializing the recurrent weight matrix to be the
identity matrix and biases to be zero, simple RNN composed of ReLU activation
function (named as IRNN) can be comparable with even outperform LSTM. In our
experiments, all basic RNN neurons are simple RNNs applied with ReLU function.
We also keep the number of the hidden units same over all RNN neurons in a NOR
model. Obviously, our baseline model is a single giant simple RNN (?) applied
with ReLU activation. At the same time, two improved RNNs (GRU (?) and LSTM
(?)) have been widely and successfully used in NLP in recent years, so we also
choose them as our baseline models.

The pre-trained 300-D Glove 6B
vectors333https://nlp.stanford.edu/projects/glove/ and 300-D Google News
vectors444https://code.google.com/archive/p/word2vec/ were obtained for the
word embeddings. During training we fix all word embeddings and learn only the
other parameters in all models. The embeddings for out-of-vocabulary words are
set to zero vectors. We pad or crop the input sentences to a fixed length. The
trainings are done through stochastic gradient optimizer descent over shuffled
mini-batches with the optimizer Adam (?). All models are regularized by using
dropout (?) method. At the same time, in order to avoid overfitting, early
stopping is applied to prevent unnecessary computation when training. More
details on hyper-parameters setting can be found in our codes, which are
publicly available at $\ddagger$.

### Sentiment Classification

We evaluate our models on the task of sentiment classification on the popular
Stanford Sentiment Treebank (SST) benchmark (?), which consists of 11855 movie
reviews and is split into train (8544), dev (1101) and test (2210). SST
provides detailed phrase-level annotation and all sentences along with the
phrases are annotated with 5 labels: very positive, positive, neural,
negative, and very negative. In our experiments, we only use the sentence-
level annotation. One of our goals is to avoid expensive phrase-level
annotation, like (?). Another is, in practice, phrase-level annotation is hard
to provide.

All models use the same architecture: embedding layer $\rightarrow$ dropout
layer $\rightarrow$ RNN/NOR layer $\rightarrow$ RNN/NOR layer $\rightarrow$
max-pooling layer $\rightarrow$ dropout Layer $\rightarrow$ softmax layer. The
first layer is the word embedding layer, next are two-layer RNN/NOR layers as
the non-linear feature transformation layer. Then a max-pooling layer max-
pools all transformed feature vectors by selecting the max value in each
position to get sentence representation. Finally, a softmax layer is used as
output layer to get the final result. To benefit from the regularization, two
dropout layers with rate of 0.5 are added after embedding layer and before
softmax layer. The initial learning rates of all models are set to 0.0002. We
use public available 300-D Glove 840B vectors to initialize word embeddings.
Three different network sizes are tested for each architecture, such that the
number of parameters are roughly 200 k, 400 k and 800 k (see Table 1). We set
the minibatch size as 20. Finally, we use the Cross-Entropy criterion as loss
function.

Model | 200k Params | 400k Params | 800k Params
---|---|---|---
IRNN | 44.30 | 44.43 | 45.28
GRU | 47.27 | 47.35 | 47.65
LSTM | 46.98 | 47.19 | 47.37
MA-NOR | 45.17 | 45.24 | 45.37
MS-NOR | 44.83 | 45.51 | 45.59
SS-NOR | 44.54 | 45.21 | 45.42
Gate-NOR | 44.95 | 45.80 | 46.06
Table 2: Accuracy (%) comparison over different experiments on SST corpus.

The results of the experiments are shown in Table 2. It is obvious that NOR
models get superior performances compared with IRNN baseline, especially when
the network size is big enough. All models improve with network size grows.
Among all NOR models, Gate-NOR gets the best results. However, we find that
LSTM and GRU get much better results in three comparative tests.

### Question Type Classification

Question classification is an important step in a question answering system
which classifies a question into a specific type. For this task, we use TREC
(?) benchmark, which divides all questions into 6 categories: location, human,
entity, abbreviation, description and numeric. TERC provides 5452 labeled
questions in the training set and 500 questions in the test. We randomly
select 10% of the training data as the validation set.

Model | 100k Params | 200k Params | 400k Params
---|---|---|---
IRNN | 92.73 | 93.22 | 93.62
GRU | 91.83 | 92.54 | 93.64
LSTM | 91.44 | 92.46 | 93.10
MA-NOR | 92.73 | 93.56 | 93.73
MS-NOR | 92.85 | 93.79 | 94.04
SS-NOR | 93.19 | 93.62 | 94.12
Gate-NOR | 92.77 | 93.35 | 93.81
Table 3: Accuracy (%) comparison over different experiments on TREC corpus.

All network types use the same architecture: embedding layer $\rightarrow$
dropout layer $\rightarrow$ RNN/NOR layer $\rightarrow$ max-pooling layer
$\rightarrow$ dropout layer $\rightarrow$ softmax layer. Dropout rates are set
to 0.5. Three hidden layer sizes are chosen such that the total number of
parameters for the whole model is roughly 100 k, 200 k, 400 k, see Table 1.
All networks use a learning rate of 0.0005 and are trained to minimize the
Cross Entropy Error.

Table 3 shows the accuracy of the different networks on the question type
classification task. Here again, NOR models get better results than baseline
IRNN model. Among all NOR models, SS-NOR also gets the best result. In this
dataset, we find the performances of LSTM and GRU are even not comparable with
IRNN, which proves the validity of results in (?).

### Named Entity Recognition

Named entity recognition (NER) is a classic NLP task which tries to identity
the proper names of persons, organizations, locations, or other entities in
the given text. We experiment on CoNLL-2003 dataset (?) which consists of
14987 sentences in the training set, 3466 sentences in the validation set and
3684 sentences in the test set.

Model | 200k Params | 400k Params | 800k Params
---|---|---|---
IRNN | 85.07 | 85.52 | 85.58
GRU | 83.97 | 84.13 | 85.05
LSTM | 84.95 | 85.70 | 86.18
MA-NOR | 85.96 | 86.17 | 86.18
MS-NOR | 85.51 | 86.10 | 86.21
SS-NOR | 86.06 | 86.20 | 86.33
Gate-NOR | 85.67 | 85.89 | 86.21
Table 4: $F_{1}$ (%) comparison over different experiments on CoNLL-2003
corpus.

Recently, popular NER models are based on bidirectional LSTM (Bi-LSTM)
combined with conditional random fields (CRF), named as Bi-LSTM-CRF (?). The
Bi-LSTM-CRF networks can effectively use past and future features via a Bi-
LSTM layer and sentence level tag information via a CRF layer. In our
experiments, we also adapt this architecture by replacing LSTM with NORs or
other variation of RNNs. So the universal architecture of all tested models
is: embedding layer $\rightarrow$ dropout layer $\rightarrow$ Bi-RNN/Bi-NOR
layer $\rightarrow$ CRF layer. Three hidden layer sizes are chosen such that
the total number of parameters for the whole network is roughly 200 k, 400 k
and 800 k, see Table 1. We apply 50% dropout after embedding layer. Initial
learning rate is set to 0.005 and every epoch it is reduced by factor 0.95.
The size of each minibatch is 20. We train all networks for 25 epochs and
early stop the training when there are 5 epochs no improvement on validation
set.

Our results are summarized in the Table 4. Not surprisingly, all NORs perform
much better than giant single RNN-ReLU model. As we can see, GRU performs the
worst, followed by IRNN. Compared to GRU and IRNN, LSTM performs very well,
especially when network size grows up. At the same time, all NOR models get
superior performances than IRNN, GRU and LSTM. Among them, SS-NOR model get
best results.

## Conclusion

In conclusion, we introduced a novel kind of systems theory based neural
networks called “Network Of Recurrent neural network” (NOR) which views
existing RNNs (for example, simple RNN, GRU, LSTM) as high-level neurons and
then utilizes RNN neurons to design higher-level layers. Then we proposed
several methodologies to design different NOR topologies according to the
evolution of systems theory (?). We conducted experiments on three kinds of
tasks (including sentiment classification, question type classification and
named entity recognition) to evaluate our proposed models. Experimental
results demonstrated that NOR models get superior performances compared with
single giant RNN models, and sometimes their performances even exceed GRU and
LSTM.

## References

  * [Anderson 1972] Anderson, P. W.  1972\.  More is different.  Science 177 4047:393–6.
  * [Arthur and others 1993] Arthur, W. B., et al.  1993\.  On the evolution of complexity.  Technical report.
  * [Bertschinger and Natschläger 2004] Bertschinger, N., and Natschläger, T.  2004\.  Real-time computation at the edge of chaos in recurrent neural networks.  Neural computation 16(7):1413–1436.
  * [Chung et al. 2014] Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y.  2014\.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555.
  * [Elman 1990] Elman, J. L.  1990\.  Finding structure in time.  Cognitive science 14(2):179–211.
  * [Fromm 2004] Fromm, J.  2004\.  The emergence of complexity.  Kassel university press Kassel.
  * [Hannebauer 2002] Hannebauer, M.  2002\.  Autonomous dynamic reconfiguration in multi-agent systems: improving the quality and efficiency of collaborative problem solving.  Springer-Verlag.
  * [Hochreiter and Schmidhuber 1997] Hochreiter, S., and Schmidhuber, J.  1997\.  Long short-term memory.  Neural computation 9(8):1735–1780.
  * [Kingma and Ba 2014] Kingma, D., and Ba, J.  2014\.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * [Kuncheva and Whitaker 2003] Kuncheva, L. I., and Whitaker, C. J.  2003\.  Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy.  Machine learning 51(2):181–207.
  * [Lample et al. 2016] Lample, G.; Ballesteros, M.; Subramanian, S.; Kawakami, K.; and Dyer, C.  2016\.  Neural architectures for named entity recognition.  arXiv preprint arXiv:1603.01360.
  * [Le, Jaitly, and Hinton 2015] Le, Q. V.; Jaitly, N.; and Hinton, G. E.  2015\.  A simple way to initialize recurrent networks of rectified linear units.  arXiv preprint arXiv:1504.00941.
  * [Lehn 2002] Lehn, J.-M.  2002\.  Toward complex matter: supramolecular chemistry and self-organization.  Proceedings of the National Academy of Sciences 99(8):4763–4768.
  * [Li and Roth 2002] Li, X., and Roth, D.  2002\.  Learning question classifiers.  In Proceedings of the 19th international conference on Computational linguistics-Volume 1, 1–7.  Association for Computational Linguistics.
  * [Mele, Pels, and Polese 2010] Mele, C.; Pels, J.; and Polese, F.  2010\.  A brief review of systems theories and their managerial applications.  Service Science 2(1-2):126–135.
  * [Pascanu, Mikolov, and Bengio 2013] Pascanu, R.; Mikolov, T.; and Bengio, Y.  2013\.  On the difficulty of training recurrent neural networks.  In ICML.
  * [Qian, Huang, and Zhu 2016] Qian, Q.; Huang, M.; and Zhu, X.  2016\.  Linguistically regularized lstms for sentiment classification.  arXiv preprint arXiv:1611.03949.
  * [Socher et al. 2013] Socher, R.; Perelygin, A.; Wu, J. Y.; Chuang, J.; Manning, C. D.; Ng, A. Y.; Potts, C.; et al.  2013\.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, 1642.  Citeseer.
  * [Sontag 1997] Sontag, E.  1997\.  Recurrent neural networks: Some systems-theoretic aspects.  In Dealing with Complexity: a Neural Network Approach.  Citeseer.
  * [Srivastava et al. 2014] Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R.  2014\.  Dropout: a simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research 15(1):1929–1958.
  * [Thompson and Stewart 2002] Thompson, J. M. T., and Stewart, H. B.  2002\.  Nonlinear dynamics and chaos.  John Wiley & Sons.
  * [Tjong Kim Sang and De Meulder 2003] Tjong Kim Sang, E. F., and De Meulder, F.  2003\.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, 142–147.  Association for Computational Linguistics.
  * [Upton, Janeka, and Ferraro 2014] Upton, J.; Janeka, I.; and Ferraro, N.  2014\.  The whole is more than the sum of its parts: aristotle, metaphysical.  Journal of Craniofacial Surgery 25(1):59–63.
  * [Von Bertalanffy 1968] Von Bertalanffy, L.  1968\.  General system theory.  New York 41973(1968):40.
  * [Von Bertalanffy 1972] Von Bertalanffy, L.  1972\.  The history and status of general systems theory.  Academy of Management Journal 15(4):407–426.
  * [Walonick 1993] Walonick, D. S.  1993\.  General systems theory.  Information on http://www. statpac. org/walonick/systems-theory. htm.
  * [Werbos 1988] Werbos, P. J.  1988\.  Generalization of backpropagation with application to a recurrent gas market model.  Neural Networks 1:339–356.
  * [Weston, Chopra, and Bordes 2014] Weston, J.; Chopra, S.; and Bordes, A.  2014\.  Memory networks.  arXiv preprint arXiv:1410.3916.
  * [Wierzbicki 2015] Wierzbicki, A. P.  2015\.  Systems theory, theory of chaos, emergence.  In Technen: Elements of Recent History of Information Technologies with Epistemological Conclusions. Springer.  175–188.