11institutetext: EPFL, Lausanne, Switzerland 22institutetext: Google Research,
Zürich, Switzerland

# DSS: Synthesizing long Digital Ink using Data augmentation, Style encoding
and Split generation

Aleksandr Timofeev work done as a student researcher at Google Research,
Zürich, SwitzerlandThese authors contributed equally to this work and share
first authorship11 Anastasiia Fadeeva22footnotemark: 2 22 Andrei
Afonin11footnotemark: 1 11 Claudiu Musat 22 Andrii Maksai 22

###### Abstract

As text generative models can give increasingly long answers, we tackle the
problem of synthesizing long text in digital ink. We show that the commonly
used models for this task fail to generalize to long-form data and how this
problem can be solved by augmenting the training data, changing the model
architecture and the inference procedure. These methods use contrastive
learning technique and are tailored specifically for the handwriting domain.
They can be applied to any encoder-decoder model that works with digital ink.
We demonstrate that our method reduces the character error rate on long-form
English data by half compared to baseline RNN and by 16% compared to the
previous approach that aims at addressing the same problem. We show that all
three parts of the method improve recognizability of generated inks. In
addition, we evaluate synthesized data in a human study and find that people
perceive most of generated data as real.

###### Keywords:

digital ink online handwriting generative models length generalization.

##  1 Introduction

With the growing usage of tablets and styluses, handwriting is an increasingly
used human computer interaction (HCI) method. Recent developments in natural
language processing make the interaction increasingly bidirectional. In the
past handwriting was mostly used to facilitate the input of information for
the human user, making its recognition the primary digital ink task [6, 24].
With the advent of highly capable digital assistants [40] however, the
interactions are becoming ever more natural. One way to add to this immersive
HCI experience is to have the digital assistant respond in the same modality
as the user input — handwriting.

Handwriting synthesis is the process of converting printed text labels into
handwriting. Traditionally, this was proposed for user-facing features like
autocompletion and error correction [1]. These tasks all need to deal only
with short textual sequences — operating mostly on a word level. For instance,
in the case of error correction, only the misspelled word needs to be
replaced. Whereas, responses from recent text generative models can contain
multiple paragraphs of text. We propose methods to synthesize long inks to
accommodate those responses.

The primary trait of synthesized ink is that it needs to be readable in order
to be useful to a user. This observation leads to character error rate being
the evaluation of choice in the past work on synthesis [30, 7]. For
applications like autocompletion, similarity between the writer’s style and
generated ink is critical [30, 1]. In the current work we adopt
recognisability metric (CER) but the evaluation of stylistic similarity is
beyond the scope of this paper.

Some concerns about the synthesis quality are directly linked to the length of
the ink. Ensuring the stability of the generated ink style is one of them, as
it is needed to construct an immersive experience, where the generated ink is
consistent and looks real to people. The presence of artifacts is another, as
the number of artifacts is linked to the ink size. We tackle evaluations in
both a qualitative and quantitative way, to see which methods of generating
long inks are the best and whether they are good enough by human standards.

We propose three distinct methods of improving long generation. To extend the
ink generation to long inks we propose data augmentation to bridge the gap
between training and test conditions. To improve style consistency for long
ink generation we utilize contrastive learning for style transfer. To ensure
synthesis generalization to any input length we propose split generation. We
show their impact using two different synthesis model architectures — LSTM and
Transformer-based, on two ink representations — points and Bézier curves.

We compare ink synthesizer using the three improvements with both internal and
external baselines and we observe recognizability improvements ranging from
16% to 50%, depending on the architecture. To sum up,

  * •

we create a system that successfully synthesizes long digital ink 111A
notebook to test model inference is available here:
https://colab.research.google.com/drive/1SB_vyDcsdSq1CtE9IOD9opBR9IDgG0ly

  * •

we blend three different approaches: Data augmentation, Style encoder and
Split generation (we refer to it as DSS), resulting in large improvements in
ink recognizability across multiple ink representations and architectures

  * •

we perform an ablation study to quantify the additive individual impact of the
three proposed components

  * •

we run a user study to strengthen the quantitative analysis with a qualitative
one, showing that most of the synthetic ink is perceived as real

##  2 Related work

Handwriting synthesis has been a topic of interest for many years [13].
Digital ink can be represented in images which carry information about stroke
style, color, and width, as well as in the background they are drawn on [2,
15] or sequence of coordinates with time information [16]. Both of those
representations have their benefits. In this study we will focus on sequence-
based approaches.

Digital ink synthesis is a generation task where a model receives text as
input and outputs a sequence of coordinates that represents handwriting.
Methods in this field have evolved from parametric models with a set of
handmade style features [26] and sigma lognormal modeling [12] to deep neural
networks such as LSTM [16] and Transformers [42]. Sequence to sequence models
are frequently used for machine translation [39], speech recognition [19],
speech synthesis [43] and abstractive text summarization [31]. These models
can be applied to ink generation as well [16]. Mixture density network [5] are
commonly used as the last layer to generate coordinates or Bézier curves [38]
to capture a variety of possible next strokes.

Generation of personalized handwriting is of particular research interest due
to many appealing applications like spelling correction [1] and completion
[37]. In this setting a model has two inputs – text to generate and style ink.
It outputs an ink with the text in the given style. Style determines the
appearance of the output ink but doesn’t change the content that is written.
In the existing approaches, style is usually represented by a vector [1] or
sequence of embeddings [7]. In this work we adopt style transfer to guarantee
style consistency of long digital inks.

Previous studies in other domains suggest that deep neural networks tend to
degrade in quality on long-form inputs [3, 41]. For recurrent models LSTM [17]
block and attention mechanisms [42] were proposed to facilitate long sequence
modeling. However, even with those methods the gap in quality between training
and out-of-distribution lengths is still present for encoder-decoder RNN
models [22, 9].

For transformer models incorporating positional information together with long
data in training is a key to modelling long sequences [36, 21, 33]. In tasks
where long-form training data is unavailable, self-supervised pre-training can
help achieve state of the art performance across all lengths [10]. However,
this pre-training requires a vast amount of unlabeled data that is not
available in the handwriting domain.

While the previous methods focused on improvements in various architectures,
they all rely on the existence of these long sequences in the training data.
For the tasks where long training data is unavailable, data augmentation is
one way to improve quality for long sequences [32, 41]. In the case of speech
recognition, simulating long-form data improved the quality by 27% [32] and
for machine translation data augmentation boosted the performance by 30% [23].

In this paper we will evaluate common architectures for digital ink synthesis
on long-form data. We will also investigate the affect of possible
improvements like data augmentation and style conditioning on the out-of-
domain length quality.

##  3 Method

In this section we propose the methods that focus on improving generalization
of encoder-decoder to long sequences models in the digital ink domain. It is
especially important because handwriting data collection is complex as it
needs special equipment such as electronic whiteboard [27] or a tablet [1].
Thus, methods that address the issue without additional data collection are of
particular interest.

###  3.1 Data augmentation

One of possible approaches to improve performance on longer sequences is to
create synthetic long training examples. This idea was explored for speech
recognition where speech fragments from the same person were concatenated
together [28]. Our approach also concatenates inks but instead of samples from
the same writer we use stylistically similar inks. Algorithm 1 shows our
general method of long ink construction from the original training dataset
$D$. Thanks to uniform sampling in the algorithm 1 (denoted as $U$) we can get
a set of long inks by simply calling the function multiple times. Algorithm 1
receives the following inputs:

  * •

Style model $f$: receives two inks and outputs a score from $0$ to $1$ which
represents the similarity between the writing styles.

  * •

Concatenation function: receives two inks and outputs one ink which consists
of two parts (example in Fig. 1).

  * •

batch size $bs$ controls the diversity of the result dataset, with higher
values the probability of overlap between two long inks increases.

  * •

Similarity threshold $t$ controls the style consistency, with lower values the
chance to concatenate inks with different styles increases.

In the following two subsections we describe the style model and concatenation
function in more detail.

1

Input : training dataset $D$, target length $l$, batch size $bs$, threshold
$t$, style model $f$, concat function

Output : long ink $r$

2 $r\sim U(D)$;

3

4while  _$|r| <l$_ do

5 $\text{ink}_{0}$, …, $\text{ink}_{bs}$ $\sim U(D)$;

6 $\forall~{}ink\in\text{inks}$: $\text{similarity}(ink)=f(r,ink)$;

7

8 candidates = $\\{i\in\text{inks}~{}|~{}\text{similarities}(i)\geq t\
or~{}i=\arg\max\text{similarity}\\}$;

9 ink $\sim U(\text{candidates})$;

10 $r$=concat($r$, ink);

11

12 end while

13return $r$;

Algorithm 1 algorithm for generating a long sample Figure 1: Example of two
inks concatenation. In order to put inks on one line, we match the medians of
their histograms and scale the second ink to have the same height.

####  3.1.1 Style model

We train a model to have similar representations of different texts written in
the same style. We use the fact that most inks are stylistically consistent,
so we can use different parts of one ink as examples with different text but
similar style. To achieve this we use contrastive learning technique. We train
a classifier to distinguish between pieces of one ink versus pieces from
different inks. We adapt the SimCLR approach from [8] where two inputs $x_{i}$
and $x_{j}$ are mapped with the same function $g$ into two representations
$h_{i}$ and $h_{j}$ of a fixed size and then the similarity of these vectors
is computed:

$\text{similarity}(i,j)=\text{cos}(h_{i},h_{j})$

This similarity represents how close in style inks $x_{i}$ and $x_{j}$ are. To
generate training data for this task we randomly split existing inks into
three pieces – beginning part, random gap and final part (we randomly choose
two cut points in an ink). The random gap is used to generate multiple
training examples from one ink. During training we want a similarity of parts
from one ink to be close to $1$ and for parts from different inks – to be
around $0$. During inference we use a full ink to compute the style
representation and subsequently in similarity computation.

####  3.1.2 Ink concatenation

In order to obtain long samples we need a method to merge two inks into one
line. In case of misalignment, a training set may lose style consistency,
which can lead to model performance degradation. Concatenation can be achieved
using a baseline – imaginary line on which handwriting is written. We estimate
baselines for both inks and apply an rigid transformation to one of them.
There are many language specific baseline estimation methods [35], but for the
purposes of this study we use a more general method of horizontal projection,
shown in Fig. 1. During concatenation we add spaces in between inks which can
be adjusted for the languages without space separation like Chinese. Hereby,
we generate synthetic long samples from training examples with a similar
style.

###  3.2 Generation with style conditioning

Autoregressive models are known to suffer from accumulation of mistakes during
inference [4, 25]. This problem becomes even more pressing for generation of
long sequences. In order to overcome this, we explicitly add a style
conditioning into the ink generation model. Conditioning on style is commonly
used for style transfer [1, 7, 18], but we use it as a tool to mitigate past
mistakes in generation.

Figure 2: On the left – training of generative model with style conditioning.
On the right – split generation procedure with two words.

We propose to train an encoder-decoder model with style conditioning end-to-
end, as shown in Fig. 2. Encoder inputs the text label and the style model
receives an example of ink to generate a style representation vector which is
used in the decoding process. Training style model weights at the same time as
encoder-decoder parameters helps to adapt style information to a given
encoder-decoder model. For model optimisation we combine the negative log
likelihood which is typically used to train generative models with SimCLR loss
[8]:

$\mathscr{L}(x)=\mathscr{L}_{\text{NLL}}+\mathscr{L}_{\text{SimCLR}}=-\sum_{i\leq
n}\log\text{P}(x_{i}|x_{<i})+\mathscr{L}_{\text{SimCLR}}(x)$

The first part of this loss is a likelihood to correctly generate an ink $x$.
This part depends on encoder, decoder and style function $f$ weights, as we
use style embedding in the decoder’s input. The second part of this loss
ensures that function $f$ outputs style representation and doesn’t contain
information about written text. Similar to the previous section we use the
SimCLR approach from [8] where two inputs $x_{i}$ and $x_{j}$ are mapped with
the same function $f$ into two representations $h_{i}$ and $h_{j}$ of a fixed
size and then the similarity of these vectors is computed as
$\text{cos}(h_{i},h_{j})$. In $\mathscr{L}_{\text{SimCLR}}$ we generate from
ink $x$ two pieces $x_{:k}$ and $x_{l:}$ (for some random $k$ and $l$, $k<l$).
These pieces are supposed to have high similarity, whereas similarity between
$x_{:k}$ and the ink pieces from other inks should be low.

During training we use a piece of target ink as a style input, which can cause
target leaking as $\mathscr{L}_{\text{NLL}}$ uses the full target including
the style input. However, the use of SimCLR loss promotes independence of
style and the text content, therefore no additional steps are needed to
prevent target leaking.

During inference we use a separate set of inks for style extraction.
Additionally, we use a full ink in the style model rather than just a piece
and provide this embedding on each step of decoding as a reminder to the
model. This helps avoid the style drift, where the model slowly changes the
style of the writing until it becomes completely unrecognizable. Thus, we
incorporated a style model into the long ink synthesizer to improve style
consistency of long generation.

###  3.3 Split generation

We propose an inference technique for long sequences inspired by dynamic
overlapping inference in recognition [9]. The idea is to split a long input
into separate pieces and calculate their results separately. We split a label
at the word boundaries and the maximum number of words in one piece is a
hyperparameter of this method, shown in Fig. 2. This way we retain the
capability to generate cursive handwriting. The main idea of this method is to
find the input length that is optimal for model’s performance and use it in
the inference. However, the main drawback of generating ink pieces separately
is the fact that the result parts don’t share any information. An example is
shown in Fig. 4 random arrangement.

In case of a style mismatch between pieces, people can easily recognize that
the ink is synthetic. We propose to use the model with style conditioning from
the previous subsection in order to ensure style consistency of the individual
generated pieces, that we combine together to obtain the result ink. To merge
pieces together, we match the medians of their horizontal projections, as in
the data augmentation step Fig. 1. We don’t need to match their scale as they
are already similar due to style conditioning.

Mismatch in slope between pieces is also easily recognizable by humans (see
Fig. 3). To fix this problem we can sample multiple candidates and choose the
most horizontal among them using linear regression on the $y$ trajectory. This
problem occurs because training inks are not always written horizontally and
similar model behavior in split generation leads to mismatched slopes seen in
Fig. 3.

Figure 3: Reason to generate multiple samples and choose the most straight ink
in each part of Split Generation.

###  3.4 Combining proposed methods

The proposed methods are complementary to each other as they focus on
different parts of an encoder-decoder model. In the training phase, we propose
the following order of actions:

  1. 1.

train a style model with SimCLR loss on dataset $D$

  2. 2.

generate an augmented dataset $\hat{D}$ with longer inks from $D$

  3. 3.

train an encoder-decoder model with style conditioning on $D\cup\hat{D}$

The final result is the trained encoder-decoder model with style conditioning.
In the inference, we suggest the following sequence of steps:

  1. 1.

choose an ink for style extraction

  2. 2.

split input text by $n$ words

  3. 3.

run inference of the encoder-decoder model on each piece with the same style
input

  4. 4.

combine pieces into the result ink

##  4 Experiments

###  4.1 Setup

In this section we will apply our modifications to two architectures – RNN
[16] and Transformer [42]. A classic approach to ink synthesis [16] proposed a
multi-layer LSTM model with monotonic attention over the labels and gaussian
mixture model output for ink synthesis. In our RNN implementation the label
encoder is simply one-hot encoding of input characters with dictionary size
70. The decoder consists of an LSTM with 128 units, GMM monotonic attention
layer (size 128, one attention head and 10 components), two LSTM blocks with
256 units, a dense layer and an output GMM model with 10 components [16].

In the transformer model we use a standard encoder-decoder architecture with
an output GMM head similar to the RNN model. In both encoder and decoder we
use sinusoidal positional embeddings [42]. The transformer label encoder has 4
self-attention blocks (8 heads, 16 units per head, ReLU activation and drop-
out rate 0.1). The decoder has 6 standard cross-attention blocks (4 heads, 32
units per head, ReLU activation and drop-out rate 0.1) followed by two dense
layers with ReLU activation.

We train both models with batch size of 128 and the Adam optimizer. For the
RNN model we use learning rate 1e-4 and for transformer learning rate is 1e-3
with the schedule from [42]. We train each of these models with two different
ink representations – raw [16] and curve [11]. At inference time, all
transformers and the raw RNN models use random sampling of the GMM with bias
$\infty$ similar to [16]. For the curve RNN we use greedy sampling due to
better validation performance.

####  4.1.1 Style encoder

For style encoding we use a model that consists of one bidirectional LSTM
layer with 256 units, followed by a unidirectional LSTM layer with 256 units
that outputs the last state and three dense layers on top with sizes 256, 256
and 16. In order to condition a model on a given style we sum a style vector
with the decoder input.

####  4.1.2 Datasets

We train our models on the DeepWriting dataset which contains more than 34,000
handwritten samples in English [1]. Examples in the training dataset on
average consist of 12 characters and 2.8 words. To evaluate the quality of
long generation quantitatively and qualitatively we need a suitable source of
handwriting in English. For this purpose, we use IAMonDo which consists of
1,000 pages with English handwriting, diagrams and drawings in them [20]. We
extracted handwriting from those pages and split them into three categories –
long – more than 7 words, medium – 4-7 words and short – less than 4 words.
For short samples we chose a threshold to match the mean length in the
training set (ss shown in Table 1). We then split the rest of the inks into
two sets to see progression of quality from in-distribution to increasingly
out-of-distribution lengths. For clarity, we include the label sets with our
submission 222A notebook with test sets and model inference
https://colab.research.google.com/drive/1SB_vyDcsdSq1CtE9IOD9opBR9IDgG0ly.

Table 1: Dataset statistics for train, validation and test datasets. dataset |  | words | mean characters (std) | samples | characters
---|---|---|---|---|---
DeepWriting | train | 2.8 (1.2) | 12.41 (5.08) | 34K | 421K
valid | 2.8 (1.2) | 12.30 (4.97) | 680 | 8K
IAMonDO | long |  $>$ 7 | 48.22 (9.06) | 310 | 15K
medium | 4-7 | 30.52 (7.98) | 864 | 26K
short |  $<$ 4 | 10.91 (5.63) | 811 | 9K

####  4.1.3 Evaluation metric

Following [29] we measure the recognizability of generated inks with Character
Error Rate (CER) on a set of test labels. Our recognizer is an RNN model
trained on a private dataset described in [6]. It performs well on all lengths
present in the test sets, see the results on the original data in Table 10.

####  4.1.4 Baseline model

We compare our method to the VRNN model [1] which generates one character at a
time and propagates RNN states between characters to control the style. This
model easily generates long input text (see Table 2) as very limited
information is shared between the characters. Another result of this procedure
is that characters are rarely connected in the final handwriting. Thus, this
model struggles to generate cursive writing.

###  4.2 Quantitative results

In this section we apply proposed changes to two architectures with two
different ink representations – raw [16] and curve [11]. We train each model 3
times in order to compute the variance between runs. In Table 2 we present the
mean CER and standard deviation of the models without any changes and with
proposed improvements. Comparing baseline models we see that RNN models
degrade on long inputs less than transformers, which matches previous results
in the text domain [33]. It is expected as positional embeddings for remote
positions are unknown to the transformer model, hurting accuracy, while RNN
has monotonic attention which helps with longer sequences. However, even in
case of RNN models CER gets 2 times worse for curve setup and 5 times for raw
between short and long validation datasets. Thus, all four models can benefit
from better out-of-domain length generalisation.

Table 2: Recognizability comparison of our method with baseline models and the VRNN model [1]. Datasets: long > 7 words, medium 4-7 words, short < 4 words.  model |  | CER long | CER medium | CER short | CER avg
---|---|---|---|---|---
VRNN [1] |  | 5.94 (0.22) | 5.05 (0.04) | 4.8 (0.1) | 5.26
transformer curve | DSS | 3.96 (0.23) | 4.0 (0.37) | 5.3 (0.57) | 4.42
base | 60.62 (3.05) | 34.91 (3.36) | 10.35 (0.47) | 35.29
transformer raw | DSS | 12.73 (1.23) | 9.89 (1.61) | 8.83 (2.06) | 10.48
base | 64.34 (1.11) | 41.24 (2.15) | 11.4 (1.4) | 38.99
RNN curve | DSS | 6.6 (0.39) | 6.03 (0.95) | 7.08 (0.95) | 6.57
base | 12.25 (0.78) | 9.27 (0.93) | 5.29 (0.28) | 8.94
RNN raw | DSS | 4.42 (0.55) | 4.22 (0.51) | 4.56 (0.48) | 4.4
base | 21.48 (5.18) | 11.74 (3.18) | 4.11 (0.43) | 12.44

In this section we provide details on the hyperparameters utilized for our
results and the criteria for their selection. For each of four models we added
10K long samples built from train dataset and picked the optimal target length
$l$ based on validation quality presented in Table 12 (in the appendix). In
style condition during inference we used random examples from DeepWriting
validation dataset as a source of style. This ensured that we did not
introduce any stylistic bias compared to unconditional generation, as the
distribution of validation dataset matches the training dataset. In order to
determine the best number of words in the split generation we applied data
augmentation method to validation dataset. We matched the target length of
this dataset to the long test dataset. Then, we evaluated models with
additional data and style conditioning on the said dataset. Results are
presented in Table 12 (in the appendix).

In conclusion, our method significantly outperforms the baseline in all four
cases, with curve transformer as well as raw RNN showing the best overall
quality. This quality is also better than VRNN model [1] which requires
character segmentation in the training data. This information is absent in
most open source handwriting datasets [34]. In our results we show that
without any additional annotation, superior quality is attainable across all
three target lengths.

###  4.3 Ablation study

In this section we remove each of the three changes that we proposed from the
combination and measure the quality to show that all three changes contribute
to the optimal model performance. In addition, we do a more detailed ablation
study on the curve transformer setup, as it provides the best overall quality.

####  4.3.1 Impact of data augmentation.

In order to measure the impact of synthetic long data, we remove it from
training. All of the other parameters stay the same and we train a new set of
models only on the original data. This experiment is especially important
because split generation significantly reduces the input text length during
inference. However, data augmentation improves model performance on short
samples as well (see Table 12 in the appendix) possibly due to the moderate
size of the training set.

Table 3: Effect of data augmentation. models | data | CER long | CER medium | CER short | CER avg
---|---|---|---|---|---
transformer curve | yes | 3.96 (0.23) | 4.0 (0.37) | 5.3 (0.57) | 4.42
no | 6.61 (0.5) | 6.89 (0.29) | 7.71 (0.95) | 7.07
transformer raw | yes | 12.73 (1.23) | 9.89 (1.61) | 8.83 (2.06) | 10.48
no | 13.35 (0.59) | 11.42 (1.08) | 11.21 (1.23) | 11.99
RNN curve | yes | 6.6 (0.39) | 6.03 (0.95) | 7.08 (0.95) | 6.57
no | 6.78 (0.62) | 6.05 (0.64) | 6.85 (0.64) | 6.56
RNN raw | yes | 4.42 (0.55) | 4.22 (0.51) | 4.56 (0.48) | 4.4
no | 4.85 (0.6) | 4.66 (0.83) | 4.94 (1.04) | 4.82

In Table 3 we can see that for transformers augmented data plays a very
important role. The decrease in quality is almost 60% for curve transformer
and around 16% for raw transformer. As mentioned in the main results,
transformer models fail to generalize to long-form sequences due to constant
positional embeddings unseen at training time. The restriction is overcome
through the use of synthetic long data. However, for raw RNN model the loss is
also quite significant – around 10%, which shows that augmented data is
beneficial in most cases.

Figure 4: Style, random and same ink arrangements.

In DSS we propose to concatenate existing training examples based on their
style similarity. We compare this approach to repetition of the same ink many
times and concatenation of random samples into one ink. In Fig.4 we can see
that random arrangement doesn’t preserve style consistency which is an
important quality of digital inks. On the other hand, repetition of the same
ink returns stylistically consistent inks but lacks diversity in text. These
strategies decrease recognizability compared to style-based arrangement in the
curve transformer setup, as shown in Table 4.

Table 4: Different data augmentation strategies for curve transformer setup. data arrangement | CER long | CER medium | CER short | CER avg
---|---|---|---|---
style | 3.96 (0.23) | 4.0 (0.37) | 5.3 (0.57) | 4.42
random | 5.01 (0.52) | 4.96 (0.33) | 5.58 (0.49) | 5.18
repetition | 7.15 (0.34) | 7.02 (0.6) | 8.56 (0.57) | 7.58

####  4.3.2 Impact of style encoding

Table 5: Effect of style encoding. No style rows show metrics for stylistically inconsistent inks (see random arrangement in Fig. 4). In priming we prompt a model with the style ink before target ink generation. models | style | CER long | CER medium | CER short | CER avg
---|---|---|---|---|---
transformer curve | yes | 3.96 (0.23) | 4.0 (0.37) | 5.3 (0.57) | 4.42
prime | 6.34 (0.65) | 6.19 (0.7) | 7.39 (1.69) | 6.64
no | 10.02 (1.77) | 10.32 (1.04) | 7.73 (0.74) | 9.36
transformer raw | yes | 12.73 (1.23) | 9.89 (1.61) | 8.83 (2.06) | 10.48
prime | 17.92 (1.99) | 23.06 (2.31) | 18.74 (2.95) | 19.91
no | 13.15 (2.55) | 10.16 (2.56) | 9.81 (2.21) | 11.04
RNN curve | yes | 6.6 (0.39) | 6.03 (0.95) | 7.08 (0.95) | 6.57
prime | 7.49 (0.63) | 6.85 (0.3) | 7.69 (0.69) | 7.34
no | 5.14 (0.95) | 4.45 (0.96) | 4.42 (0.95) | 4.67
RNN raw | yes | 4.42 (0.55) | 4.22 (0.51) | 4.56 (0.48) | 4.4
prime | 4.49 (1.46) | 4.25 (1.28) | 4.08 (1.13) | 4.27
no | 3.66 (0.2) | 3.59 (0.86) | 2.16 (0.45) | 3.14

Next, we remove style encoder from encoder-decoder architecture, leaving
everything else unchanged. In Table 5 we show results for split generation
where we generate label pieces independently and don’t share any information
between them (rows with no style). In this case split generation returns
stylistically inconsistent inks similar to random data augmentation as shown
in Fig. 4. In a qualitative evaluation, we show that stylistically
inconsistent inks are 33% less likely to be recognized as real. For RNN models
character error rate is better without style than with style conditioning, but
it is not the case for transformer models where CER gets significantly higher
without any style – 110% for the curves setup.

We also compare these results to a different method of ensuring style
consistency, proposed in [16]. In split generation we can prime a model on the
style ink before generating each piece by teacher-forcing a style input and
asking a model to complete it with input text. This way we get result ink with
similar style to the style ink. We implicitly determine consistency of the
result by using the same style ink in each piece of split generation. The main
disadvantage of this approach is that by doing a completion of style ink we
increase the target ink length which may result in subpar quality. In Table 5
we can see that for curve transformer completion decreases CER by 30% compared
to no style and for RNN raw the quality of completion is similar to style
encoding.

####  4.3.3 Impact of split generation

Finally, we evaluate models quality without split generation. Table 6 shows
that models with fully autoregressive inference perform by 30-100% worse than
with split generation. The gap in quality is especially pronounced for long
evaluation sets as we split the labels there into many pieces. Whereas, the
quality on short data stays almost the same as in many cases we don’t split
targets there. However, raw RNN even without split generation still performs
similarly to VRNN (see Table 2).

Table 6: Effect of split generation. model | split | CER long | CER medium | CER short | CER avg
---|---|---|---|---|---
transformer curve | yes | 3.96 (0.23) | 4.0 (0.37) | 5.3 (0.57) | 4.42
no | 10.6 (1.42) | 10.6 (1.42) | 5.33 (0.53) | 8.84
transformer raw | yes | 12.73 (1.23) | 9.89 (1.61) | 8.83 (2.06) | 10.48
no | 22.44 (4.37) | 22.4 (2.04) | 9.88 (1.6) | 18.24
RNN curve | yes | 6.6 (0.39) | 6.03 (0.95) | 7.08 (0.95) | 6.57
no | 14.82 (2.87) | 10.28 (1.43) | 7.43 (0.97) | 10.84
RNN raw | yes | 4.42 (0.55) | 4.22 (0.51) | 4.56 (0.48) | 4.4
no | 7.32 (1.95) | 4.86 (0.95) | 4.54 (0.5) | 5.57

In Table 7 we show that quality can vary quite a bit depending on the number
of words in one split. It may seem that generating only one word at a time is
the most simple task and that would lead to the best performance. However, we
noticed that curve transformer fails to generate one word with punctuation at
the end probably due to lack of similar data in training. Thus, it is
important to choose number of words that is most suited for each model. This
value is consistent for curve transformer between different datasets: long
synthetic validation in table 12, long and medium validation in Table 7.

Table 7: Different number of words in split generation for curve transformer setup. n words | CER long | CER medium | CER short | CER avg
---|---|---|---|---
1 | 6.3 (0.67) | 5.96 (0.75) | 6.96 (0.97) | 6.41
2 | 4.66 (0.5) | 4.56 (0.33) | 5.62 (0.23) | 4.95
3 | 3.96 (0.23) | 4.0 (0.37) | 5.3 (0.57) | 4.42
5 | 4.53 (0.01) | 4.83 (0.2) | 5.3 (0.57) | 4.89

###  4.4 Qualitative evaluation

Figure 5: Examples of inks from human study.

To ensure that the generated samples are not only recognizable but also look
realistic, we performed a human evaluation. In it, we asked the participants
to evaluate a mix of generated and real inks – 100 each with long texts from
IAMonDO dataset, see examples in Fig. 5. Our model’s performance was evaluated
on long texts because baseline models struggled the most with generating long
samples (see Table 2). We used a curve transformer model with style encoding,
split generation of two words as the direction pattern (see Fig. 3) is less
pronounced there compared to three words. We also use the sampling of 5
candidates to pick the most horizontal ink. In Table 10 in the appendix we
show that picking the most horizontal pieces in split generation of two words
doesn’t significantly decrease recognition quality.

We study whether people can differentiate between real and generated data by
showing one ink at a time with a question Does this ink look real?. We
collected 600 responses from 12 people who has worked with digital ink before
(3 answers per ink). The results are presented in Table 8. The majority of
generated inks were labeled as real – 79%, as a result the F1 score of human
answers is only 0.34. To check whether original inks are labeled real with
higher probability we use Fisher’s exact test [14]. We get the p-value equal
to 0.04. Thus, we can reject the null hypothesis and conclude that original
inks are more likely to be labeled real than generated ones. To sum up,
generated inks are frequently perceived as real but there is still a
statistically significant difference between the two datasets.

Table 8: Human study results. data | look real | consistent style | no artifacts | readable
---|---|---|---|---
real | 0.89 | 0.96 | 0.77 | 0.8
generated | 0.79 | 0.85 | 0.7 | 0.77

Moreover, we asked about the style consistency, presence of artifacts like
additional lines or dots and readability of the ink see in Fig. 6. In the
latter two cases the gap between real and original quality is not as
pronounced as in the first two and is not statistically significant. We see
that according to people real and generated inks have similar readability
which matches our quantitative results (see tables 10 and 10). It is also
important to note that generated inks with artifacts are 42% less likely to be
perceived as real and for style inconsistency this number is 33%.

Figure 6: Human study questionnaire on the left and examples of synthesizer’s
mistakes on the right. The mistakes are marked with rectangles.

##  5 Conclusion

We have presented three improvements to encoder-decoder models for digital ink
generation – data augmentation, style conditioning and split generation. We
have been able to decrease character error rate compared to baseline RNN model
by 50% (baseline RNN curve 8.94 vs DSS transformer curve 4.42) and by 16%
compared to the previous approach. We’ve shown that all three proposed methods
play an important part in optimal model performance: data augmentation is very
important for transformer models, style conditioning has higher
recognisability than priming in split generation, split generation has the
biggest impact on quality among the three parts of our approach. In addition,
we have conducted a qualitative evaluation where we have verified the quality
of generated long inks. People perceive most of synthesized inks as real, but
there is still a statistically significant discrepancy between answers in real
and generated buckets. We believe that our findings regarding long ink
synthesis can be used in real world applications and as a stepping stone for
future research in this field.

## References

  * [1] Aksan, E., Pece, F., Hilliges, O.: DeepWriting: Making Digital Ink Editable via Deep Generative Modeling. In: SIGCHI Conference on Human Factors in Computing Systems. CHI ’18, ACM, New York, NY, USA (2018)

  * [2] Alonso, E., Moysset, B., Messina, R.O.: Adversarial generation of handwritten text images conditioned on sequences. 2019 International Conference on Document Analysis and Recognition (ICDAR) pp. 481–486 (2019)
  * [3] Anil, C., Wu, Y., Andreassen, A.J., Lewkowycz, A., Misra, V., Ramasesh, V.V., Slone, A., Gur-Ari, G., Dyer, E., Neyshabur, B.: Exploring length generalization in large language models. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022), https://openreview.net/forum?id=zSkYVeX7bC4
  * [4] Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N.M.: Scheduled sampling for sequence prediction with recurrent neural networks. ArXiv abs/1506.03099 (2015)
  * [5] Bishop, C.M.: Mixture density networks (1994)
  * [6] Carbune, V., Gonnet, P., Deselaers, T., Rowley, H., Daryin, A., Calvo, M., Wang, L.L., Keysers, D., Feuz, S., Gervais, P.: Fast multi-language lstm-based online handwriting recognition. International Journal on Document Analysis and Recognition (IJDAR) (2020)
  * [7] Chang, J.H.R., Shrivastava, A., Koppula, H.S., Zhang, X., Tuzel, O.: Style equalization: Unsupervised learning of controllable generative sequence models (2022), https://arxiv.org/abs/2110.02891
  * [8] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations (02 2020)
  * [9] Chiu, C.C., Narayanan, A., Han, W., Prabhavalkar, R., Zhang, Y., Jaitly, N., Pang, R., Sainath, T., Nguyen, P., Cao, L., Wu, Y.: Rnn-t models fail to generalize to out-of-domain audio: Causes and solutions. pp. 873–880 (01 2021). https://doi.org/10.1109/SLT48900.2021.9383518
  * [10] Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context (2019). https://doi.org/10.48550/ARXIV.1901.02860, https://arxiv.org/abs/1901.02860
  * [11] Das, A., Yang, Y., Hospedales, T., Xiang, T., Song, Y.Z.: BezierSketch: A Generative Model for Scalable Vector Sketches, pp. 632–647 (11 2020). https://doi.org/10.1007/978-3-030-58574-7_38
  * [12] Djioua, M., Plamondon, R.: An interactive system for the automatic generation of huge handwriting databases from a few specimens. 2008 19th International Conference on Pattern Recognition pp. 1–4 (2008)
  * [13] Elarian, Y., Abdel-Aal, R., Ahmad, I., Parvez, M., Zidouri, A.: Handwriting synthesis: Classifications and techniques. International Journal of Document Analysis and Recognition 17 (09 2014). https://doi.org/10.1007/s10032-014-0231-x
  * [14] Fisher, R.A.: On the interpretation of from contingency tables, and the calculation of p. Journal of the Royal Statistical Society 85 (1922), http://www.medicine.mcgill.ca
  * [15] Gan, J., Wang, W.: Higan: Handwriting imitation conditioned on arbitrary-length texts and disentangled styles. In: AAAI Conference on Artificial Intelligence (2021)
  * [16] Graves, A.: Generating sequences with recurrent neural networks. ArXiv abs/1308.0850 (2013)
  * [17] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9, 1735–80 (12 1997). https://doi.org/10.1162/neco.1997.9.8.1735
  * [18] Hsu, W.N., Zhang, Y., Weiss, R.J., Chung, Y.A., Wang, Y., Wu, Y., Glass, J.: Disentangling correlated speaker and noise for speech synthesis via data augmentation and adversarial factorization. In: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5901–5905 (2019). https://doi.org/10.1109/ICASSP.2019.8683561
  * [19] Huber, C., Hussain, J., Stüker, S., Waibel, A.H.: Instant one-shot word-learning for context-specific neural sequence-to-sequence speech recognition. 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) pp. 1–7 (2021)
  * [20] Indermühle, E., Liwicki, M., Bunke, H.: Iamondo-database: an online handwritten document database with non-uniform contents. In: International Workshop on Document Analysis Systems (2010)
  * [21] Ke, G., He, D., Liu, T.Y.: Rethinking the positional encoding in language pre-training (06 2020)
  * [22] Koehn, P., Knowles, R.: Six challenges for neural machine translation. In: Proceedings of the First Workshop on Neural Machine Translation. pp. 28–39. Association for Computational Linguistics, Vancouver (Aug 2017). https://doi.org/10.18653/v1/W17-3204, https://aclanthology.org/W17-3204
  * [23] Kondo, S., Hotate, K., Hirasawa, T., Kaneko, M., Komachi, M.: Sentence concatenation approach to data augmentation for neural machine translation. pp. 143–149 (01 2021). https://doi.org/10.18653/v1/2021.naacl-srw.18
  * [24] Krishnan, P., Jawahar, C.: Hwnet v2: An efficient word image representation for handwritten documents. International Journal on Document Analysis and Recognition (IJDAR) (2019)
  * [25] Lamb, A., Goyal, A., Zhang, Y., Zhang, S., Courville, A., Bengio, Y.: Professor forcing: A new algorithm for training recurrent networks (10 2016)
  * [26] Lin, Z., Wan, L.: Style-preserving english handwriting synthesis. Pattern Recognition 40(7), 2097–2109 (2007). https://doi.org/https://doi.org/10.1016/j.patcog.2006.11.024, https://www.sciencedirect.com/science/article/pii/S0031320306004985
  * [27] Liwicki, M., Bunke, H.: Iam-ondb - an on-line english sentence database acquired from handwritten text on a whiteboard. In: Eighth International Conference on Document Analysis and Recognition (ICDAR’05). pp. 956–961 Vol. 2 (2005). https://doi.org/10.1109/ICDAR.2005.132
  * [28] Lu, Z., Pan, Y., Doutre, T., Haghani, P., Cao, L., Prabhavalkar, R., Zhang, C., Strohman, T.: Input length matters: Improving rnn-t and mwer training for long-form telephony speech recognition (2021)
  * [29] Luo, C., Zhu, Y., Jin, L., Li, Z., Peng, D.: Slogan: Handwriting style synthesis for arbitrary-length and out-of-vocabulary text (2022). https://doi.org/10.48550/ARXIV.2202.11456, https://arxiv.org/abs/2202.11456
  * [30] Maksai, A., Rowley, H., Berent, J., Musat, C.: Inkorrect: Online handwriting spelling correction (02 2022)
  * [31] Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., Xiang, B.: Abstractive text summarization using sequence-to-sequence RNNs and beyond. In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. pp. 280–290. Association for Computational Linguistics, Berlin, Germany (Aug 2016). https://doi.org/10.18653/v1/K16-1028, https://aclanthology.org/K16-1028
  * [32] Narayanan, A., Prabhavalkar, R., Chiu, C.C., Rybach, D., Sainath, T.N., Strohman, T.: Recognizing long-form speech using streaming end-to-end models. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 920–927 (2019). https://doi.org/10.1109/ASRU46091.2019.9003913
  * [33] Neishi, M., Yoshinaga, N.: On the relation between position information and sentence length in neural machine translation. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). pp. 328–338. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/K19-1031, https://aclanthology.org/K19-1031
  * [34] Nguyen, H., Nguyen, C., Bao, P., Nakagawa, M.: A database of unconstrained vietnamese online handwriting and recognition experiments by recurrent neural networks. Pattern Recognition (2018)
  * [35] Pechwitz, M., Margner, V.: Baseline estimation for arabic handwritten words. In: Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition. pp. 479–484 (2002). https://doi.org/10.1109/IWFHR.2002.1030956
  * [36] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer (2019). https://doi.org/10.48550/ARXIV.1910.10683, https://arxiv.org/abs/1910.10683
  * [37] Ribeiro, L.S.F., Bui, T., Collomosse, J., Ponti, M.: Sketchformer: Transformer-based representation for sketched structure (2020). https://doi.org/10.48550/ARXIV.2002.10381, https://arxiv.org/abs/2002.10381
  * [38] Schaldenbrand, P., Liu, Z., Oh, J.: Styleclipdraw: Coupling content and style in text-to-drawing translation (2022). https://doi.org/10.48550/ARXIV.2202.12362, https://arxiv.org/abs/2202.12362
  * [39] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. p. 3104–3112. NIPS’14, MIT Press, Cambridge, MA, USA (2014)
  * [40] Thoppilan, R., Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Le, Q.: Lamda: Language models for dialog applications (01 2022)
  * [41] Varis, D., Bojar, O.: Sequence length is a domain: Length-based overfitting in transformer models. pp. 8246–8257 (01 2021). https://doi.org/10.18653/v1/2021.emnlp-main.650
  * [42] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [43] Zhang, J.X., Ling, Z.H., Dai, L.R.: Forward attention in sequence- to-sequence acoustic modeling for speech synthesis. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4789–4793 (2018). https://doi.org/10.1109/ICASSP.2018.8462020

##  Appendix 0.A Appendix

Table 9: Recognizer CER on test data, caused by data noise and the model’s mistakes. metric | value
---|---
CER long | 4.8
CER medium | 4.41
CER short | 5.5
CER avg | 4.9

Table 10: CER of the curve transformer on long test with and without the most horizontal ink. n words | 1 attempt | 5 attempts
---|---|---
1 | 6.29 (0.67) | 7.43 (0.53)
2 | 4.64 (0.48) | 5.0 (0.17)
3 | 3.98 (0.26) | 4.26 (0.32)
5 | 4.53 (0.01) | 4.71 (0.36)

Table 11: CER for different lengths in data augmentation procedure with candidate pool of 15K and threshold $0.5$. We compare sets with average lengths of 26, 37, 48, 65, 79, 100. For RNN models, training with lengths $>26$ results in poor performance. model | mean length | CER valid
---|---|---
transformer curves | - | 8.0 (0.69)
26 | 7.46 (1.02)
37 | 7.58 (1.56)
48 | 6.19 (0.42)
65 | 6.98 (0.23)
79 | 6.08 (0.66)
100 | 6.62 (0.36)
transformer raw | - | 9.72 (1.85)
48 | 9.97 (1.1)
65 | 9.08 (3.68)
79 | 11.01 (2.6)
RNN curves | - | 5.94 (0.74)
26 | 5.23 (0.5)
RNN raw | - | 3.4 (0.58)
26 | 2.06 (0.64)

Table 12: CER for different number of words in split qeneration on synthetic long validation dataset. model | n words | CER long valid
---|---|---
transformer curves | 1 | 8.0 (0.64)
2 | 6.03 (0.27)
3 | 5.57 (0.18)
5 | 5.78 (0.2)
transformer raw | 1 | 17.83 (0.92)
2 | 14.21 (1.17)
3 | 14.63 (0.23)
5 | 20.17 (3.48)
RNN curves | 1 | 7.15 (0.87)
2 | 7.47 (0.76)
3 | 7.92 (0.95)
5 | 9.28 (1.03)
RNN raw | 1 | 7.4 (0.34)
2 | 7.17 (0.37)
3 | 6.66 (0.31)
5 | 7.03 (0.7)


# Prototyping three key properties of specific curiosity in computational
reinforcement learning

Nadia M. Ady
Department of Computing Science
University of Alberta
Edmonton, AB T6G 2E8
Alberta Machine Intelligence Institute
<EMAIL_ADDRESS>
Roshan Shariff
Department of Computing Science
University of Alberta
Alberta Machine Intelligence Institute
<EMAIL_ADDRESS>
Johannes Günther
Alberta Machine Intelligence Institute
Department of Computing Science
University of Alberta
<EMAIL_ADDRESS>
Patrick M. Pilarski
Department of Medicine
Department of Computing Science
University of Alberta
Alberta Machine Intelligence Institute
<EMAIL_ADDRESS>
Corresponding author. Website:
https://ualberta.ca/ñmady/https://ualberta.ca/$\sim$nmady/

###### Abstract

Curiosity for machine agents has been a focus of intense research. The study
of human and animal curiosity, particularly _specific curiosity_ , has
unearthed several properties that would offer important benefits for machine
learners, but that have not yet been well-explored in machine intelligence. In
this work, we introduce three of the most immediate of these
properties—directedness, cessation when satisfied, and voluntary exposure—and
show how they may be implemented together in a proof-of-concept reinforcement
learning agent; further, we demonstrate how the properties manifest in the
behaviour of this agent in a simple non-episodic grid-world environment that
includes curiosity-inducing locations and induced targets of curiosity. As we
would hope, the agent exhibits short-term directed behaviour while updating
long-term preferences to adaptively seek out curiosity-inducing situations.
This work therefore presents a novel view into how specific curiosity operates
and in the future might be integrated into the behaviour of goal-seeking,
decision-making agents in complex environments.

Keywords:

curiosity, specific curiosity, computational reinforcement learning

### Acknowledgements

Thanks to all the amazing people who made this work better and clearer,
especially Kate Pratt. The authors wish to thank their funding providers. NMA
was supported by a scholarship from The Natural Sciences and Engineering
Research Council of Canada (NSERC). Work by PMP was supported by grants or
awards from the Canada CIFAR AI Chairs Program, the Alberta Machine
Intelligence Institute (Amii), NSERC, and the Canada Research Chairs program.

##  1 Specific Curiosity: An Introduction

Curiosity is in vogue; it is considered socially desirable, relating to a set
of behaviours that employers want in their employees, teachers want in their
students, and life coaches encourage in our interpersonal relationships. It
should come as no surprise that machine intelligence researchers have wanted
to provide the benefits of curiosity to their computational creations for
decades (Schmidhuber, 1991). Pursuing these benefits, researchers have
developed numerous mechanisms.

Figure 1: Turn to pg. 2 to rotate the mug.

While these mechanisms are different from each other (Ady & Pilarski, 2017;
Linke et al., 2019), many of them are centred on generating and using special
reward-like signals called intrinsic rewards. Put simply, the learner, or
agent, obtains intrinsic reward for encountering situations with good
potential for learning. Researchers have associated different concepts with
“potential for learning” and translated these concepts into metrics to shape
the behaviour of their agents.111 Examples include confidence (Schmidhuber,
1991), learning progress (Oudeyer et al., 2007, p. 269), surprise (White et
al., 2014, p. 14), interest/interestingness (Gregor & Spalek, 2014, p. 435;
Frank et al., 2014, pp. 5-6), novelty (Gregor & Spalek, 2014, p. 435; Singh et
al., 2004, pp. 1, 5), uncertainty (Pathak et al., 2017, pp. 1-2), compression
progress (Graziano et al., 2011, p. 44), competence (Oddi et al., 2020, pp.
2417-2418), and information gain (Bellemare et al., 2016, p. 4; Houthooft et
al., 2016, pp. 2-3). In general, these metrics are used to determine the
amount of intrinsic reward the agent receives for a given experience.
Intrinsic rewards have been shown to be very useful for increasing exploration
in some situations (Pathak et al., 2017).

However, intrinsic rewards produce behaviour that is conceptually unlike
curiosity. Critically, by rewarding a learner for getting into a particular
situation, the designer is encouraging the learner to return to that same
situation, or ones very like it. This behaviour is missing a conceptual aspect
of curiosity.

When humans describe curiosity, they talk about their curiosity being piqued
or triggered. They take action, not to observe the same situation that piqued
their curiosity, but to observe a different situation that they believe will
satisfy their curiosity (see Fig. 1). Humans take advantage of their learned
knowledge of how the world works to learn something specific, answering a
question that the curiosity-piquing situation led them to ask. This is
sometimes referred to as specific curiosity.

As a main contribution, we distill from the related literature three key
properties of specific curiosity, and argue that these properties are
beneficial to machine learners; specifically, we introduce the appropriateness
of a reinforcement learning approach for designing agents that exhibit these
properties. These properties are as follows, and are a subset of five key
properties described and argued for in more detail by Ady et al. (2022).

Directedness:

By separating a curiosity-piquing situation from a curiosity-satisfying
situation, we can think about the appropriate behaviour for a learner whose
curiosity has been piqued: take a sequence of actions that seems most likely
to take them to a situation that satisfies their curiosity. The tendency for
learners experiencing specific curiosity to undertake directed behaviour has
been well-documented in the literature (e.g. Hagtvedt et al. 2019, p. 2;
Berlyne, 1960, p. 297). This view contrasts with many other machine
reinforcement learning methods which rely on injecting randomness into their
choices of actions to experience new situations, rather than heading directly
for a situation they know they don’t know.

Cessation when satisfied:

Once a learner has found a situation that satisfies their curiosity, they
don’t need to experience the same situation again, so we have no need to
incentivize returning to a curiosity-satisfying situation. Examples
documenting the satisfiability of curiosity are prevalent in the literature,
including the works of Wiggin et al. (2019, p. 1194), Buyalskaya & Camerer
(2020, p. 141, who refer to it as ‘fulfillment’) and Dan et al. (2020, p. 150,
who refer to curiosity being ‘satiated’).

Voluntary exposure:

While the learner does not benefit from returning to a curiosity-satisfying
situation (that would only answer a question for which they have a
satisfactory answer!), they do benefit from experiencing curiosity-inducing
situations. A curiosity-inducing situation offers an appropriate jumping-off
point for learning, for metaphorically picking up a puzzle piece fitting into
what the learner already knows. Choosing to partake in activities likely to
induce curiosity—for example, picking up puzzles or mysteries or turning on
Netflix—has been called voluntary exposure to curiosity (Loewenstein, 1994, p.
84). This property might be thought of as developing an increased preference
for situations like those that have been curiosity-inducing in the past.

Finally, we provide a case study of an agent demonstrating these three
properties together, actively separating curiosity-piquing situations from
curiosity-satisfying situations—a separation which does not exist in the type
of behaviour motivated by intrinsic rewards. This proof-of-concept shows that
it is possible to create an agent with attributes of specific curiosity, but
should not be thought of as _the_ way to implement specific curiosity. Rather,
we hope it inspires the community to build upon our ideas and think up
improved machine agents that benefit from knowledge of specific curiosity.

##  2 The Appropriateness of a Reinforcement Learning Framework for Specific
Curiosity

Figure 2:  If you were curious about what was written on the mug, you probably
didn’t keep staring at Fig. 1. Instead, you took action to reach a situation
that would satisfy your curiosity.

With the goal to offer the benefits of specific curiosity to a computational
learner, we want to take advantage of an existing learning framework, ideally
one that supports the properties we want to achieve. The properties of
directedness and voluntary exposure require a learner to have some specific
capabilities. For directedness, the learner must be able to decide how to act
in the world so they can take the specific actions that they believe will
allow them to satisfy their curiosity. For voluntary exposure, the learner
must be able to decide how to act in the world so they can visit situations
that they suspect will result in their curiosity being piqued, and also learn
those enduring preferences for curiosity-inducing situations.

The framework of reinforcement learning supports these requirements, as
reinforcement learning centres around learners who are able to choose actions
that affect their experiences (Sutton and Barto, 2018, p. 3). Instantiations
of computational reinforcement learning algorithms are often called agents
because they shape their own experience in the world and learn from their
actions. This quality makes the framework well-suited for the design of
machine curiosity algorithms.

A reinforcement learning framework is particularly promising for the property
of voluntary exposure, as learners can estimate the value of different
situations (for more detail, see Sutton & Barto, 2018, p. 58). In this way, a
learner may develop a preference (i.e., a higher value) for situations that
are more likely to pique curiosity. This preference aligns with voluntary
exposure as observed in humans.

While, to the best of our knowledge, our conceptual separation of curiosity-
piquing situations from curiosity-satisfying situations and our argument for
specific properties of specific curiosity are new contributions, computational
reinforcement learning researchers have shown strong interest in aspects of
these properties in other contexts. In particular, the property of
directedness parallels work done on options (as early as Sutton et al., 1999)
and planning. Approaches using directedness also often exhibit cessation when
satisfied. For example, the options framework includes termination conditions
for each option (often naturally defined by goal states; Stolle & Precup,
2002, p. 212).

Prior work has aimed to address the lack of directedness that is a
characteristic of intrinsic-reward methods. For example, the Model-Based
Active eXploration algorithm presented by Shyam et al. (2019) focuses on
planning behaviour to allow the agent “to observe novel events” (p. 1).
Similarly, the Go-Explore family of algorithms centres on the idea of taking a
direct sequence of actions to move to a specific state for the purpose of
exploring from it (Ecoffet et al., 2020).

##  3 Case Study & Experiments

In this section, we describe how we created a simple agent exhibiting some of
the properties of specific curiosity. We specifically hope to show that, even
in a simple and focused setting, using our properties as guidelines allows
machine behaviour to emerge that approximates the specific curiosity of animal
learners. This example is not intended as a final or definitive computational
implementation of specific curiosity. The intended purpose of this section is
for the reader to gain insight and motivation to further investigate how to
integrate the properties of specific curiosity into different machine learning
frameworks and problem settings.

A complete agent with specific curiosity will require some additional
functionality that we have not implemented. A human can recognize that there
is something they do not know, sometimes referred to as an information gap
(Loewenstein, 1994) or an inostensible concept (Inan, 2012). Moreover, humans
can imagine what they would need to observe to rectify such a gap, and even
suggest actions that might lead to those observations. For example, you could
reach out and turn the mug in Fig. 1, or move your own body to see the mug at
a different angle. We do not tackle how agents can recognize information gaps
or how they can predict what will satisfy their resulting curiosity. In our
case study, we gave the agent a module that could recognize a curiosity-
inducing situation and a corresponding curiosity-satisfying target. We hard-
coded this module so only a single location in the world piqued curiosity and
the targets were randomly selected locations from a pre-determined subset (see
Fig. 4). We envision that the gaps in this architecture can be filled as more
aspects of specific curiosity are understood and become computationally
tractable.

Directedness: The first property we explored in the design of our agent was
directedness. As we have already suggested, directedness seems to involve
making and following a plan, which, in a reinforcement learning framework,
suggested to us that we might use a model of the world. In computational
reinforcement learning, a model-based approach using dynamic programming or an
approximation of dynamic programming can provide a learner with a plan to
follow from one state to another (Sutton & Barto, 2018, Ch. 4), suggesting
existing algorithms we could exploit in our pursuit of the property of
directedness.

Cessation when satisfied:  Directedness in the pursuit of satisfying curiosity
is temporary and should not persist when the agent is not in a state of
curiosity.222In the literature, there is a distinction between curiosity as a
state experienced temporarily by a learner versus as a trait, or general
propensity of the learner (Loewenstein, 1994, p. 78). Specific curiosity makes
the most sense as a form of state curiosity because it only persists until the
specific information of interest is found. We were inspired by an architecture
introduced by Silver et al. in 2008: Dyna-2. We realized that Dyna-2 has a
mechanism (originally used for playing games) that differentiates temporary
behaviour from the long-term preferences of the agent: two value functions. In
Silver’s (2009, p. 87) work, a temporary value function captures unique
aspects of the current match of a game, while a persistent value function
captures principles of the game that hold across matches. In our design, we
adapt this idea so the learner follows the temporary value function while
curious but uses a separate persistent value function to capture long-term
preferences. The temporary value function can lead the learner directly to a
situation they believe will satisfy their curiosity. Since what will satisfy
curiosity differs each time curiosity is piqued, the exact plan to satisfy
curiosity can be discarded at satisfaction.

Figure 3:  Domain used for the experiments. The orange boxes are the potential
curiosity-satisfying locations. The heavily-outlined box is the curiosity-
inducing location. If the agent leaves the top row, it teleports to the dashed
box in the bottom row.

Figure 4:  An example temporary value function, $V_{\mathrm{curious}}$. The
target has a heavy dashed outline and a value of zero. Magnitudes of negative
values are shown using shades of red.

Voluntary exposure: A temporary value function doesn’t have a lasting effect
on the learner’s preferences. However, we want an enduring effect associated
with curiosity: voluntary exposure. Humans develop a preference for curiosity-
inducing situations, so we wanted to experiment with algorithm modifications
that would lead to agents with this property. We came up with a small
modification to the standard TD learning algorithm (Sutton & Barto, 2018, p.
120) that would leave an enduring effect of having experienced curiosity on
the persistent value function, $V$. Our modified temporal difference,
$\delta$, is:

$\delta\leftarrow R+\gamma\cdot
V(x^{\prime})-\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{[V(x)+V_{\mathrm{curious}}(x)]}$
(1)

where $V_{\mathrm{curious}}$ refers to the temporary value function. For this
modification to result in the accrual of positive value in the persistent
value function, $V$, it is necessary for $V_{\mathrm{curious}}$ to be non-
positive everywhere. Given this modification, we ran experiments to see if the
enduring effect looked like voluntary exposure.

Experiment setup:  We ran our experiments in an $11\times 11$ grid world shown
in Fig. 4. In this grid world, the learner is located in one of the squares
and their actions let them move to adjacent squares, but they can only take
sideward or upward actions (including diagonals). Any upward action from the
top row of the grid teleports the agent to the middle of the bottom row. The
agent never receives any reward from this domain ($R$ in Eq. 1 is always
zero).

As long as the agent is not in a state of curiosity, the temporary value
function is zero and the agent acts $\epsilon$-greedily with respect to its
persistent value function. The centre of the grid is a curiosity-inducing
location: when the agent visits it, a curiosity-satisfying target is generated
and the agent is given a non-positive temporary value function (Fig. 4) that
guides it directly to that target. The target is randomly selected with equal
probability for each of the non-edge squares in the second row from the top
(highlighted in Fig. 4). When the agent visits the target, the temporary value
function is zeroed out again.

The results reported here reflect 30 trials of 5000 steps. The agent started
each trial in the curiosity-inducing location. We ran further experiments
varying the size and dynamics of the grid world, and the key results we
present are representative of all these experiments.

Results & Discussion: Fig. 6 shows the persistent value function at the end of
5000 steps, averaged over 30 trials. Fig. 6 shows how the values of the
curiosity-inducing location and the curiosity-satisfying locations change over
time.

The curiosity-inducing location accrues the most value and the agent learns a
trail of increasing value leading directly to that location. The curiosity-
satisfying locations, on the other hand, do not accumulate much value over
time. In effect, our modification to the learning update results in voluntary
exposure while retaining the property of cessation when satisfied.

Looking at the behaviour of the agent over time (an example video can be
viewed at https://youtu.be/TDUpB7OefFc) we can see the agent moves more and
more directly towards the curiosity-inducing location.

##  4 Conclusions

Figure 5:  Persistent value function, $V$, learned by the end of 5000 steps,
averaged over 30 trials. White squares have zero value. More positive values
are shown in deeper shades of blue.

Curiosity-inducinglocationCuriosity-satisfyinglocations

Figure 6:  Persistent value of the curiosity-inducing location is in blue
while the average persistent value of all of the curiosity-satisfying
locations is in orange. The lines are the average over 30 trials while the
shaded area shows the standard deviation.

We argue that learners benefit from specific curiosity, something previously
not demonstrated in the computational literature. In particular, specific
curiosity has properties of directedness, cessation when satisfied, and
voluntary exposure that yield important benefits that have been missing from
other approaches to machine curiosity. In this work we developed a case study
showing how these properties might be implemented in a computational
reinforcement learning framework.

We appreciate reinforcement learning for its flexibility to express key
properties of specific curiosity. We found, through our case study, that
separating persistent and temporary value functions was a useful mechanism to
encode some of the properties of biological curiosity. This separation gives
our agent directed behaviour as well as learned preferences, which have
traditionally been associated with model-based and model-free approaches to
reinforcement learning, respectively. The way our agent balances these two
approaches may offer a new perspective on how we might balance model-based and
model-free learning.

Future directions:  While directedness, cessation when satisfied, and
voluntary exposure are the first properties we have explored empirically, our
conceptual synthesis of the literature also demonstrates the importance of two
more properties: transience and connection to long-term information search
(Ady et al., 2022). While our agent has begun to exhibit some aspects of
specific curiosity, including these additional properties and continuing to
incorporate new ideas from the study of human curiosity can only strengthen
our computationally curious agents.

## References

Ady, Shariff, Günther, and Pilarski, in preparation for J. Artificial
Intelligence Research, 2022.

Ady and Pilarski, Multidiscip. Conf. Reinforcement Learning and Decision
Making, 2017.

Bellemare, Srinivasan, Ostrovski, et al., NeurIPS, 2016, 1471–1479.

Berlyne, Conflict, arousal, and curiosity. McGraw-Hill Book Company, 1960.

Buyalskaya and Camerer, Current Opinion in Behavioral Sciences, 2020,
35:141–149.

Dan, Leshkowitz, and Hassin, Current Opinion in Behavioral Sciences, 2020,
35:150–156.

Ecoffet, Huizinga, Lehman, Stanley, and Clune, Nature, 2021, 590, 580–586.

Frank, Leitner, Stollenga, et al., Frontiers in Neurorobotics, 2014, 7:25.

Graziano, Glasmachers, Schaul, et al., Acta Futura, 2011, 4, 41–52.

Gregor and Spalek, ELEKTRO 2014 Conference, 2014, 435–440.

Hagtvedt, Dossinger, Harrison, et al., Org. Behav. and Human Decision Proc.,
2019, 150:1-13.

Houthooft, Chen, Duan, et al., NeurIPS, 2016, 1109–1117.

Inan, The Philosophy of Curiosity, 2012. Routledge.

Linke, Ady, Degris, et al., Multidiscip. Conf. Reinforcement Learning and
Decision Making, 2017.

Loewenstein, Psychological Bulletin, 1994, 116:1, 75–98.

Markey and Loewenstein, International handbook of emotions in education, 2014,
238–255.

Pathak, Agrawal, Efros, et al., Int. Conf. on Machine Learning, 2017,
2778–2787.

Oddi, Rasconi, Santucci, et al., European Conference on Artificial
Intelligence, 2020, 2417–2424.

Oudeyer, Kaplan, and Hafner, IEEE Trans. on Evolutionary Computation, 2007,
11:2, 265–286.

Schmidhuber, International Joint Conference on Neural Networks, 1991,
1458–1463.

Shyam, Jaśkowski, and Gomez, Int. Conf. on Machine Learning, 2019, 5779–5788.

Silver, Sutton, and Müller, Int. Conf. on Machine Learning, 2008, 968–975.

Silver, “Reinforcement Learning and Simulation-Based Search in Computer Go,”
Doctoral Thesis, 2009.

Singh, Barto, and Chentanez, NeurIPS, 2004.

Spielberger and Starr, In: Motivation: Theory and Research, 1994, 221–243.
Lawrence Erlbaum Associates, Hillsdale, New Jersey.

Stolle and Precup, International Symposium on Abstraction, Reformulation, and
Approximation, 2002, 212–223.

Sutton and Barto, Reinforcement Learning: An Introduction, 2018. MIT Press,
second edition.

White, Modayil, and Sutton, Sequential Decision-Making with Big Data: AAAI-14
Workshop, 2014, 19–23.

Wiggin, Reimann, and Jain, Journal of Consumer Research, 2019,
45(6):1194–1212.