††thanks: These authors contributed equally to this work††thanks: These
authors contributed equally to this work

# Collapse of active nematic order through a two-stage dynamic transition

Aleksandra Ardaševa1 Ignasi Vélez-Cerón2,3 Martin Cramer Pedersen1 Jordi
Ignés-Mullol2,3<EMAIL_ADDRESS>Francesc Sagués2,3<EMAIL_ADDRESS>Amin
Doostmohammadi1<EMAIL_ADDRESS>1Niels Bohr Institute, University of
Copenhagen, Blegdamsvej 17, Copenhagen, Denmark
2Department of Materials Science and Physical Chemistry, Universitat de
Barcelona, Barcelona 08028, Spain
3Institute of Nanoscience and Nanotechnology, IN2UB, Universitat de Barcelona,
Barcelona 08028, Spain

###### Abstract

We present a novel two-stage transition of the ordered active nematic state of
a system of bundled microtubules into a biphasic active fluid. Specifically,
we show that upon light-induced solidification of the underlying medium,
microtubule-kinesin motor mixtures first undergo a cascade of folding in on
themselves, which is then followed by a progressive formation of isotropic
fluid domains, transforming the system into a biphasic active suspension.
Using an active lyotropic model, we show that the two-stage transition in the
system is governed by screening effects that become important upon substrate
changes, leading to higher activity modes to become important. Specifically,
the combined effect of friction and quadrupolar activity leads to the
hierarchical folding that follows the intrinsic bend instability. Our results
demonstrate the dynamics of the collapse of orientational order in active
nematics and present a new route for controlling active matter by modifying
the activity through changing the surrounding environment.

Living materials, such as eukaryotic cells and bacteria, are distinct from
their inanimate counterparts in their ability to continuously convert chemical
energy to mechanical work at the level of their constituent elements [1]. This
activity, in turn, orchestrates the formation of collective patterns of motion
on scales much larger than the size of the individuals [2, 3, 4, 5], and
guides important physiological processes, such as organ development [6, 7] and
collective invasion [8, 9, 10]. Similar activity-induced collective patterns
are increasingly emulated in bio-inspired synthetic materials, with the aim of
rendering them capable of self-organization and self-healing [11, 12].

In the bulk, the mechanistic basis and fundamental instabilities of activity-
induced collective pattern formation have been intensely studied both in
theoretical models [13, 14, 15] and in experiments [16]. Importantly, however,
living materials in most natural setups are constrained by confining
boundaries and substrates, where the dynamics of the active systems are
closely controlled through their interaction with the surrounding environment.
Striking examples include subcellular active flows confined to the cell
boundaries [17], twitching motility of surface-attached bacteria [18], and
dynamics of cellular monolayers interacting with their underlying substrate
[19].

Dynamics of active matter confined in channels of varying shapes and sizes
have been the focus of intense research. While various modes of collective
self-organization have been identified in theoretical models [20, 21, 22, 23]
and experimental systems including subcellular filaments [24, 25, 26],
bacteria [27, 28], and confined epithelial and mesenchymal cells [29, 30], the
impact of active matter interaction with a tunable and dynamically changing
environment is not well understood. Theoretical studies, focusing on the
frictional and momentum-damping effects induced by substrates, have suggested
possibilities of stabilization, control, and significant alterations in
steady-state collective patterns of active materials [31, 32, 33, 34, 35].
Nevertheless, experimental realization of such effects and deciphering the
complex impact of the substrate beyond frictional effects, has remained a
significant challenge. Moreover, while the focus so far has been on steady-
state patterns of motion, the study of transient effects of active matter
interacting with substrates, and the possibility of transitioning between
different steady-states are yet to be explored.

Here, using a mixture of microtubule-kinesin motor proteins as a model active
system residing on a tunable hydrogel material, we reveal a novel two-stage
dynamic transition that extinguishes the nematic order of the system and
transforms it into a biphasic, yet still active, suspension of the bundled
microtubules. In the first stage, cascades of self-similar patterns emerge
upon active nematic folding in on itself, which is then followed by the growth
of the number of isotropic domains in the second stage (Fig. 1). We combine
experimental observations with a continuum model of lyotropic active nematic
to reveal the mechanisms that govern the transition between these two
dynamical steady states and uncover the previously overlooked impact of the
substrate on tuning the dominant forms of active stresses.

Figure 1:  Activation of light leads to a two-stage dynamic transition. (a)
Schematic representation of the experimental cell. The flow cell is filled
with fluorinated oil and an aqueous solution containing the active material,
monomer, and initiator of the hydrogel. A flow cell of 50 $\mu$m thickness is
prepared using a superhydrophilic slide and a hydrophobic cover slide. Low-
power UV light leads to the formation of the hydrogel, confining the active
material between it and the oil phase. (b) – (d) Director correlation length,
$L_{C_{Q}}(r)$, order parameter, $S$, and root-mean-squared velocity,
$v_{\text{rms}}$, respectively, as a function of time. The gray dashed line
marks the time when the light is turned on. $L_{C_{Q}}(r)$ and
$v_{\text{rms}}$ are scaled by pre-light averages. (e) Snapshots from
experiments, showing patterns for different transition checkpoints. The color
of each frame corresponds to the corresponding time point indicated in panel
(b) (see also Movie S1).

### Two-stage transition from active nematic to active biphasic fluid.

In order to explore the effect of contact with the substrate in a controllable
manner, we designed an experimental system, where we employ the standard
formulation of a kinesin/tubulin active gel to which we add the precursors of
a poly(ethylene glycol) (PEG)-based hydrogel that can be photopolymerized with
UV light with the concourse of a photoinitiator [36] (see Supplementary
Material for more details). Experiments are performed in a flow cell of 50 µm
thickness, made with a superhydrophilic and a hydrophobic glass slide (Fig.
1a, and Supplemental Material). The cell is filled with the active mixture and
fluorinated oil to allow the formation of the active nematic at the water-oil
interface. The interfaced material displays, at this stage, the well-
established, seemingly chaotic flow patterns characteristic of an active
nematic layer [11, 37]. We then subject the cell to a sustained (120 s) low-
power-illumination. The hydrogel polymerizes, leaving a wetting film that is
still capable of accommodating the microtubule-based fluid (more details of
the experimental protocols can be found in Supplementary Material).

Figure 2:  Continuum simulations reproduce the two-stage transition from
active nematic fluid to active biphasic fluid. (a) Schematic depiction of the
simulation setup. In the beginning, simulation is carried out with only
dipolar activity. At $t_{0}$, in addition to the dipolar activity, friction
and quadrupolar activity are turned on to mimic the effect of turning on light
in the experiments. (b)–(c) Director correlation length (rescaled by the value
before $t_{0}$), $L_{C_{Q}(r)}$, and the order parameter, $S$, respectively,
as a function of time (in simulation units, s.u.). The gray dashed line
denotes the onset of friction and quadrupolar activity
($t_{0}^{f}=t_{0}^{q}$). Plots are averaged over 5 realizations. (d) Line
integral convolution (LIC) render of snapshots of the director at different
stages of transition (see Movie S2). Black regions correspond to the isotropic
phase. The color of each frame corresponds to the corresponding stage
indicated in panel (b).

The observed changes in the orientational textures following this simple
intervention are drastic (see Fig. 1e and Supplementary Movie S1). The
material starts to fold in on itself, following what can be interpreted as a
cascade of hierarchical bending instabilities. The result is a highly
corrugated material, as exhibited by the practical collapse of its orientation
two-point correlations (see Fig. 1b). In the final state, the microtubule
suspension seems to rarefy while losing nematic order (see Fig. 1c). We term
this state active biphasic fluid, as despite being largely parceled by
isotropic non-fluorescent domains, the material does not cease to move at any
time, but at much lower velocities (see Fig. 1d). A closer inspection of the
temporal evolution of the system reveals that upon turning on the light the
crossover from active nematic to active biphasic fluid goes through a two-
stage process: a cascade of self-similar folding patterns, followed by a drop
in the orientational order and increase in the number of isotropic domains
that span the system (see Supplementary Fig. 1a). A hallmark of this two-stage
transition is also evident in the evolution of the director correlation length
with time (see Fig. 1b): as hierarchical folding emerges during the first
stage of the transition, the correlation length and the magnitude of the
orientational order sharply drop (Fig. 1c). This is followed by a subsequent
increase and a peak in the correlation length, as the isotropic domains start
to elongate, which is then followed by a second drop in the correlation length
and the magnitude of the orientational order. This identifies the second stage
of the transition leading to an active biphasic fluid state being established.

### Continuum theory reproduces the two-stage transition.

To investigate the physical mechanism driving the experimentally observed
transition, we used a continuum model of lyotropic active nematics [38] (see
Supplementary Material for model description). Here, a non-conserved scalar
parameter, $\phi\in[0,1]$, is introduced to distinguish the active nematic
phase driven by dipolar active stresses [33, 39, 40], from a passive,
isotropic phase. When the UV light is on, the gel underneath the substrate
solidifies. This leads to a friction-induced screening effect that affects the
dominant forms of active stresses exerted by active particles [41]. Therefore,
higher-order contributions to active stresses become important [41, 42, 43].
To account for this, we introduce quadrupolar activity to the momentum
equation, together with the isotropic friction. To replicate the experimental
condition as closely as possible, these terms are switched on instantaneously
at a given time after the system has reached the active turbulence state (Fig
2a). This corresponds to switching the UV light on in the experiment.

Figure 3:  Physical mechanism behind the two-stage dynamic transition. (a) LIC
render of a stripe of nematic phase surrounded by isotropic phase (black) with
bend instability evolved under different conditions: (– –) no friction and no
quadrupolar activity; (+ –) friction only; (– +) quadrupolar activity only; (+
+) both friction and quadrupolar activity. Dipolar activity is on at all
times. See Movies S3–S6. (b) Correlation of director before the friction and
quadrupolar activity are activated (black dashed line) and after (pink line).
(c) Director correlation length (rescaled by pre-light average) and the order
parameter as a function of time for high and low values of dipolar activity
obtained via simulations. Plots are the average over 5 realizations. (d) Same
as (c) but for experiments (single realization, see Movie S7).

The model qualitatively reproduces the dynamics in the director field (see
Fig. 2d and Supplementary Movie S2), accurately capturing the two-stage
transition. Ordered regions undergo folding, breaking into smaller
perpendicular domains. Afterward, the domains elongate until the active
biphasic regime with smaller length scales is established. Remarkably, not
only do the qualitative features mirror experimental observations, but the
transition follows a two-stage dynamics as both the two-point director
correlation and the order collapse (Fig. 2b–c), and the number of isotropic
domains subsequently increases, in agreement with experiments (see also
Supplementary Fig. 1b). Additionally, in line with the experimental
measurements, the crossover between the two stages is marked by the temporal,
step-like (see Supplementary Fig. 2) increase and a peak in the correlation
length before the transition to biphasic active fluid. Together, the
experimental observations and continuum modeling reveal a novel two-stage
process for the dynamic transition between active nematic and active biphasic
fluids that is mediated by the interaction with the substrate.

### Physical mechanism.

We next investigate the mechanism for the emergence of self-similar patterns
during the two-stage transition. To understand the separate roles of friction
and quadrupolar activity in inducing hierarchical folding, we simulate a
minimal setup that consists in setting a stripe of low-activity nematic fluid
surrounded by an isotropic phase (Supplementary Movies S3–S6). Here, we let
the system evolve through the well-established bend instability (Fig. 3a,
panel (i)) under dipolar activity only, and then turn on friction and/or
quadrupolar activity.

Turning on only isotropic friction stabilizes the system and suppresses the
growth of the bend instability (Fig. 3a, panel (ii)). On the other hand, when
quadrupolar activity acts on the system (and the friction is turned off), the
nematic phase folds in on itself, creating mushroom-like structures (Fig. 3a,
panel (iii)). Finally, when both friction and quadrupolar activities are
turned on (Fig. 3a, panel (iv)), the hierarchical buckling of the existing
bend instability is observed. This secondary buckling occurs at a much smaller
wavelength, compared to the primary bend instability, as shown by the
measurements of the orientation correlation (see Fig. 3b). These results
indicate that the observed patterns during the two-stage transition result
from a balance between the dipolar and quadrupolar activities (destabilizing)
and friction (stabilizing) that leads to cascades and eventual decreases in
length scales. Hence, the ratios between these three parameters determine the
possibility of a substrate-induced transition.

### Controlling the transition by tuning activity.

Having established the mechanism of the two-stage collapse of the orientation
order in active nematics, we use the model to explore the possibility of
controlling the dynamics by varying the activity of the system. The model
(Fig. 3c) predicts that for a less active system with lower dipolar activity,
the two-stage transition of a system is delayed, the decay to smaller length
scales and collapse of the order are slowed down. The increase in the number
of isotropic domains is also slowed down (see Supplementary Fig. 2a). Similar
behavior can be seen in the experiments (Fig. 3d) when the activity of the
microtubules is reduced by lowering ATP concentration before shining light.
The resulting experimental patterns (see Supplementary Movie S7) show similar
stages of the transition, including folding and elongation. The temporal
evolution of the correlation length, order parameter and number of domains
qualitatively confirm the model predictions (Fig. 3d and Supplementary Fig.
1b). Together, the modeling and experimental results show how the interplay of
dipolar activity with frictional damping and quadrupolar activity shapes the
dynamics of the transition from active nematic to active biphasic fluid.

### Discussion.

In this work, we present novel controlled experiments, using a reconstituted
microtubule-kinesin motor mixture system that is in contact with a tunable
hydrogel material, to demonstrate the dynamic pattern evolution of an active
fluid upon perturbations in the surrounding environment. Specifically, we show
that upon controlled, light-induced solidification of the underlying hydrogel,
the active material undergoes a two-stage dynamic transition in which an
initially ordered active nematic fluid evolves into an active biphasic fluid.
The two stages are marked by distinct pattern formation: in the first stage,
quickly after solidification of the underlying hydrogel, active nematic
domains evolve into cascades of self-similar patterns by folding in on
themselves. This folding cascade is then followed by a significant increase in
the number of isotropic domains that are interleaved with active nematic
filaments until the final stage is characterized by an active biphasic fluid.

Our continuum model of lyotropic active fluid reveals the mechanistic basis of
the two-stage evolution of the system upon solidification of the substrate and
shows that increased friction alone is insufficient to explain the observed
transition. Substrate friction not only extracts momentum from the system but
also induces screening effects, which alter the dominant form of active stress
from dipolar to quadrupolar stress. The emergence of significant quadrupolar
contribution, induced by hydrodynamic screening effect of the substrate, is in
line with theoretical predictions in the context of single active particles
[41] and dense collections of active particles, both in the presence of
hydrodynamics [42] and in over-damped systems [43]. To our knowledge, the
experiments presented in this work provide the first evidence of the important
role of quadrupolar active stresses in dense active matter systems under
friction. The continuum model further demonstrates how such quadrupolar
activity combined with frictional damping can induce the folding of active
domains onto themselves, leading to the emergence of cascades of self-similar
folding patterns.

Our findings present one of the first examples of the dynamics of active fluid
interaction with a reconfigurable surrounding that transitions between fluid-
like and solid-like states. Understanding the dynamics of active materials,
such as cells and bacteria, requires accounting for the dynamic changes in
their surrounding environment. Examples like the extracellular matrix (ECM)
and bacterial biofilms underscore the importance of this dynamic interplay.
The ECM, with its ability to transition between fluid-like and solid-like
states, profoundly influences cell behavior, tissue remodeling, and disease
progression [44]. Similarly, bacterial biofilms, through their dynamic matrix
properties, govern bacterial community formation, resistance to external
stresses, and biofilm-associated infections [45]. Ignoring the dynamic nature
of the environment could oversimplify the complexities of cell and bacterial
behaviors, hindering our ability to comprehend processes like wound healing,
cancer metastasis, and microbial infections. Hence, integrating the
understanding of dynamic environmental changes is crucial for unraveling the
intricate dynamics of active materials in biological systems.

##  I Author contributions

J.I.-M and F.S. designed the experiments and A.D. designed the model. I.V.-C.
performed the experiments and extracted the director field and velocity field
data. A.A implemented the model, performed the simulations, analyzed the
simulations and experimental data, and prepared figures with help from I.V.-C.
and M.C.P., J.I.-M., F.S and A.D. all contributed to the interpretation of
results. A.D. prepared the first draft. A.D. and A.A. wrote the paper with
input from all authors. This collaborative effort was led by J.I.-M, F.S and
A.D.

##  II Acknowledgements

We thank Kristian Thijssen for the helpful discussions. A.A. acknowledges
support from the EU’s Horizon Europe research and innovation program under the
Marie Sklodowska-Curie grant agreement No. 101063870 (TopCellComm). A.D.
acknowledges funding from the Novo Nordisk Foundation (grant no.
NNF18SA0035142 and NERD grant no. NNF21OC0068687), Villum Fonden Grant no.
29476, and the European Union via the ERC-Starting Grant PhysCoMeT, grant no.
101041418. We thank M. Pons, A. LeRoux, and G. Iruela (Universitat de
Barcelona) for their assistance in the expression of motor proteins. I.V.-C.
acknowledges funding from Generalitat de Catalunya though a FI-2020 PhD.
Fellowship. I.V.-C., J.I.-M., and F.S. acknowledge funding from
MICIU/AEI/10.13039/501100011033 (Grant No. PID2022-137713NB-C21). I.V.-C. and
J.I.-M. acknowledge funding from MICIU/AEI/10.13039/501100011033 (Grant No.
PDC2022-133625-I00).

## References

  * Prost  _et al._ [2015] J. Prost, F. Jülicher, and J.-F. Joanny, Nature Physics 11, 111 (2015).
  * Hartmann  _et al._ [2019] R. Hartmann, P. K. Singh, P. Pearce, R. Mok, B. Song, F. Díaz-Pascual, J. Dunkel, and K. Drescher, Nature Physics 15, 251 (2019).
  * Needleman and Dogic [2017] D. Needleman and Z. Dogic, Nature Reviews Materials 2, 1 (2017).
  * Doostmohammadi  _et al._ [2018] A. Doostmohammadi, J. Ignés-Mullol, J. M. Yeomans, and F. Sagués, Nature Communications 9, 3246 (2018).
  * Sagués [2023] F. Sagués,  _Colloidal Active Matter_ (Taylor and Francis Group, Boca Raton, FL, 2023).
  * Bianco  _et al._ [2007] A. Bianco, M. Poukkula, A. Cliffe, J. Mathieu, C. M. Luque, T. A. Fulga, and P. Rørth, Nature 448, 362 (2007).
  * Maroudas-Sacks  _et al._ [2021] Y. Maroudas-Sacks, L. Garion, L. Shani-Zerbib, A. Livshits, E. Braun, and K. Keren, Nature Physics 17, 251 (2021).
  * Ariel  _et al._ [2015] G. Ariel, A. Rabani, S. Benisty, J. D. Partridge, R. M. Harshey, and A. Be’Er, Nature Communications 6, 8396 (2015).
  * Clark and Vignjevic [2015] A. G. Clark and D. M. Vignjevic, Current Opinion in Cell Biology 36, 13 (2015).
  * Zhang  _et al._ [2022] D.-Q. Zhang, P.-C. Chen, Z.-Y. Li, R. Zhang, and B. Li, Proceedings of the National Academy of Sciences 119, e2122494119 (2022).
  * Sanchez  _et al._ [2012] T. Sanchez, D. T. Chen, S. J. DeCamp, M. Heymann, and Z. Dogic, Nature 491, 431 (2012).
  * Schaller  _et al._ [2010] V. Schaller, C. Weber, C. Semmrich, E. Frey, and A. R. Bausch, Nature 467, 73 (2010).
  * Ramaswamy [2010] S. Ramaswamy, Annual Review of Condensed Matter Physics 1, 323 (2010).
  * Marchetti  _et al._ [2013] M. C. Marchetti, J.-F. Joanny, S. Ramaswamy, T. B. Liverpool, J. Prost, M. Rao, and R. A. Simha, Reviews of Modern Physics 85, 1143 (2013).
  * Alert  _et al._ [2022] R. Alert, J. Casademunt, and J.-F. Joanny, Annual Review of Condensed Matter Physics 13, 143 (2022).
  * Doostmohammadi and Ladoux [2022] A. Doostmohammadi and B. Ladoux, Trends in Cell Biology 32, 140 (2022).
  * Callan-Jones and Voituriez [2016] A. C. Callan-Jones and R. Voituriez, Current Opinion in Cell Biology 38, 12 (2016).
  * Gomez  _et al._ [2023] S. Gomez, L. Bureau, K. John, E.-N. Chêne, D. Débarre, and S. Lecuyer, Elife 12, e81112 (2023).
  * Krishnan  _et al._ [2011] R. Krishnan, D. D. Klumpers, C. Y. Park, K. Rajendran, X. Trepat, J. Van Bezu, V. W. Van Hinsbergh, C. V. Carman, J. D. Brain, J. J. Fredberg, _et al._ , American Journal of Physiology-Cell Physiology 300, C146 (2011).
  * Doostmohammadi  _et al._ [2017] A. Doostmohammadi, T. N. Shendruk, K. Thijssen, and J. M. Yeomans, Nature Communications 8, 15326 (2017).
  * Shendruk  _et al._ [2017] T. N. Shendruk, A. Doostmohammadi, K. Thijssen, and J. M. Yeomans, Soft Matter 13, 3853 (2017).
  * Samui  _et al._ [2021] A. Samui, J. M. Yeomans, and S. P. Thampi, Soft Matter 17, 10640 (2021).
  * Coelho  _et al._ [2019] R. C. Coelho, N. A. Araújo, and M. M. T. da Gama, Soft Matter 15, 6819 (2019).
  * Hardoüin  _et al._ [2019] J. Hardoüin, R. Hughes, A. Doostmohammadi, J. Laurent, T. Lopez-Leon, J. M. Yeomans, J. Ignés-Mullol, and F. Sagués, Communications Physics 2, 121 (2019).
  * Opathalage  _et al._ [2019] A. Opathalage, M. M. Norton, M. P. Juniper, B. Langeslay, S. A. Aghvami, S. Fraden, and Z. Dogic, Proceedings of the National Academy of Sciences 116, 4788 (2019).
  * Hardoüin  _et al._ [2022] J. Hardoüin, C. Doré, J. Laurent, T. Lopez-Leon, J. Ignés-Mullol, and F. Sagués, Nature Communications 13, 6675 (2022).
  * Lushi  _et al._ [2014] E. Lushi, H. Wioland, and R. E. Goldstein, Proceedings of the National Academy of Sciences 111, 9733 (2014).
  * You  _et al._ [2021] Z. You, D. J. Pearce, and L. Giomi, Science Advances 7, eabc8685 (2021).
  * Deforet  _et al._ [2014] M. Deforet, V. Hakim, H. G. Yevick, G. Duclos, and P. Silberzan, Nature Communications 5, 3747 (2014).
  * Duclos  _et al._ [2018] G. Duclos, C. Blanch-Mercader, V. Yashunsky, G. Salbreux, J.-F. Joanny, J. Prost, and P. Silberzan, Nature Physics 14, 728 (2018).
  * Thampi  _et al._ [2014] S. P. Thampi, R. Golestanian, and J. M. Yeomans, Physical Review E 90, 062307 (2014).
  * Thijssen  _et al._ [2020] K. Thijssen, M. R. Nejad, and J. M. Yeomans, Physical Review Letters 125, 218004 (2020).
  * Ramaswamy  _et al._ [2003] S. Ramaswamy, R. A. Simha, and J. Toner, Europhysics Letters 62, 196 (2003).
  * Thijssen  _et al._ [2021] K. Thijssen, D. A. Khaladj, S. A. Aghvami, M. A. Gharbi, S. Fraden, J. M. Yeomans, L. S. Hirst, and T. N. Shendruk, Proceedings of the National Academy of Sciences 118, e2106038118 (2021).
  * Zhang  _et al._ [2021] R. Zhang, S. A. Redford, P. V. Ruijgrok, N. Kumar, A. Mozaffari, S. Zemsky, A. R. Dinner, V. Vitelli, Z. Bryant, M. L. Gardel, _et al._ , Nature Materials 20, 875 (2021).
  * Vélez-Cerón  _et al._ [2024] I. Vélez-Cerón, P. Guillamat, F. Sagués, and J. Ignés-Mullol, Proceedings of the National Academy of Sciences 121, e2312494121 (2024).
  * Guillamat  _et al._ [2016] P. Guillamat, J. Ignés-Mullol, and F. Sagués, Proceedings of the National Academy of Sciences 113, 5498 (2016).
  * Blow  _et al._ [2014] M. L. Blow, S. P. Thampi, and J. M. Yeomans, Physical Review Letters 113, 248303 (2014).
  * Marenduzzo  _et al._ [2007] D. Marenduzzo, E. Orlandini, M. Cates, and J. Yeomans, Physical Review E 76, 031921 (2007).
  * Martínez-Prat  _et al._ [2019] B. Martínez-Prat, J. Ignés-Mullol, J. Casademunt, and F. Sagués, Nature Physics 15, 362 (2019).
  * Mathijssen  _et al._ [2016] A. J. Mathijssen, A. Doostmohammadi, J. M. Yeomans, and T. N. Shendruk, Journal of Fluid Mechanics 806, 35 (2016).
  * Maitra  _et al._ [2018] A. Maitra, P. Srivastava, M. C. Marchetti, J. S. Lintuvuori, S. Ramaswamy, and M. Lenz, Proceedings of the National Academy of Sciences 115, 6934 (2018).
  * Sultan  _et al._ [2022] S. A. Sultan, M. R. Nejad, and A. Doostmohammadi, Soft Matter 18, 4118 (2022).
  * Elosegui-Artola [2021] A. Elosegui-Artola, Current Opinion in Cell Biology 72, 10 (2021).
  * Flemming  _et al._ [2023] H.-C. Flemming, E. D. van Hullebusch, T. R. Neu, P. H. Nielsen, T. Seviour, P. Stoodley, J. Wingender, and S. Wuertz, Nature Reviews Microbiology 21, 70 (2023).


# Improving historical spelling normalization with bi-directional LSTMs and
multi-task learning

Marcel Bollmann
Department of Linguistics
Ruhr-Universität Bochum
Germany
<EMAIL_ADDRESS>
&Anders Søgaard
Dpt. of Computer Science
University of Copenhagen
Denmark
<EMAIL_ADDRESS>

###### Abstract

Natural-language processing of historical documents is complicated by the
abundance of variant spellings and lack of annotated data. A common approach
is to normalize the spelling of historical words to modern forms. We explore
the suitability of a deep neural network architecture for this task,
particularly a deep bi-LSTM network applied on a character level. Our model
compares well to previously established normalization algorithms when
evaluated on a diverse set of texts from Early New High German. We show that
multi-task learning with additional normalization data can improve our model’s
performance further.

00footnotetext:  This work is licenced under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/

##  1 Introduction

Interest in computational processing of historical documents is on the rise,
as evidenced by the growing field of digital humanities and the increasing
number of digitally available resources of historical data. Spelling
normalization, i.e. the mapping of historical spelling variants to
standardized/modernized forms, is often employed as a pre-processing step to
allow the utilization of existing tools for the respective modern target
language [Piotrowski (2012].

Training data for supervised learning of spelling normalization is typically
scarce in the historical domain. Furthermore, dialectal influences and even
individual preferences of an author can have a huge impact on the spelling
characteristics in a particular text, meaning that even training data from
other corpora of the same language and time period cannot always be reliably
used.

Algorithms have often been developed with this fact in mind, e.g. by being
based on some form of phonetic, graphematic, or semantic similarity measure
[Jurish (2010, Bollmann (2012, Amoia and Martinez (2013]. On the other hand,
neural networks – and particularly deep networks with several hidden layers –
are assumed to work best when trained on large amounts of data. It is
therefore not clear whether neural networks are a good choice for this
particular domain.

We frame spelling normalization as a character-based sequence labeling task,
and explore the suitability of a deep bi-directional long short-term memory
model (bi-LSTM) in this setting. By basing our model on individual characters
as input, along with performing some basic preprocessing (e.g., downcasing all
characters), we keep the vocabulary size small, which in turn reduces the
model’s complexity and the amount of data required to train it effectively. We
show that this model outperforms both the existing normalization tool Norma
[Bollmann (2012] and a CRF-based tagger when evaluated on a diverse dataset
from Early New High German.

Furthermore, we experiment with a multi-task learning setup using auxiliary
data that has similar, but not identical spelling characteristics to the
target text. We show that using bi-LSTMs with this multi-task learning setup
can improve normalization accuracy further, while Norma and CRF do not profit
much from the additional data in a traditional setup.

##  2 Datasets

We use a total of 44 texts from the Anselm corpus [Dipper and Schultz-Balluff
(2013] of Early New High German.111https://www.linguistics.rub.de/anselm/ The
corpus is a collection of several manuscripts and prints of the same core
text, a religious treatise. Although the texts are semi-parallel and share
some vocabulary, they were written in different time periods (between the 14th
and 16th century) as well as different dialectal regions, and show quite
diverse spelling characteristics. For example, the modern German word _Frau_
‘woman’ can be spelled as _fraw/vraw_ (Me), _frawe_ (N2), _frauwe_ (St),
_fraüwe_ (B2), _frow_ (Stu), _vrowe_ (Ka), _vorwe_ (Sa), or _vrouwe_ (B),
among others.222Abbreviations in brackets refer to individual texts using the
same internal IDs that are found in the Anselm corpus.

All texts in the Anselm corpus are manually annotated with gold-standard
normalizations following guidelines described in ?). For our experiments, we
excluded texts from the corpus that are shorter than 4,000 tokens, as well as
a few texts for which annotations were not yet available at the time of
writing (mostly Low German and Dutch versions). Nonetheless, the remaining 44
texts are still quite short for machine-learning standards, ranging from about
4,200 to 13,200 tokens, with an average length of 7,353 tokens.

For all texts, we removed tokens that consisted solely of punctuation
characters. We also lowercase all characters, since it helps keep the size of
the vocabulary low, and uppercasing of words is usually not very consistent in
historical texts.

###  2.1 Conversion to labeled character sequences

Normalization is annotated on a word level; to reframe the problem as a
character-based sequence labeling task, we need to align the historical
wordforms and their normalizations on a character level. Ideally, we would
like these alignments to be linguistically plausible, i.e., characters that
most likely correspond to each other (e.g., historical  _j_ and modern  _i_ ,
as in _jn – ihn_ ‘him’) should be aligned whenever possible.

The Levenshtein algorithm [Levenshtein (1966] can be used to produce
alignments that preferably align identical characters, but is ambiguous when
multiple alignments with the same Levenshtein distance exist. We therefore use
iterated Levenshtein distance alignment [Wieling et al. (2009], which uses
pointwise mutual information on aligned segments to estimate statistical
dependence, and favors alignments of characters that tend to cooccur often
within the dataset. Since different texts can use the same characters in
different ways, we perform this iterated alignment separately for each text.

A difficulty of these alignments is that the two wordforms can be of different
lengths. We introduce a special epsilon label ($\epsilon$) whenever a
historical character is not aligned to any character in the normalization. We
cannot do that for the inverse case, since the historical characters are our
units of annotation and therefore need to be fixed, so we choose to perform a
leftward merging of normalized characters whenever they are not aligned to any
character in the historical wordform. For the word-initial case, we introduce
a special “start of word” symbol (_). This symbol is prepended to each word
during both training and testing, and is assigned the epsilon label during
training when there is no word-initial insertion.

Here is an example of the final character sequence representation for the word
pair _vsfuret – ausführt_ ‘(he) leads out’:

8 _& v s f u r e t a & u s f üh r $\epsilon$ t

A consequence of this approach is that our labels cannot only be characters,
but also combinations of characters (such as _üh_ in the example above); our
label set is therefore potentially unbounded. However, we found that this is
not much of a problem in practice, since these cases tend to be comparatively
rare.

##  3 Model

Our model architecture consists of:  (i) an embedding layer for the input
characters;  (ii) a stack of bi-directional long short-term memory units (bi-
LSTMs); and  (iii) a final dense layer with a softmax activation to generate a
probability distribution over the output classes at each timestep.  An
illustration of the model can be found in Figure 1.

The embedding layer maps one-hot input vectors (representing historical
characters) to dense vectors. We did not use pre-trained embeddings; the
embeddings are initialized randomly and learned as part of the regular network
training process.

LSTMs [Hochreiter and Schmidhuber (1997] are a form of recurrent neural
network (RNN) designed to better learn long-term dependencies, and have proven
advantageous to plain RNNs on many tasks. Bi-directional LSTMs read their
input in both normal and reversed order, allowing the model to learn from both
left and right context at each input timestep. A stack of bi-LSTMs, or a deep
bi-LSTM, is a configuration of several bi-LSTM units so that the output of the
$i$th unit is the input of the $(i+1)$th unit. In our model, we use a stack of
three bi-LSTM layers.

_jn$\epsilon$ihn _embedding layer_ _stack of_ _bi-LSTM layers_ _prediction
layer_ Figure 1: Flow diagram of the bi-LSTM character sequence labeling
model, unrolled for time, for the word pair _jn_ – _ihn_ ‘him’.

The final dense layer is used to generate the output predictions, based on a
linear transformation of the bi-LSTM outputs for each timestep followed by a
softmax activation. We train the model by minimizing the cross-entropy loss
across all output characters, and using backpropagation to update the weights
in all layers (including the embedding layer). During prediction, we generate
output labels in a greedy fashion, choosing the label with the highest
probability for each timestep.

###  3.1 Multi-task learning setup

In multi-task learning (MTL), the performance of a model on a given task is
improved by additionally training it on one or more auxiliary tasks [Caruana
(1993]. For our bi-LSTM model, this means that all layers of the model are
shared between the tasks apart from the final prediction layer, which is kept
separate for the main and auxiliary tasks. This way, errors in an auxiliary
task that are backpropagated through the network also affect the prediction of
the main task, helping to regularize the network’s weights and prevent
overfitting.

Multi-task learning with (deep) neural network architectures was shown to be
effective for a variety of NLP tasks, such as part-of-speech tagging,
chunking, named entity recognition [Collobert et al. (2011]; sentence
compression [Klerke et al. (2016]; or machine translation [Luong et al.
(2016].

In our experiments, we regard spelling normalization within a target domain
(i.e., a given historical text) as our main task, while using normalization
within related domains (i.e., texts from a similar time period, but with
distinct spelling characteristics) as our auxiliary task. During training, we
alternate between training on a random instance from the main and the
auxiliary tasks.

###  3.2 Hyperparameters

We set aside one of the texts (B) from the Anselm corpus for testing different
hyperparameter configurations. On this text, we achieved the best results with
a dimensionality of 128 for the embedding and bi-LSTM layers, using a dropout
of 0.1, and training the model for 30 iterations. These settings were
subsequently used for all further experiments.

###  3.3 Other models used for comparison

For comparison, we also train and evaluate with the Norma tool described by
?), since it was originally developed for the Anselm corpus and the
implementation is publicly available.333https://github.com/comphist/norma
However, Norma actually consists of a combination of three different
normalization methods, one of which is a simple wordlist mapping of historical
tokens to normalized forms. Since this wordlist mapping is conceptually very
simple and could easily be added to our (or any other) normalization method,
we exclude it for the comparison, and only use Norma’s remaining two
algorithms (which we denote Norma∗).

Additionally, since we frame the problem as a sequence labeling task, we
compare our results to a simple sequence labeling model using conditional
random fields (CRF). The CRF model gets the same input/output sequences as our
bi-LSTM model (cf. Sec. 2.1), and uses the two preceding and following
characters from the historical wordform as additional features. Implementation
was done with CRFsuite [Okazaki (2007] using the averaged perceptron algorithm
for training.

##  4 Evaluation

ID | Region | Tokens | Norma∗ | CRF | Bi-LSTM
---|---|---|---|---|---
|  |  | s | s+a | s | s+a | s | s+a†
B | East Central | 4,718 | 80.30% | 77.80% | 76.30% | 72.80% | 79.20% | 81.70%
D3 | East Central | 5,704 | 80.50% | 80.20% | 77.20% | 73.00% | 80.10% | 81.20%
H | East Central | 8,427 | 82.70% | 82.90% | 78.60% | 76.00% | 85.00% | 82.30%
B2 | West Central | 9,145 | 76.10% | 77.60% | 74.60% | 71.70% | 82.00% | 79.60%
KÄ1492 | West Central | 7,332 | 77.50% | 74.40% | 74.80% | 68.40% | 81.60% | 80.50%
KJ1499 | West Central | 7,330 | 77.00% | 72.90% | 73.50% | 68.40% | 84.50% | 79.20%
N1500 | West Central | 7,272 | 76.70% | 75.30% | 72.70% | 67.20% | 79.00% | 79.20%
N1509 | West Central | 7,418 | 78.10% | 73.30% | 74.30% | 68.80% | 80.80% | 80.10%
N1514 | West Central | 7,412 | 78.30% | 73.80% | 72.20% | 69.90% | 79.00% | 80.10%
St | West Central | 7,407 | 72.60% | 73.80% | 70.30% | 68.70% | 75.50% | 75.20%
D4 | Upper/Central | 5,806 | 75.60% | 75.60% | 72.40% | 70.90% | 76.50% | 76.60%
N4 | Upper | 8,593 | 78.20% | 78.10% | 80.00% | 78.40% | 81.80% | 83.40%
s1496/97 | Upper | 5,840 | 81.70% | 83.40% | 77.70% | 76.90% | 83.00% | 84.10%
B3 | East Upper | 6,222 | 80.80% | 80.60% | 79.50% | 79.10% | 81.50% | 83.20%
Hk | East Upper | 8,690 | 77.80% | 79.30% | 78.20% | 77.90% | 80.90% | 82.20%
M | East Upper | 8,700 | 74.30% | 74.40% | 72.80% | 68.40% | 83.90% | 80.90%
M2 | East Upper | 8,729 | 75.80% | 76.00% | 75.10% | 72.40% | 76.70% | 80.20%
M3 | East Upper | 7,929 | 79.00% | 79.70% | 77.30% | 74.10% | 80.40% | 79.60%
M5 | East Upper | 4,705 | 80.60% | 80.70% | 76.40% | 78.30% | 77.70% | 82.90%
M6 | East Upper | 4,632 | 75.90% | 76.30% | 73.70% | 74.40% | 75.20% | 79.30%
M9 | East Upper | 4,739 | 82.20% | 81.50% | 79.00% | 76.90% | 80.40% | 83.60%
M10 | East Upper | 4,379 | 77.00% | 78.60% | 76.00% | 75.80% | 75.10% | 81.30%
Me | East Upper | 4,560 | 79.70% | 80.10% | 76.90% | 75.50% | 80.30% | 83.70%
Sb | East Upper | 7,218 | 78.00% | 76.60% | 75.70% | 74.80% | 80.00% | 78.50%
T | East Upper | 8,678 | 76.70% | 78.60% | 73.40% | 72.20% | 75.80% | 79.00%
W | East Upper | 8,217 | 75.90% | 78.30% | 78.20% | 77.00% | 81.40% | 80.80%
We | East Upper | 6,661 | 83.10% | 81.50% | 78.60% | 75.80% | 81.50% | 83.10%
Ba | North Upper | 5,934 | 79.80% | 81.20% | 80.20% | 78.70% | 80.70% | 82.80%
Ba2 | North Upper | 5,953 | 81.40% | 80.00% | 78.10% | 77.90% | 82.50% | 84.10%
M4 | North Upper | 8,574 | 76.90% | 76.70% | 75.70% | 75.00% | 79.40% | 82.30%
M7 | North Upper | 4,638 | 79.40% | 79.80% | 75.60% | 74.20% | 78.20% | 82.10%
M8 | North Upper | 8,275 | 78.50% | 77.00% | 78.20% | 78.40% | 81.10% | 82.50%
n | North Upper | 9,191 | 79.60% | 81.30% | 81.90% | 78.20% | 84.40% | 84.70%
N | North Upper | 13,285 | 75.50% | 76.30% | 71.70% | 68.90% | 79.00% | 76.90%
N2 | North Upper | 7,058 | 82.20% | 81.90% | 80.30% | 81.60% | 84.30% | 83.40%
N3 | North Upper | 4,192 | 79.10% | 80.80% | 76.40% | 77.50% | 77.60% | 84.20%
Be | West Upper | 8,203 | 75.50% | 76.40% | 75.30% | 73.40% | 78.80% | 78.00%
Ka | West Upper | 12,641 | 73.80% | 74.10% | 75.40% | 72.80% | 80.10% | 80.30%
SG | West Upper | 7,838 | 80.10% | 79.90% | 78.00% | 76.80% | 81.70% | 80.90%
Sa | West Upper | 8,668 | 72.60% | 73.50% | 71.90% | 71.40% | 76.10% | 76.50%
St2 | West Upper | 8,834 | 73.20% | 73.40% | 73.20% | 73.00% | 78.20% | 79.90%
Stu | West Upper | 8,686 | 77.70% | 77.10% | 76.50% | 72.10% | 79.40% | 77.00%
Sa2 | West Upper | 8,011 | 77.50% | 77.90% | 73.50% | 73.30% | 79.50% | 79.70%
Le | Dutch | 7,087 | 69.50% | 60.30% | 65.00% | 55.80% | 75.60% | 67.50%
Average |  | 7,353 | 77.83% | 77.48% | 75.73% | 73.70% | 79.90% | 80.55%
Table 1: Word accuracy on the Anselm dataset, evaluated on the first 1,000
tokens; s = training set from the same text, s+a = like s, but augmented with
10,000 tokens randomly sampled from the other texts; † = Bi-LSTM (s+a) is the
multi-task learning setup. Best results shown in bold.

We evaluate our model separately for each text in our dataset. From each text,
we use 1,000 tokens as our evaluation set, set aside another 1,000 tokens as a
development set (which was not currently used), and train on the remaining
tokens (between 2,000 and 11,000, depending on the text). Both CRF and our bi-
LSTM model get their input as character sequences (as described in Sec. 2.1),
while Norma requires full words as input.

For the multi-task learning setup, we randomly sample from all Anselm texts
and regard each text as its own task. Effectively, we are learning a joint
model over all Anselm texts with shared parameters but distinct prediction
layers, while viewing the text we are currently evaluating on as our main task
and the others as auxiliary tasks. The MTL setup is only applicable to our bi-
LSTM model; however, since the auxiliary task consists of spelling
normalization with data from the same corpus (although with a higher variety
of different spelling characteristics compared to the target text), it is
possible that the other methods could also profit from this additional
training data. We therefore also evaluate Norma and CRF when the training sets
have been augmented by 10,000 randomly sampled training examples from all
texts.

###  4.1 Word accuracy

Evaluation results in terms of word-level accuracy are presented in Table 1.

Columns “s” show results for the traditional setup without multi-task
learning. The basic bi-LSTM model performs better than Norma on 34 of the 44
texts. On average, there is an increase of $2.1$ percentage points (pp),
although the differences on individual texts vary wildly, from $-2.9$ pp (M5)
to $+9.6$ pp (M), giving a standard deviation of $2.7$ pp. The CRF model, on
the other hand, is almost always worse than Norma, averaging a difference of
$-2.1$ pp ($\pm 2.0$). This indicates that the reformulation of the task as
character-based sequence labeling cannot alone be responsible for the bi-LSTM
results, but the choice of a neural network architecture is crucial, too.

Columns “s+a” present the results when using the augmented training set. For
bi-LSTM, this is the multi-task learning setup—using MTL improves the results
by $+0.7$ pp ($\pm 2.8$) on average, but again there is a high variance within
the individual scores. However, for the other methods, adding the 10,000
randomly selected samples to the training set actually decreases the average
accuracy, by $-0.4$ pp for Norma and $-2.0$ pp for CRF. This is likely due to
the fact that this additional training set introduces a variety of spelling
characteristics that are not found in the target text. While Norma and CRF
cannot handle this out-of-domain training data well, the MTL setup can
actually profit from it in many cases.

Table 1 also shows a rough classification of the dialectal regions from which
the texts originate. There is a slight trend for multi-task learning to be
advantageous on texts from the East and North Upper German regions, while for
the Central and West Upper German texts, there are more instances of the
standard bi-LSTM model (s) being better than the MTL model (s+a). This could
either be due to linguistic properties of these dialectal regions, or due to
the fact that East/North Upper German texts make up the majority of the
dataset, thereby also featuring more prominently in the “s+a” settings.

The latter hypothesis is supported by the case of the ‘Le’ text, which is the
only Dutch text in the sample (but which was nonetheless normalized to modern
German in the corpus). Here, the “s+a” settings of the experiments all show a
dramatic decrease in accuracy (up to $-9.2$ pp), suggesting that it is
disadvantageous to augment the training set with samples that are too
different from the target domain, even for the MTL setup.

In general, however, one of the bi-LSTM models is always best; there is only
one text (We) for which Norma achieves an equal accuracy. This indicates that
deep neural networks can be applied successfully to the spelling normalization
task even with a comparatively small amount of training data. Also, we note
that Norma always requires a lexical resource which it uses to filter results,
while we do not.

###  4.2 Effect of training set size

$100$$500$$1{,}000$$1{,}500$$2{,}000$$2{,}718$$0.35$$0.4$$0.45$$0.5$$0.55$$0.6$$0.65$$0.7$$0.75$$0.8$$0.85$Training
set sizeWord-level accuracysNorma∗CRFBi-LSTM

$100$$500$$1{,}000$$1{,}500$$2{,}000$$2{,}718$$0.35$$0.4$$0.45$$0.5$$0.55$$0.6$$0.65$$0.7$$0.75$$0.8$$0.85$Training
set sizes+aNorma∗CRFBi-LSTM (MTL)

Figure 2: Word accuracy on the ‘B’ text for different sizes of the training
set; left = train only on the training set from ‘B’ (s); right = use augmented
training set/multi-task learning (s+a).

In our evaluation, we use all but the first 2,000 tokens from a text for
training (cf. the beginning of Sec. 4). Consequently, the training sets for
each text are of different sizes. We calculate Spearman’s rank correlation
coefficient ($\rho$) between the size of the training sets and the
normalization accuracy for each column in Table 1. We find no significant
correlation for the CRF and bi-LSTM models ($|\rho|<0.25$), although there
seems to be a moderate _inverse_ correlation for the Norma results
($\rho\approx-0.48$ on Norma “s”). The reasons for this are beyond the scope
of this paper, though.

The question of how much training data is needed to effectively train a model
is particularly relevant for historical spelling normalization, since training
data can be very sparse in this domain. We therefore choose to evaluate each
method in a scenario where we consider a single text, but vary the size of the
training set, to estimate how well they perform with fewer data.

Figure 2 shows the results for different training set sizes on the ‘B’ text.
Not surprisingly, when training on only 100 tokens, accuracy is bad ($<41\%$)
for CRF and bi-LSTM. Norma, on the other hand, already achieves $67.4\%$ in
this scenario. The biggest gains for all three methods can be seen for
training set sizes between 100 and 1,000 tokens—for larger set sizes, the
gains become less, and all three methods are within close range of each other.

For the “s+a” scenario, all models have noticeably higher accuracy even with
only 100 tokens from the ‘B’ text. However, the increases for Norma and CRF
are not as high as in the “s” scenario; this is not surprising, since the
total training set for these methods always contains at least 10,000 tokens
(from the auxiliary set), and it is only the proportion of tokens coming from
the ‘B’ text that increases. The bi-LSTM model with multi-task learning
behaves differently, though: while it starts off as the weakest model (on 100
tokens), it is the best model when training on 1,000 tokens or more.

These learning curves illustrate that the MTL setup is fundamentally different
from adding the auxiliary data to the training set normally, as is the case
with CRF and the Norma tool. They also show that our bi-LSTM models can be
better than or at least competitive with CRF/Norma for training set sizes as
low as 1,000 tokens.

###  4.3 Multi-task learning with grapheme-to-phoneme mappings

It is conceivable to use different tasks than historical spelling
normalization as the auxiliary task in a multi-task learning setup. In
particular, we also experimented with grapheme-to-phoneme mapping as the
auxiliary task, since it can be seen as a similar form of character-based
sequence transduction.

For our dataset, we used the German part of the CELEX lexical database [Baayen
et al. (1995], particularly the database of phonetic transcriptions of German
wordforms. The database contains a total of 365,530 wordforms with
transcriptions in DISC format, which assigns one character to each distinct
phonological segment (including affricates and diphthongs). For example, the
word _Jungfrau_ ‘virgin’ is represented as ’jUN-frB. We randomly sampled 4,000
tokens from this dataset for our experiment, and used the same algorithm as
for the historical data to convert these mappings to a character-based
sequence representation (cf. Sec. 2.1).

The evaluation, however, showed no real benefit of this MTL setup compared to
the bi-LSTM model without MTL. While accuracy increased for some texts by up
to $2.6$ pp, it decreased slightly for the majority of texts, averaging to a
$-0.4$ pp difference to the basic model.

##  5 Related Work

Various methods have been proposed to perform spelling normalization on
historical texts; for an overview, see ?). Many approaches use edit distance
calculations or some form of character-level rewrite rules, but require either
hand-crafting of the rules [Baron and Rayson (2008] or a lexical resource to
filter their output [Bollmann (2012, Porta et al. (2013].

A newer approach is the application of character-based statistical machine
translation [Pettersson et al. (2013, Sánchez-Martínez et al. (2013, Scherrer
and Erjavec (2013]. In contrast to our sequence labeling approach, these
methods do not require a fixed character alignment between wordforms, but it
is not clear whether this is actually an advantage. To our knowledge, a
comparative evaluation between these methods and other approaches has not yet
been done.

?) present the only other approach we are aware of that applies neural
networks to normalization of historical data. They also use bi-directional
LSTMs, but differ from our approach in the way they perform alignment between
historical and modern wordforms. More importantly, they evaluate their model
on a single dataset, the Luther bible, which has much more regular spelling
than the texts in the Anselm corpus and is also significantly longer: they use
about 200,000 tokens for their training set.

##  6 Conclusion and Future Work

We presented an approach to historical spelling normalization using bi-
directional long short-term memory networks and showed that it outperforms a
CRF baseline and the Norma tool by ?) for almost all of the texts in our
dataset, a diverse corpus of Early New High German, despite using a relatively
low amount of training data (about 2,000 to 11,000 tokens) and not making use
of a lexical resource (like Norma does). We showed further that multi-task
learning with additional normalization data can improve accuracy with bi-
LSTMs, while adding the same data to the training set of Norma and CRF does
not help on average, and can even be detrimental.

Many improvements to this approach are conceivable. Character-based
statistical machine translation has been successfully applied to spelling
normalization (cf. Sec. 5), but we are not aware of any experiments with
neural machine translation [Cho et al. (2014] on this domain. Using an
encoder–decoder architecture, e.g. similar to ?), would remove the need for an
explicit character alignment (cf. Sec. 2.1) but could also make the model more
complex and potentially more difficult to train, so it is unclear whether this
would be an improvement to our approach.

With regard to multi-task learning, our results seem to indicate that for the
auxiliary task, it is preferable to use data with similar characteristics to
the data in the main task. On the other hand, depending on the language
variety to be annotated, such data might not always be readily available. We
would therefore like to do further experiments with auxiliary data from
different corpora or even different string transduction tasks, to see if and
under which conditions they can have a beneficial effect on the spelling
normalization task.

## Acknowledgments

Marcel Bollmann was supported by Deutsche Forschungsgemeinschaft (DFG), Grant
DI 1558/4.

## References

  * [Amoia and Martinez (2013]  Marilisa Amoia and Jose Manuel Martinez.  2013\.  Using comparable collections of historical texts for building a diachronic dictionary for spelling normalization.  In Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 84–89, Sofia, Bulgaria.
  * [Azawi et al. (2013]  Mayce Al Azawi, Muhammad Zeshan Afzal, and Thomas M. Breuel.  2013\.  Normalizing historical orthography for OCR historical documents using LSTM.  In Proceedings of the 2nd International Workshop on Historical Document Imaging and Processing, pages 80–85. ACM.
  * [Baayen et al. (1995]  R. Harald Baayen, Richard Piepenbrock, and Léon Gulikers.  1995\.  The CELEX lexical database (Release 2) (CD-ROM).  Linguistic Data Consortium, University of Pennsylvania, Philadelphia, PA.
  * [Baron and Rayson (2008]  Alistair Baron and Paul Rayson.  2008\.  VARD 2: A tool for dealing with spelling variation in historical corpora.  In Proceedings of the Postgraduate Conference in Corpus Linguistics.
  * [Bollmann (2012]  Marcel Bollmann.  2012\.  (Semi-)automatic normalization of historical texts using distance measures and the Norma tool.  In Proceedings of the Second Workshop on Annotation of Corpora for Research in the Humanities (ACRH-2), Lisbon, Portugal.
  * [Caruana (1993]  Rich Caruana.  1993\.  Multitask learning: A knowledge-based source of inductive bias.  In Proceedings of the 10th International Conference on Machine Learning (ICML), pages 41–48.
  * [Cho et al. (2014]  Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio.  2014\.  On the properties of neural machine translation: Encoder–decoder approaches.  In Proceedings of the Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), pages 103–111, Doha, Qatar.
  * [Collobert et al. (2011]  Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  2011\.  Natural language processing (almost) from scratch.  The Journal of Machine Learning Research, 12:2493–2537.
  * [Dipper and Schultz-Balluff (2013]  Stefanie Dipper and Simone Schultz-Balluff.  2013\.  The Anselm corpus: Methods and perspectives of a parallel aligned corpus.  In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics.
  * [Hochreiter and Schmidhuber (1997]  Sepp Hochreiter and Jürgen Schmidhuber.  1997\.  Long short-term memory.  Neural Computation, 9(8):1735–1780.
  * [Jurish (2010]  Bryan Jurish.  2010\.  More than words: using token context to improve canonicalization of historical German.  Journal for Language Technology and Computational Linguistics, 25(1):23–39.
  * [Klerke et al. (2016]  Sigrid Klerke, Yoav Goldberg, and Anders Søgaard.  2016\.  Improving sentence compression by learning to predict gaze.  In Proceedings of NAACL-HLT 2016, pages 1528–1533, San Diego, CA.
  * [Krasselt et al. (2015]  Julia Krasselt, Marcel Bollmann, Stefanie Dipper, and Florian Petran.  2015\.  Guidelines for normalizing historical German texts.  Bochumer Linguistische Arbeitsberichte, 15.
  * [Levenshtein (1966]  Vladimir I. Levenshtein.  1966\.  Binary codes capable of correcting deletions, insertions, and reversals.  Soviet Physics Doklady, 10(8):707–710.
  * [Luong et al. (2016]  Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser.  2016\.  Multi-task sequence to sequence learning.  arXiv:1511.06114v4.
  * [Okazaki (2007]  Naoaki Okazaki.  2007\.  CRFsuite: a fast implementation of conditional random fields (CRFs).  http://www.chokkan.org/software/crfsuite/.
  * [Pettersson et al. (2013]  Eva Pettersson, Beáta Megyesi, and Jörg Tiedemann.  2013\.  An SMT approach to automatic annotation of historical text.  In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics, Oslo, Norway.
  * [Piotrowski (2012]  Michael Piotrowski.  2012\.  Natural Language Processing for Historical Texts.  Number 17 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool, San Rafael, CA.
  * [Porta et al. (2013]  Jordi Porta, José-Luis Sancho, and Javier Gómez.  2013\.  Edit transducers for spelling variation in Old Spanish.  In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics, Oslo, Norway.
  * [Scherrer and Erjavec (2013]  Yves Scherrer and Tomaž Erjavec.  2013\.  Modernizing historical Slovene words with character-based SMT.  In Proceedings of the 4th Biennial Workshop on Balto-Slavic Natural Language Processing, Sofia, Bulgaria.
  * [Sutskever et al. (2014]  Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  2014\.  Sequence to sequence learning with neural networks.  In Advances in Neural Information Processing Systems (NIPS 2014), number 27, pages 3104–3112.
  * [Sánchez-Martínez et al. (2013]  Felipe Sánchez-Martínez, Isabel Martínez-Sempere, Xavier Ivars-Ribes, and Rafael C. Carrasco.  2013\.  An open diachronic corpus of historical Spanish: annotation criteria and automatic modernisation of spelling.  arXiv:1306.3692v1, 06.
  * [Wieling et al. (2009]  Martijn Wieling, Jelena Prokić, and John Nerbonne.  2009\.  Evaluating the pairwise string alignment of pronunciations.  In Proceedings of the EACL 2009 Workshop on Language Technology and Resources for Cultural Heritage, Social Sciences, Humanities, and Education (LaTeCH – SHELT&R 2009), pages 26–34, Athens, Greece.