An $S^{3}\times S^{3}$ bundle over $S^{4}$ and new supergravity solutions

Ali Imaanpur

††footnotetext: Email<EMAIL_ADDRESS>
Department of Physics, School of Sciences
Tarbiat Modares University, P.O.Box 14155-4838, Tehran, Iran

The metric of $S^{7}$ can be written as an $SU(2)$-instanton bundle over
$S^{4}$. It is also possible to write it differently as an anti-instanton
bundle. We use this observation to construct an instanton–anti-instanton,
$SU(2)\times SU(2)$, bundle over $S^{4}$. We show that this 10d manifold
admits two Einstein metrics. We then rewrite the metric to isolate two $U(1)$
directions of the fibre and dimensionally reduce along them to get an eight
dimensional metric describing an $S^{2}\times S^{2}$ bundle over $S^{4}$. This
metric allows a harmonic 4-form which we use to derive new supergravity
solutions in eleven and ten dimensions as $AdS_{3}\times M_{8}$ and
$AdS_{2}\times M_{8}$, respectively.

##  1 Introduction

Among many different types of supergravity solutions, compactification to anti
de Sitter (AdS) spaces are more typical, and depending on the compact manifold
and the fluxes that one turns on different amounts of supersymmetries are
preserved, or it is totally broken [1, 2, 3, 4, 5]. Exact vacuum solutions,
i.e., without source, are of particular interest in AdS/CFT duality, which
relates string theory in AdS space to a particular gauge theory on its
boundary. In this context, since supersymmetry puts strong constraints on the
dynamics, supersymmetric solutions are more tractable on the gauge theory
side. On the other hand, nonsupersymmetric solutions are closer to model
realistic theories like QCD [6].

$AdS_{4}\times S^{7}$ is an exact solution of eleven-dimensional supergravity
which also shows up as the near horizon limit of an M2-brane solution. Writing
the metric of $S^{7}$ as an $SU(2)$ bundle over $S^{4}$ has allowed to obtain
new solutions where the fibre is squashed [7, 8, 9, 10]. In fact, this way of
expressing the metric reveals how the Yang-Mills solutions, as instantons on
$S^{4}$, are embedded into the metric of $S^{7}$ as an Einstein manifold.
Interestingly, one can replace the instanton with an anti-instanton to get an
equivalent metric on $S^{7}$. Now, from gauge theory we know that, if the
gauge group is large enough, instanton–anti-instanton configurations are also
a solution to the Yang-Mills equations, so it is curious to think if such
bundles can be constructed on $S^{4}$. This is what we do in this paper by
constructing an $SU(2)\times SU(2)$ bundle over $S^{4}$. The ten-dimensional
metric is Einstein for two specific values of the squashing parameters.
Noticing that the $S^{7}$ metric can also be written as a $U(1)$ bundle over
${\bf CP}^{3}$, we rewrite the ten-dimensional metric as a $U(1)\times U(1)$
bundle over an eight-dimensional base. It then becomes possible to reduce
along these directions and get an eight-dimensional metric. This metric admits
a harmonic 4-form which we then use to write down our ansatz and derive the
solutions.

In the next section we begin with recalling the $S^{3}$ fibration of $S^{7}$.
The fibers are woven with an instanton connection over $S^{4}$. Replacing the
instanton connection with an anti-instanton one gives an equivalent metric of
$S^{7}$. We combine the two structures to write a 10-dimensional metric. The
point of using an anti-instanton connection, instead of an instanton one, is
to get the Ricci tensor with constant components, in the corresponding
orthonormal basis. To derive supergravity solutions, in section 3, we rewrite
the 10d metric so as to isolate two $U(1)$ fiber directions. This allows to
reduce the 10d metric to an 8d metric simply by dropping the two $U(1)$
fibers, and at the same time have a Ricci tensor with constant components. We
will show that the 8d manifold, $M_{8}$, endowed with this metric admits a
harmonic 4-form, which we then use to write our ansatz for the 4-form field
strength, $F_{4}$, in eleven-dimensional and type IIA supergravities. The
solutions will be in the form of $AdS_{3}\times M_{8}$ and $AdS_{2}\times
M_{8}$. Eleven-dimensional supergravity compactification to $AdS_{3}$, and
type IIB compactification to $AdS_{2}$ have been studied earlier in [11, 12].

##  2 An $SU(2)\times SU(2)$ bundle over $S^{4}$

In this section, we begin with describing the metric of $S^{7}$ as an
$SU(2)$-instanton bundle over $S^{4}$, and then proceed to add an extra
$SU(2)$-anti-instanton fiber to construct an $SU(2)\times SU(2)$ bundle over
$S^{4}$. We could of course choose the extra fiber structure to be an
instanton, but the corresponding Ricci tensor components of the metric would
not be constant. The round metric on $S^{7}$ can be written as an $SU(2)$
bundle over $S^{4}$ [7, 13],

$ds^{2}_{S^{7}}=d\mu^{2}+\frac{1}{4}\sin^{2}\mu\,\Sigma_{i}^{2}+(\sigma_{i}-\cos^{2}{\mu}/{2}\
\Sigma_{i})^{2}\,,$ (1)

with $0\leq\mu\leq\pi$. $\Sigma_{i}$’s and $\sigma_{i}$’s are two sets of
left-invariant one-forms

$\displaystyle\Sigma_{1}=\cos\gamma\,d\alpha+\sin\gamma\sin\alpha\,d\beta\,,$
$\displaystyle\Sigma_{2}=-\sin\gamma\,d\alpha+\cos\gamma\sin\alpha\,d\beta\,,\
$ $\displaystyle\Sigma_{3}=d\gamma+\cos\alpha\,d\beta\,,$

where $0\leq\gamma\leq 4\pi,\,0\leq\alpha\leq\pi,\,0\leq\beta\leq 2\pi$, and
with a similar expression for $\sigma_{i}$’s. They satisfy the $SU(2)$
algebra, namely,

$d\Sigma_{i}=-\frac{1}{2}\,\epsilon_{ijk}\,\Sigma_{j}{\wedge}\Sigma_{k}\,,\ \
\ \ d\sigma_{i}=-\frac{1}{2}\,\epsilon_{ijk}\,\sigma_{j}{\wedge}\sigma_{k}\,,$
(2)

with $i,j,k,\ldots=1,2,3$.

Squashing corresponds to modifying the round metric on $S^{7}$ as follows

$ds^{2}_{S^{7}}=d\mu^{2}+\frac{1}{4}\sin^{2}\mu\,\Sigma_{i}^{2}+\lambda^{2}(\sigma_{i}-A_{i})^{2}\,,$
(3)

with $\lambda$ the squashing parameter. Here, to see explicitly how the
instantons are embedded into the metric, we have written the fibers in terms
of three $SU(2)$ one-form gauge fields

$A_{i}=\cos^{2}{\mu}/{2}\ \Sigma_{i}\,,$ (4)

where $i$ is considered as the algebra index. The gauge field strength is

$F_{i}=dA_{i}+\frac{1}{2}\epsilon_{ijk}A^{j}{\wedge}A^{k}\,.$ (5)

If we choose the following orthonormal basis of vielbeins

$e^{0}=\,d\mu\,,\ \ e^{i}=\frac{1}{2}\sin\mu\,\Sigma_{i}\,,\ \ \ \
{\hat{e}}^{i}=\lambda(\sigma_{i}-A_{i})\,,$ (6)

we can see that the gauge field strengths of connection (4) are anti-self dual
on the base manifold

$\displaystyle F_{1}\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\\!dA_{1}+A_{2}{\wedge}A_{3}=-e^{0}{\wedge}e^{1}+e^{2}{\wedge}e^{3}\,,$
$\displaystyle F_{2}\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\\!dA_{2}+A_{3}{\wedge}A_{1}=-e^{0}{\wedge}e^{2}+e^{3}{\wedge}e^{1}\,,$
$\displaystyle F_{3}\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\\!dA_{3}+A_{1}{\wedge}A_{2}=-e^{0}{\wedge}e^{3}+e^{1}{\wedge}e^{2}\,.$
(7)

Interestingly, we could have instead taken

$\tilde{A}_{i}=\sin^{2}{\mu}/{2}\ \Sigma_{i}\,,$ (8)

and get a self-dual field strength;

$\displaystyle{\tilde{F}}_{1}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!e^{0}{\wedge}e^{1}+e^{2}{\wedge}e^{3}\,,$
$\displaystyle{\tilde{F}}_{2}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!e^{0}{\wedge}e^{2}+e^{3}{\wedge}e^{1}\,,$
$\displaystyle{\tilde{F}}_{3}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!e^{0}{\wedge}e^{3}+e^{1}{\wedge}e^{2}\,.$ (9)

Either of gauge connections (4) or (8), could be used in (3) to get a metric
with constant Ricci tensor components. In basis (6), we derive

$\displaystyle R_{\alpha\beta}$ $\displaystyle=$
$\displaystyle\left(3-\frac{3\lambda^{2}}{2}\right)\,\delta_{\alpha\beta}\,,$
$\displaystyle R_{\hat{\alpha}\hat{\beta}}$ $\displaystyle=$
$\displaystyle\left(\frac{1+2\lambda^{4}}{2\lambda^{2}}\right)\,\delta_{\hat{\alpha}\hat{\beta}}\,,$
(10)

where $\alpha,\beta=1,2,3,4$ and $\hat{\alpha},\hat{\beta}=5,6,7$ are the base
and the fiber indices, respectively. Note that the metric becomes Einstein for
$\lambda=1$ and $\lambda=1/\sqrt{5}$.

Now, we would like to embed the above two structures into a single ten-
dimensional metric, namely, an instanton–anti-instanton bundle over $S^{4}$.
This is done simply by adding an extra 3-dimensional anti-instanton fiber to
metric (3);

$ds^{2}=d\mu^{2}+\frac{1}{4}\sin^{2}\mu\,\Sigma_{i}^{2}+\lambda^{2}(\sigma_{i}-\cos^{2}{\mu}/{2}\,\Sigma_{i})^{2}+\tilde{\lambda}^{2}(\tau_{i}-\sin^{2}{\mu}/{2}\,\Sigma_{i})^{2}\,,$
(11)

with $\tilde{\lambda}$ a constant parameter. Computing the Ricci tensor in the
following orthonormal basis

$e^{0}=\,d\mu\,,\ \ e^{i}=\frac{1}{2}\sin\mu\,\Sigma_{i}\,,\ \ \ \
{\hat{e}}^{i}=\lambda(\sigma_{i}-A_{i})\,,\ \ \ \
\tilde{e}^{i}=\lambda(\tau_{i}-\tilde{A}_{i})\,,$ (12)

results in

$\displaystyle R_{\alpha\beta}$ $\displaystyle=$
$\displaystyle\left(3-\frac{3\lambda^{2}}{2}-\frac{3\tilde{\lambda}^{2}}{2}\right)\,\delta_{\alpha\beta}\,,$
$\displaystyle R_{\hat{\alpha}\hat{\beta}}$ $\displaystyle=$
$\displaystyle\left(\frac{1+2\lambda^{4}}{2\lambda^{2}}\right)\,\delta_{\hat{\alpha}\hat{\beta}}\,,$
$\displaystyle R_{{\tilde{\alpha}}{\tilde{\beta}}}$ $\displaystyle=$
$\displaystyle\left(\frac{1+2\tilde{\lambda}^{4}}{2\tilde{\lambda}^{2}}\right)\,\delta_{{\tilde{\alpha}}{\tilde{\beta}}}\,,$
(13)

where ${\tilde{\alpha}},{\tilde{\beta}}=8,9,10$ are the tangent indices of the
extra anti-instanton fiber. We observe that for
$\lambda^{2}=\tilde{\lambda}^{2}=1/2$, and
$\lambda^{2}=\tilde{\lambda}^{2}=1/4$, the metric is Einstein, with
$R_{MN}=3/2\,\delta_{MN}$, and $R_{MN}=9/4\,\delta_{MN}$, respectively, ($M$
and $N$ being the collective indices). In the present method of deriving
supergravity solutions it is crucial for the Ricci tensor components to be
constant. Had we used instanton connections as an extra 3-dimensional fiber,
the Ricci tensor would not have had constant components.

It is interesting to examine the structure of the metric a bit further, and
show that it indeed admits a harmonic 3-form. Let us first introduce
$\omega_{3}$ and $\tilde{\omega}_{3}$, the volume elements of the instanton
and the anti-instanton fibers, respectively:

$\omega_{3}={\hat{e}}^{1}{\wedge}{\hat{e}}^{2}{\wedge}{\hat{e}}^{3}\,,\ \ \ \
\tilde{\omega}_{3}=\tilde{e}^{1}{\wedge}\tilde{e}^{2}{\wedge}\tilde{e}^{3}\,,$
(14)

taking the exterior derivative, we obtain

$d\omega_{3}=\frac{\lambda}{2}\left(\epsilon_{ijk}\,e^{0}{\wedge}e^{i}{\wedge}{\hat{e}}^{j}{\wedge}{\hat{e}}^{k}+\,e^{i}{\wedge}e^{j}{\wedge}{\hat{e}}^{i}{\wedge}{\hat{e}}^{j}\right)\,.$
(15)

The Hodge dual, excluding the extra three anti-instanton fibers, reads

$*_{7}d\omega_{3}={\lambda}\,{\hat{e}}^{i}{\wedge}\,(e^{0}{\wedge}e^{i}+\frac{1}{2}\epsilon_{ijk}\,e^{j}{\wedge}e^{k})\,,$
(16)

so, we derive

$d*_{7}d\omega_{3}={6\lambda^{2}}\,\omega_{4}-\frac{1}{\lambda}\,d\omega_{3}\,,$
(17)

where

$\omega_{4}=e^{0}{\wedge}e^{1}{\wedge}e^{2}{\wedge}e^{3}\,,$ (18)

is the volume element of the base. Note that $\omega_{4}$ is closed;
$d\omega_{4}=0$. Similarly,

$d{\tilde{*}}_{7}d\tilde{\omega}_{3}=-{6\tilde{\lambda}^{2}}\,\omega_{4}-\frac{1}{\tilde{\lambda}}\,d\tilde{\omega}_{3}\,,$
(19)

with

${\tilde{*}}_{7}d\tilde{\omega}_{3}={\tilde{\lambda}}\,\tilde{e}^{i}{\wedge}\,(e^{0}{\wedge}e^{i}-\frac{1}{2}\epsilon_{ijk}\,e^{j}{\wedge}e^{k})\,.$
(20)

As with $*_{7}$ which defines the Hodge dual with anti-instanton fibers
excluded, ${\tilde{*}}_{7}$ is the Hodge dual with instanton fibers excluded.
Now, we claim that the metric admits a harmonic 3-form. For simplicity, let us
set $\lambda=\tilde{\lambda}$. Looking at eqs. (17) and (19), we observe that

$\Omega_{3}=*_{7}d\omega_{3}+{\tilde{*}}_{7}d\tilde{\omega}_{3}+\frac{1}{\lambda}(\omega_{3}+\tilde{\omega}_{3})\,,$
(21)

is closed. Taking the Hodge dual, we get

$*\Omega_{3}=d\omega_{3}{\wedge}\tilde{\omega}_{3}+\omega_{3}{\wedge}d\tilde{\omega}_{3}+\frac{1}{\lambda}\omega_{4}{\wedge}(\tilde{\omega}_{3}-\omega_{3})\,,$
(22)

this is also closed since $d\omega_{3}{\wedge}d\tilde{\omega}_{3}=0$, as
$d\omega_{3}$ is self-dual (with respect to the base manifold $S^{4}$) while
$d\tilde{\omega}_{3}$ is anti-self-dual. Also note that
$\omega_{4}{\wedge}d\omega_{3}=\omega_{4}{\wedge}d\tilde{\omega}_{3}=0$. So,
we conclude that

$d\Omega_{3}=d*\Omega_{3}=0\,,$ (23)

namely, the 10-dimensional manifold with metric (11) admits a harmonic 3-from.

Using the three closed 4-forms, $\omega_{4},d\omega_{3}$, and
$d\tilde{\omega}_{3}$, we could proceed along the lines of [14] to try an
ansatz for the 4-form field, $F_{4}$, and solve the equations of motion.
However, in type IIA case, the number of independent equations will exceed the
number of free parameters, and so there will be no solution. For eleven-
dimensional supergravity, we may consider a trivial circle bundle,
$S^{1}\times M_{10}$, with $M_{10}$ the 10d manifold we constructed in this
section. But, in this case, the energy-momentum tensor of $F_{4}$ backreacts
on $S^{1}$, and since $S^{1}$ is flat there will be no solution. One way out
is to observe that the 10d metric can be rewritten as a $U(1)\times U(1)$
bundle over an 8d manifold. The $U(1)$ factors can then be scaled away leaving
an eight-dimensional manifold. This is what we do in the next section.

##  3 $S^{2}\times S^{2}$ bundle over $S^{4}$

In the beginning of the previous section, the metric of $S^{7}$ was written as
an $SU(2)$ bundle over $S^{4}$. It is also possible to write the metric as a
$U(1)$ bundle over ${\bf CP}^{3}$. Further, it is observed that ${\bf CP}^{3}$
itself can be written as an $S^{2}$ bundle over $S^{4}$. In this form one can
construct a family of homogeneous metrics by rescaling the fibers [15, 16]. In
this section, first we rewrite the $S^{7}$ metric, (1), as a $U(1)$ bundle
over ${\bf CP}^{3}$ [14], and then repeat the same process for the 10d metric
we constructed in the previous section. This allows to isolate two $U(1)$
directions of the metric and then dimensionally reduce along them to get an
8-dimensional metric.

Beginning with metric (3), note that we can also write it as

$\displaystyle ds^{2}_{S^{7}}\\!$ $\displaystyle=$
$\displaystyle\\!d\mu^{2}+\frac{1}{4}\sin^{2}\mu\,\Sigma_{i}^{2}+\lambda^{2}(\sigma_{i}-\cos^{2}{\mu}/{2}\
\Sigma_{i})^{2}$ (24) $\displaystyle=$
$\displaystyle\\!d\mu^{2}+\frac{1}{4}\sin^{2}\mu\,\Sigma_{i}^{2}+\lambda^{2}\sin^{2}\theta_{1}\,(d\phi_{1}-\cot\theta_{1}(\cos\phi_{1}A_{1}+\sin\phi_{1}A_{2})+A_{3})^{2}$
$\displaystyle+$
$\displaystyle\\!\lambda^{2}(d\theta_{1}-\sin\phi_{1}A_{1}+\cos\phi_{1}A_{2})^{2}+\lambda^{2}(d\tau-A)^{2}\,,$

where

$A=\cos\theta_{1}\,d\phi_{1}+\sin\theta_{1}(\cos\phi_{1}A_{1}+\sin\phi_{1}A_{2})+\cos\theta_{1}A_{3}\,,$
(25)

and $\sigma_{i}$’s are left-invariant one-forms that are chosen as follows:

$\displaystyle\sigma_{1}=\sin\phi_{1}\,d\theta_{1}+\sin\theta_{1}\cos\phi_{1}\,d\chi_{1}\,,$
$\displaystyle\sigma_{2}=-\cos\phi_{1}\,d\theta_{1}+\sin\theta_{1}\sin\phi_{1}\,d\chi_{1}\,,\
$ $\displaystyle\sigma_{3}=-d\phi_{1}+\cos\theta_{1}\,d\chi_{1}\,.$

The point of writing the $S^{7}$ metric in this form is that the last $U(1)$
factor in (24) can further be rescaled, i.e., $\lambda\to\lambda^{\prime}$, so
that the Ricci tensor components (in a basis we introduce shortly) are still
constant. We use this observation to rewrite 10d metric (11) as a $U(1)\times
U(1)$ bundle over a base, which itself is an $S^{2}\times S^{2}$ bundle over
$S^{4}$:

$\displaystyle ds^{2}_{10}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!d\mu^{2}+\frac{1}{4}\sin^{2}\mu\,\Sigma_{i}^{2}+\lambda^{2}(d\theta_{1}-\sin\phi_{1}A_{1}+\cos\phi_{1}A_{2})^{2}$
(26) $\displaystyle+$
$\displaystyle\\!\\!\lambda^{2}\sin^{2}\theta_{1}\,(d\phi_{1}-\cot\theta_{1}(\cos\phi_{1}A_{1}+\sin\phi_{1}A_{2})+A_{3})^{2}+\lambda^{\prime
2}(d\chi_{1}-A)^{2}$ $\displaystyle+$
$\displaystyle\\!\\!\tilde{\lambda}^{2}(d\theta_{2}-\sin\phi_{2}\tilde{A}_{1}+\cos\phi_{2}\tilde{A}_{2})^{2}$
$\displaystyle+$
$\displaystyle\\!\\!\tilde{\lambda}^{2}\sin^{2}\theta_{2}\,(d\phi_{2}-\cot\theta_{2}(\cos\phi_{2}\tilde{A}_{1}+\sin\phi_{2}\tilde{A}_{2})+\tilde{A}_{3})^{2}+\tilde{\lambda}^{\prime
2}(d\chi_{2}-\tilde{A})^{2}$

whereas, just as in (25), $\tilde{A}$ is defined in terms of $\tilde{A}_{i}$;

$\tilde{A}=\cos\theta_{2}\,d\phi_{2}+\sin\theta_{2}(\cos\phi_{2}\tilde{A}_{1}+\sin\phi_{2}\tilde{A}_{2})+\cos\theta_{2}\tilde{A}_{3}\,.$
(27)

Note that here we have taken $\tau_{i}$’s in (11) as follows:

$\displaystyle\tau_{1}=\sin\phi_{2}\,d\theta_{2}+\sin\theta_{2}\cos\phi_{2}\,d\chi_{2}\,,$
$\displaystyle\tau_{2}=-\cos\phi_{2}\,d\theta_{2}+\sin\theta_{2}\sin\phi_{2}\,d\chi_{2}\,,\
$ $\displaystyle\tau_{3}=-d\phi_{2}+\cos\theta_{2}\,d\chi_{2}\,,$

and rescaled two $U(1)$ fibers in (26) as $\lambda\to\lambda^{\prime}\,,\
\tilde{\lambda}\to\tilde{\lambda}^{\prime}$.

Choosing the following basis

$\displaystyle e^{0}=\,d\mu\,,\ \ \ \ e^{i}=\frac{1}{2}\sin\mu\,\Sigma_{i}\,,$
$\displaystyle
e^{5}=\lambda(d\theta_{1}-\sin\phi_{1}A_{1}+\cos\phi_{1}A_{2})\,,$
$\displaystyle
e^{6}=\lambda\sin\theta_{1}(d\phi_{1}-\cot\theta_{1}(\cos\phi_{1}A_{1}+\sin\phi_{1}A_{2})+A_{3})\,,$
$\displaystyle e^{7}=\lambda^{\prime}(d\chi_{1}-A)\,,$
$\displaystyle\tilde{e}^{5}=\tilde{\lambda}(d\theta_{2}-\sin\phi_{2}\tilde{A}_{1}+\cos\phi_{2}\tilde{A}_{2})\,,$
$\displaystyle\tilde{e}^{6}=\tilde{\lambda}\sin\theta_{2}(d\phi_{2}-\cot\theta_{2}(\cos\phi_{2}\tilde{A}_{1}+\sin\phi_{2}\tilde{A}_{2})+\tilde{A}_{3})\,,$
$\displaystyle\tilde{e}^{7}=\tilde{\lambda}^{\prime}(d\chi_{2}-\tilde{A})\,,$
(28)

the Ricci tensor turns out to be diagonal and reads

$\displaystyle
R_{00}=R_{11}=R_{22}=R_{33}=3-\lambda^{2}-\tilde{\lambda}^{2}-{\lambda^{\prime
2}}/{2}-{\tilde{\lambda}^{\prime 2}}/{2}\,,$ $\displaystyle
R_{55}=R_{66}=\lambda^{2}+{1}/{\lambda^{2}}-{\lambda^{\prime
2}}/{2\lambda^{4}}\,,\ \ \ \ \ R_{77}=\lambda^{\prime 2}+{\lambda^{\prime
2}}/{2\lambda^{4}}\,,$ $\displaystyle
R_{88}=R_{99}=\tilde{\lambda}^{2}+{1}/{\tilde{\lambda}^{2}}-{\tilde{\lambda}^{\prime
2}}/{2\tilde{\lambda}^{4}}\,,\ \ \ \ \ R_{10,10}=\tilde{\lambda}^{\prime
2}+{\tilde{\lambda}^{\prime 2}}/{2\tilde{\lambda}^{4}}\,.$ (29)

This explicitly shows that we can rescale two $U(1)$ fiber directions along
$\chi_{1}$ and $\chi_{2}$ and still have a diagonal Ricci tensor with constant
components. In fact, we can set $\lambda^{\prime}=\tilde{\lambda}^{\prime}=0$
in (26) and get an 8-dimensional metric. In the following, we show that this
manifold admits a harmonic 4-form which is then used to derive a supergravity
solution in 11 dimensions.

###  3.1 The ansatz for $F_{4}$

To discuss our ansatz for $F_{4}$ and the solutions, henceforth, we take the
following eight-dimensional metric

$\displaystyle ds^{2}_{8}$ $\displaystyle=$ $\displaystyle
d\mu^{2}+\frac{1}{4}\sin^{2}\mu\,\Sigma_{i}^{2}+\lambda^{2}(d\theta_{1}-\sin\phi_{1}A_{1}+\cos\phi_{1}A_{2})^{2}$
(30) $\displaystyle+$
$\displaystyle\lambda^{2}\sin^{2}\theta_{1}\,(d\phi_{1}-\cot\theta_{1}(\cos\phi_{1}A_{1}+\sin\phi_{1}A_{2})+A_{3})^{2}$
$\displaystyle+$
$\displaystyle\tilde{\lambda}^{2}(d\theta_{2}-\sin\phi_{2}\tilde{A}_{1}+\cos\phi_{2}\tilde{A}_{2})^{2}$
$\displaystyle+$
$\displaystyle\tilde{\lambda}^{2}\sin^{2}\theta_{2}\,(d\phi_{2}-\cot\theta_{2}(\cos\phi_{2}\tilde{A}_{1}+\sin\phi_{2}\tilde{A}_{2})+\tilde{A}_{3})^{2}\,,$

and assume that the eleven-dimensional manifold is given by $AdS_{3}\times
M_{8}$, with the metric

$ds_{11}^{2}=ds_{3}^{2}+ds_{8}^{2}\,.$ (31)

As in [14], it also proves useful to define the following self-dual 2-forms

$\displaystyle R_{1}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\sin\phi_{1}(e^{01}+e^{23})-\cos\phi_{1}(e^{02}+e^{31})\,,$
$\displaystyle R_{2}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\cos\theta_{1}\cos\phi_{1}(e^{01}+e^{23})+\cos\theta_{1}\sin\phi_{1}(e^{02}+e^{31})-\sin\theta_{1}(e^{03}+e^{12})\,,$
$\displaystyle K\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\sin\theta_{1}\cos\phi_{1}(e^{01}+e^{23})+\sin\theta_{1}\sin\phi_{1}(e^{02}+e^{31})+\cos\theta_{1}(e^{03}+e^{12})\,,$
(32)

where we have used the shorthand notation $e^{ij}=e^{i}{\wedge}e^{j}$. The key
feature of this definition, that we will use frequently in this paper, is that
these three forms are orthogonal to each other, i.e.,

$R_{1}{\wedge}R_{2}=K{\wedge}R_{1}=K{\wedge}R_{2}=0\,.$ (33)

We can also define a set of anti self-dual 2-forms which are orthogonal (with
respect to the wedge product) to the above self-dual set:

$\displaystyle{\tilde{R}}_{1}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\sin\phi_{2}(-e^{01}+e^{23})-\cos\phi_{2}(-e^{02}+e^{31})\,,$
$\displaystyle{\tilde{R}}_{2}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\cos\theta_{2}\cos\phi_{2}(-e^{01}+e^{23})+\cos\theta_{2}\sin\phi_{2}(-e^{02}+e^{31})-\sin\theta_{2}(-e^{03}+e^{12})\,,$
$\displaystyle{\tilde{K}}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\sin\theta_{2}\cos\phi_{2}(-e^{01}+e^{23})+\sin\theta_{2}\sin\phi_{2}(-e^{02}+e^{31})+\cos\theta_{2}(-e^{03}+e^{12})$
(34)

and obey relations like the ones in (33). $K$ and ${\tilde{K}}$ naturally
appear in the field strength of $A$ and $\tilde{A}$, respectively;

$\displaystyle F\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!dA=-K-e^{56}/{\lambda^{2}}\,,$
$\displaystyle{\tilde{F}}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!d\tilde{A}=-{\tilde{K}}-\tilde{e}^{56}/{\tilde{\lambda}^{2}}\,.$
(35)

Taking the exterior derivative, it is easy to see that

$\displaystyle dK\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!-\frac{1}{\lambda^{2}}de^{56}=-\frac{1}{\lambda}\,{\rm
Im}\,\Omega\,,$ $\displaystyle d{\tilde{K}}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!-\frac{1}{\tilde{\lambda}^{2}}d\tilde{e}^{56}=-\frac{1}{\tilde{\lambda}}\,{{\rm
Im}\,{\tilde{\Omega}}}\,,$ (36)

where,

$\displaystyle{\rm Im}\,\Omega=R_{1}{\wedge}e^{6}-R_{2}{\wedge}e^{5}\,,$
$\displaystyle{{\rm
Im}\,{\tilde{\Omega}}}={\tilde{R}}_{1}{\wedge}\tilde{e}^{6}-{\tilde{R}}_{2}{\wedge}\tilde{e}^{5}\,.$
(37)

Here, $\Omega$ is the holomorphic 3-form of the ${\bf CP}^{3}$ which appears
in the metric of $S^{7}$ viewed as a $U(1)$ bundle over ${\bf CP}^{3}$, and it
should not be confused with $\Omega_{3}$ of the previous section, and
$\Omega_{4}$ that will appear in the next section. From (36), it trivially
follows that $d{\rm Im}\,\Omega=d{{\rm Im}\,{\tilde{\Omega}}}=0$.

Now, let us examine if there is any harmonic 4-form on the 8d manifold endowed
with metric (30). A closed 4-form which is easy to guess is

$\Omega_{4}=\alpha\omega_{4}+\beta
K{\wedge}e^{56}+\gamma{\tilde{K}}{\wedge}\tilde{e}^{56}\,,$ (38)

for constant coefficients $\alpha,\beta,\gamma$. The first term is closed,
$d\omega_{4}=0$. Using (36) together with ${\rm Im}\,\Omega{\wedge}e^{56}={\rm
Im}\,\Omega{\wedge}K=0$, and ${{\rm
Im}\,{\tilde{\Omega}}}{\wedge}\tilde{e}^{56}={{\rm
Im}\,{\tilde{\Omega}}}{\wedge}{\tilde{K}}=0$, we see that the second and the
third terms are also separately closed, so $d\Omega_{4}=0$. For the Hodge dual
we have

$*_{8}\Omega_{4}=\alpha e^{56}{\wedge}\tilde{e}^{56}+\beta
K{\wedge}\tilde{e}^{56}-\gamma{\tilde{K}}{\wedge}e^{56}\,,$ (39)

the minus sign is because ${\tilde{K}}$ is anti self-dual. Demanding
$d*_{8}\Omega_{4}=0$, and noticing that

${{\rm Im}\,{\tilde{\Omega}}}{\wedge}K=0\,,\ \ \ \ {\rm
Im}\,\Omega{\wedge}{\tilde{K}}=0\,,$ (40)

as untilted and tilted 2-forms are self-dual and anti self-dual, respectively,
the $\beta$ and $\gamma$ get fixed in terms of an overall constant $\alpha$.
So, here is the harmonic 4-form;

$\Omega_{4}=\omega_{4}+\lambda^{2}K{\wedge}e^{56}-\tilde{\lambda}^{2}{\tilde{K}}{\wedge}\tilde{e}^{56}\,.$
(41)

Either $\Omega_{4}$ or $*_{8}\Omega_{4}$ can be used as an ansatz for the
4-form field strength in its 11d supergravity equation of motion

$d*_{11}F_{4}=-\frac{1}{2}\,F_{4}{\wedge}\,F_{4}\,.$ (42)

Let us take

$F_{4}=\alpha*_{8}\Omega_{4}=\alpha(e^{56}{\wedge}\tilde{e}^{56}+\lambda^{2}K{\wedge}\tilde{e}^{56}+\tilde{\lambda}^{2}{\tilde{K}}{\wedge}e^{56})\,,$
(43)

since $*\Omega_{4}{\wedge}*\Omega_{4}=0$, the right hand side of (42) becomes
zero. Therefore, since $*\Omega_{4}$ is harmonic, $F_{4}$ satisfies both the
Bianchi identity and the equation of motion, (42). One might think that
similar ansatz may also work for simpler 8d manifolds like $S^{4}\times S^{4}$
or ${\bf CP}^{4}$, which admit 4-form harmonics too. The point, however, is
that for such ansatz the right hand side of Maxwell equation does not vanish.
So, here, it is important that we have $*\Omega_{4}{\wedge}*\Omega_{4}=0$.
Having found a solution for the Maxwell equation, (42), let us now turn to the
eleven-dimensional Einstein equations;

$R_{MN}=\frac{1}{12}\,F_{MPQR}F_{N}^{\,\ PQR}-\frac{1}{3\cdot 48}\,g_{MN}\
F_{PQRS}F^{PQRS}\,,$ (44)

where $M,N,P,\ldots=0,1,\ldots,10$. Setting for simplicity
$\lambda=\tilde{\lambda}$, the Ricci tensor components of $M_{8}$ on the left
hand side can be read off from (29) (with
$\lambda^{\prime}=\tilde{\lambda}^{\prime}=0$). Further, with ansatz (43), we
can calculate the right hand side of the above equations to get;

$\displaystyle
3-2\lambda^{2}=\alpha^{2}\left(\frac{\lambda^{4}}{3}-\frac{1}{6}\right)\,,$
$\displaystyle\frac{1+\lambda^{4}}{\lambda^{2}}=\alpha^{2}\left(\frac{\lambda^{4}}{3}+\frac{1}{3}\right)\,.$
(45)

The solution reads

$\lambda^{2}=\frac{1}{6}(\sqrt{15}+3)\,,\ \ \ \ \alpha^{2}=3(\sqrt{15}-3)\,.$
(46)

As $\lambda^{2}>1$, the $S^{2}\times S^{2}$ fibers in (30) are stretched with
respect to the base. For the Ricci tensor along $AdS_{3}$, we obtain

$R_{\mu\nu}=-\frac{1}{6}\left(5\sqrt{15}-3\right)g_{\mu\nu}\,,$ (47)

with $\mu,\nu=0,1,2$.

If we take $F_{4}=\alpha\Omega_{4}$, it is clear that Maxwell equation (42) is
satisfied. However, for this ansatz we have found no real solution of the
Einstein equations.

##  4 10d IIA supergravity solution

The ansatz we found in the previous section can also be used to obtain a ten-
dimensional type IIA supergravity solution. For type IIA field equations in
the string frame we have

$\displaystyle d{\tilde{F}}_{4}=-F_{2}\wedge H\,,\ \ \ \
d*{\tilde{F}}_{4}=-{\tilde{F}}_{4}\wedge H\,,\ \ \ \ dH=0\,,\ \ \ \
dF_{2}=0\,,$ $\displaystyle
d*(e^{-2\phi}H)=-F_{2}\wedge*{\tilde{F}}_{4}+\frac{1}{2}{\tilde{F}}_{4}\wedge{\tilde{F}}_{4}\,,\
\ \ \ d*F_{2}=H\wedge*{\tilde{F}}_{4}\,,$ $\displaystyle
d*d\phi-d\phi\wedge*d\phi-\frac{1}{8}H\wedge*H+\frac{1}{4\cdot
3!}R\,\epsilon_{4}{\wedge}J^{3}=0\,,$
$\displaystyle{\tilde{F}}_{4}=F_{4}-A_{1}\wedge H\,.$ (48)

If we only turn on $F_{4}=\alpha*\Omega_{4}$ as in the previous section, since
$F_{4}{\wedge}F_{4}=0$ we can set $H=F_{2}=0$. But then $R$, the scalar
curvature, will not be zero and hence, acts via the last equation as a source
for the dilaton, $\phi$. However, if we further turn on
$F_{2}=\eta\epsilon_{2}$, with $\epsilon_{2}$ the volume element of an
$AdS_{2}$ factor, we can satisfy $R=0$, and have a constant dilaton. Our
ansatz for the metric thus will be

$ds_{10}^{2}=ds_{AdS_{2}}^{2}+ds_{8}^{2}\,,$ (49)

together with

$F_{4}=\alpha*\Omega_{4}\,,\ \ \ \ F_{2}=\eta\epsilon_{2}\,,\ \ \ \ H=0\,,\ \
\ \ \phi=const.$ (50)

We can check it easily that with this ansatz all the equations of motion,
(48), are satisfied. For the Einstein equations we have,

$\displaystyle R_{MN}$ $\displaystyle=$
$\displaystyle\frac{e^{2\phi}}{12}\,\left(F_{MPQR}F_{N}^{\,\
PQR}-\frac{3}{32}\,g_{MN}\ F_{PQRS}F^{PQRS}\right)\,$ (51) $\displaystyle+$
$\displaystyle\frac{e^{2\phi}}{2}\,\left(F_{MP}F_{N}^{\,\
P}-\frac{1}{4}\,g_{MN}\ F_{PQ}F^{PQ}\right)\,,$

so requiring $R=0$, fixes the relative scale of $F_{4}$ and $F_{2}$;

$\eta^{2}=\frac{\alpha^{2}}{3}(1+4\lambda^{4})\,.$ (52)

Using (29) and ansatz (50), the Einstein equations along $M_{8}$ become

$\displaystyle
3-2\lambda^{2}=e^{2\phi}\alpha^{2}\left(\frac{\lambda^{4}}{3}-\frac{1}{6}\right)\,,$
$\displaystyle\frac{1+\lambda^{4}}{\lambda^{2}}=e^{2\phi}\alpha^{2}\left(\frac{\lambda^{4}}{3}+\frac{1}{3}\right)\,,$
(53)

which are the same equations as in (45), with the solution

$\lambda^{2}=\frac{1}{6}(\sqrt{15}+3)\,,\ \ \ \
e^{2\phi}\alpha^{2}=3(\sqrt{15}-3)\,.$ (54)

For the Ricci tensor along $AdS_{2}$ we obtain

$R_{\mu\nu}=-\frac{1}{3}\left(5\sqrt{15}-3\right)\,g_{\mu\nu}\,,$ (55)

with $\mu,\nu=0,1$.

The above type IIA solution can be understood as a reduction of the
$AdS_{3}\times M_{8}$ solution of the previous section. In fact, $AdS_{3}$
metric can be written as a $U(1)$ bundle over $AdS_{2}$. If we dimensionally
reduces along this $U(1)$ direction, then, in ten dimensions, $F_{2}$ appears
as the field strength of the corresponding $U(1)$ connection.

##  5 Conclusions

In this paper, we studied adding an extra structure of anti-instanton fibers
to the metric of $S^{7}$, and constructed a 10d metric, which had constant
Ricci tensor components in the corresponding orthonormal basis. The metric
also admitted a harmonic 3-form. To get a supergravity solution, however, we
argued that the metric could be rewritten as a $U(1)\times U(1)$ bundle over
an 8d manifold. In this way, it was possible to scale away the $U(1)$
directions and get a new 8d manifold with metric (30). We showed that this
manifold admits a harmonic 4-form, which we then used as an ansatz for the
4-form field strength in eleven-dimensional and ten-dimensional type IIA
supergravities. At the end, we saw that, through this choice of the metric and
the ansatz for $F_{4}$, the equations of motion could be reduced to a set of
simple algebraic equations. The solutions, in turn, fixed the free parameter
$\lambda$, the scale factor of fibers in the metric, and the overall scale of
$F_{4}$ (and $F_{2}$ in the case of type IIA).

## References

  * [1] P. G. O. Freund and M. A. Rubin, Dynamics Of Dimensional Reduction, Phys. Lett. B 97, 233 (1980).
  * [2] F. Englert, Spontaneous Compactification Of Eleven-Dimensional Supergravity, Phys. Lett. B 119, 339 (1982).
  * [3] L. Castellani and L. J. Romans, N=3 And N=1 Supersymmetry In A New Class Of Solutions For D = 11 Supergravity, Nucl. Phys. B 238, 683 (1984).
  * [4] F. Englert, M. Rooman and P. Spindel, Supersymmetry Breaking By Torsion And The Ricci Flat Squashed Seven Spheres, Phys. Lett. B 127, 47 (1983).
  * [5] D. V. Volkov, D. P. Sorokin and V. I. Tkach, Supersymmetry Vacuum Configurations in D = 11 Supergravity, JETP Lett. 40, 1162 (1984).
  * [6] E. Witten, Anti-de Sitter space, thermal phase transition, and confinement in gauge theories, Adv. Theor. Math. Phys. 2 (1998), 505-532, [arXiv:hep-th/9803131 [hep-th]].
  * [7] M. A. Awada, M. J. Duff and C. N. Pope, N = 8 supergravity breaks down to N = 1, Phys. Rev. Lett. 50, 294 (1983).
  * [8] M. J. Duff, B. E. W. Nilsson and C. N. Pope, Spontaneous Supersymmetry Breaking By The Squashed Seven Sphere, Phys. Rev. Lett. 50, 2043 (1983).
  * [9] F. A. Bais, H. Nicolai and P. van Nieuwenhuizen, Geometry Of Coset Spaces And Massless Modes Of The Squashed Seven Sphere In Supergravity, Nucl. Phys. B 228, 333 (1983).
  * [10] M. J. Duff, Supergravity, The Seven Sphere, And Spontaneous Symmetry Breaking, Nucl. Phys. B 219, 389 (1983).
  * [11] C. N. Pope and P. van Nieuwenhuizen, Compactifications of $d=11$ Supergravity on Kähler Manifolds, Commun. Math. Phys. 122, 281 (1989).
  * [12] A. Imaanpur, Type IIB flux compactifications on twistor bundles, Phys. Lett. B 729 (2014), 45-49, [arXiv:1309.5773 [hep-th]].
  * [13] M. J. Duff, B. E. W. Nilsson and C. N. Pope, Kaluza-Klein Supergravity, Phys. Rept. 130, 1 (1986).
  * [14] A. Imaanpur, A New Solution of Eleven-dimensional Supergravity, Class. Quant. Grav. 30, 065021 (2013), [arXiv:1205.1349 [hep-th]].
  * [15] B. E. W. Nilsson and C. N. Pope, Hopf Fibration Of Eleven-Dimensional Supergravity, Class. Quant. Grav. 1, 499 (1984).
  * [16] G. Aldazabal and A. Font, A Second look at N=1 supersymmetric AdS(4) vacua of type IIA supergravity, JHEP 0802, 086 (2008), [arXiv:0712.1021].


# Evaluation of HTR models without Ground Truth Material

###### Abstract

The evaluation of Handwritten Text Recognition (HTR) models during their
development is straightforward: because HTR is a supervised problem, the usual
data split into training, validation, and test data sets allows the evaluation
of models in terms of accuracy or error rates. However, the evaluation process
becomes tricky as soon as we switch from development to application. A
compilation of a new (and forcibly smaller) ground truth (GT) from a sample of
the data that we want to apply the model on and the subsequent evaluation of
models thereon only provides hints about the quality of the recognised text,
as do confidence scores (if available) the models return. Moreover, if we have
several models at hand, we face a model selection problem since we want to
obtain the best possible result during the application phase. This calls for
GT-free metrics to select the best model, which is why we (re-)introduce and
compare different metrics, from simple, lexicon-based to more elaborate ones
using standard language models and masked language models (MLM). We show that
MLM-based evaluation can compete with lexicon-based methods, with the
advantage that large and multilingual transformers are readily available, thus
making compiling lexical resources for other metrics superfluous.

Keywords: handwritten text recognition, digital humanities, evaluation, ground
truth data, resources, model selection

languageresourceLanguage Resources

Evaluation of HTR models without Ground Truth Material

Phillip Benjamin Ströbel1, Simon Clematide1, Martin Volk1, Raphael Schwitter2,
---
Tobias Hodel3, David Schoch3
1Department of Computational Linguistics, 2Department of Classics, 3Walter
Benjamin Kolleg
1,2University of Zurich, 3University of Bern
{pstroebel, siclemat<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
{tobias.hodel<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Optical Character Recognition (OCR) has become a well-established technique
for digitising historical printed collections in libraries and archives. At
the same time, Handwritten Text Recognition (HTR) is also increasingly finding
its way into these institutions [Terras (in press]. The digitisation efforts
of libraries and archives simplify access to sought-after documents for
researchers from various disciplines and allow them to pursue research
questions they could either not or only very cumbersomely have answered
without digital copies of the sources.

Still, we can only consider OCR and HTR to be “research facilitators” as long
as they perform within reasonable accuracy ranges. Several studies have shown
that inaccuracies in OCRed documents harm information retrieval and text
mining techniques like named entity recognition and linking, topic modelling,
and language modelling [Alex and Burns (2014, Chiron et al. (2017, van Strien
et al. (2020, Hamdi et al. (2020, Pontes et al. (2019, Hill and Hengchen
(2019].

But what do we mean by “reasonable accuracy ranges”? ?) shared the experiences
of the National Library of Australia Newspaper Digitisation Program, which
found that OCR contractors and libraries usually agree that “good” OCR means
$>98\%$, “average” OCR between 90% and 98%, and “poor” OCR any score $<90\%$
character accuracy, respectively. ?) defined ranges of $>95\%$ and between 90%
and 95% character accuracy, but one time referring to the former as “good” and
later as “excellent”, and to the latter as “reasonable” and in another place
as “good”, respectively. Although opinions of what good OCR is differs, there
is consensus that any score $<90\%$ means poor quality111In this paper, we
mainly look at character error rates, which are simply the inverse of
character accuracy. E.g., a 98% character accuracy corresponds to a 2%
character error rate..

While ?) and ?) were evaluating OCR for printed texts, these accuracy ranges
naturally extend to text recognised from handwritten documents. However, HTR
for personal handwriting styles is particularly challenging since a personal
hand has much more variance than printed fonts or regular scripts like
blackletter. Nonetheless, HTR has seen considerable improvements over the last
decade, mainly thanks to the advent of neural networks [Graves and Schmidhuber
(2008, Graves et al. (2009]. These developments fostered the creation of
platforms like Transkribus222https://readcoop.eu/de/transkribus/, which
facilitate the production of ground truth (GT), i.e., humanly transcribed
material, and large-scale training of HTR models. Given suitable training
data, the Transkribus-internal HTR+ model [Michael et al. (2018] can achieve
character error rates (CERs) of about 5% [Mühlberger et al. (2019], therefore
coming close to what ?) considered as “good” and ?) already as “excellent”.

Although HTR has made considerable progress, it has not yet found its way to a
broader application in libraries and archives. A look at the digital platform
for manuscript material for Swiss libraries and archives
e-manuscripta333https://www.e-manuscripta.ch/ shows that in the category
“correspondence” containing $\sim$45k titles, only 313, or $0.1\%$, contain
transcriptions. With the output quality of modern HTR, libraries and archives
can further process the scans that they accumulated over the years of mass
digitisation and thus provide researchers and the public access to the content
of manuscripts of various sorts.

However, the deployment of HTR models to produce this content comes with
uncertainties. While we can evaluate the performance of HTR models during the
development phase via accuracy or error scores on validation and test sets, we
cannot do so during the application phase. Hence, there are only a few options
to estimate the quality of the resulting texts:

  1. 1.

We compile a new GT from a representative sample of the new data.

  2. 2.

We rely on confidence scores.

  3. 3.

We find GT-free evaluation metrics.

As concerns the first option, compiling a GT is costly, time-consuming, and
requires human experts. Moreover, we suspect that we would only obtain a rough
estimate of the overall quality [Hodel et al. (2021]. As for the second
option, confidence scores are not always available, and if they are, it is
unclear whether and to what extent they correlate with actual accuracy scores
across different documents and distinct models [Springmann et al. (2016]. This
leaves the third option as the only remaining one, which we will address in
this paper. We propose different metrics in Section 3.1 and investigate
whether there is a significant correlation between the introduced metrics and
the actual CER.

The scrutiny of metrics for performance evaluation has practical implications
for libraries and archives: on the one hand, they can provide an estimate of
the text quality, which is relevant for researchers working with this data. On
the other hand, GT-free metrics allow for model selection when several models
have produced hypotheses. Factors like the number of authors in a particular
collection, the size of the GT, the skew of the author distribution, scan
quality, and the like justify the training and testing of multiple models with
different parameters and training data. In order to produce the best possible
text during the application phase, it is necessary to apply all models and
decide which model produced the best result for each document. In such a case,
GT-free evaluation metrics can assist libraries and archives in making this
choice. Hence, we also test whether GT-free evaluation metrics are suitable
for model selection.

To summarise, in this paper, we investigate a collection of GT-free evaluation
metrics to provide a quality estimation of HTR model results. Moreover, we
examine whether we can use the metrics to select the best performing models.

## 2\. Related Work

Studies examining the evaluation of HTR output are few, which might be because
baselines on popular data sets like the IAM [Marti and Bunke (2002] database
for handwritten English sentences are indicated in standard measures (CERs or
word error rates) [Sánchez et al. (2019]. ?) pointed out that although the
evaluation of OCRed output is standardised by measures like CERs, different
implementations in tools that assist the evaluation make it difficult to
compare model performances. Additionally, ?) stated that such standard
measures to rank models “may be of limited significance for the decision to
choose a recognition system for a real-life application.” His work showed that
for the recognition of postal addresses, the final performance metric and,
e.g., error limitation, i.e., the rejection of uncertain results, for
different model architectures do not necessarily correlate.

Whenever confidence scores are available, we can exploit them to estimate the
text quality. For example, ?) used confidence score outputs by an OCR model to
build a triage system for OCRed documents. They classified documents to bypass
manual inspection, which they successfully achieved for 41% of all documents
in their validation set. Overall, they could show that their triage method
significantly sped up the document verification process.

?) suggested an OCR quality prediction system that assesses the accuracy of
OCR results by any given system. Their classifier relies on 28 features,
including metadata, image and layout information, textual features, and
confidence scores. Their model was trained on newspaper data and provided
reliable OCR quality estimates with an average error rate of 6.1% on a bag-of-
words basis, indicating suitability for OCR quality estimation. In addition, a
more detailed feature analysis of their classifier showed that the confidence
scores are the third most important feature, only topped by lexical features.

Work by ?) on the automatic OCR quality estimation of historical printings
also showed that the relationship between accuracy and confidence scores is
significant (in contrast to ?)).444We believe that the extent of how much we
can trust a model’s confidence scores strongly depends on the model itself.
Their approach investigated two measures: a mean token lexicality score based
on the text-channel-model by ?) and confidence values that the OCR model
assigns to its output characters. Moreover, they further improved their
“standard” models that they had already trained by continuous training with
different amounts of “pseudo ground truths” consisting of text output where
standard models return high confidence values. Lastly, ?) used the confidence
values in order to rank the models during training, which enabled them to
choose the best one according to the confidence values after all training runs
had been completed.

?) illustrated the usefulness of lexicality in OCR outputs and advocated for a
much simpler evaluation procedure. They examined the influence of OCR errors
on named entity recognition in historical documents and found that especially
recall is affected. They suggested measuring the quality of a text by simply
computing the ratio of all words in a document and words occurring in an
English dictionary. The test of their metric against human evaluation showed a
correlation between their score and human judgements.

?) developed an approach that operates without external language resources.
Their method relied on two OCR outputs: an OCR result and a second OCR output
by another system called the reference (i.e., an uncertain GT). They used
word-based character agreement rates and mutual information scores as features
for a Support Vector Regressor that predicts the word recognition rate. They
found their approach reliable, with an R² score of 0.95.

We can also use error classifiers in OCRed texts as quality estimators. ?)
proposed an error detector that uses several word- and character-n-gram
measures as features to train an OCR error detector. They also introduced a
so-called peculiarity index which they defined as the root-mean-square of the
indices of its trigrams. The higher the peculiarity index of a word, the more
unusual its trigram composition is and, consequently, the more unlikely it is
to be correct. Their error detector works best with all features included,
reaching an F-score of 79% on monographs and 70% on periodicals.

## 3\. Method

Our goal is to evaluate different GT-free metrics and to check for each of
them if it is suitable to rank HTR models. In the first step, we train many
models on data sets we specify in more detail in Section 4. Next, we apply all
models to different test sets and compute all metrics introduced in this
Section. This procedure results in a ranking for each metric. At the same
time, we can rank all the models we have trained according to the CER they
achieve on the test set, which we use as our reference ranking. Ideally, a
metric’s ranking would be the same as the ranking according to the CER. We use
Spearman’s rank correlation measure to learn whether the reference ranking
correlates with the metrics’ rankings. Its coefficients tell us at what
significance level each two ranking pairs correlate. Fig. 1 summarises the
pipeline, which we will describe in more detail in this and the forthcoming
sections.

Figure 1: Training and evaluation procedure.

### 3.1. Metrics

The code and language resources to run the metrics will be made available with
the publication of the conference
proceedings.555https://github.com/pstroe/atr-eval

#### 3.1.1. Lexicon-based Metrics

?) showed that lexical features are most suitable to predict OCR quality,
while ?) indicated that a simple ratio of recognised words against a lexicon
correlates well with human judgement about OCR quality. We thus include
lexicon-based methods in our metrics.

##### Token Ratio

The token ratio determines the percentage of tokens recognised by the HTR
model that also occur in a reference. Let $T$ be the tokens in the HTR result
and $\mathcal{V}$ the corresponding vocabulary. The token ratio is given by

$\mathcal{R}_{\mathrm{token}}=\frac{c(T\in\mathcal{V})}{c(T)}.$ (1)

This measure performs a lexicon lookup, where the lexicon is based on a
collection of texts in the same language.

##### Character N-gram Ratio

?) included an n-gram analysis in the OCR error detection process. We also
include character n-gram ratios against a lexicon, but without the peculiarity
index that ?) introduced.

We consider the ratios of $2$\- to $7$-grams. For each n-gram variant, we
build the corresponding n-gram vocabulary from the reference text and the HTR
results. Be $\mathcal{G}$ the set of n-grams in the reference text and $N$ the
n-grams in the HTR result. The n-gram ratio is given by

$\mathcal{R}_{n\mathrm{-gram}}=\frac{c(N\in\mathcal{G})}{c(N)}.$ (2)

#### 3.1.2. Language Modeling Perplexity

RNN-based HTR architectures learn an implicit character language model (LM)
[Sabir et al. (2017]. Other approaches pre-train a LM and incorporate it in an
HTR model to boost performance [Kang et al. (2021]. LMs thus form an integral
part of the HTR process, so we suggest using external ones to evaluate HTR
results. This is the first approach that uses LMs for HTR quality estimation
to the best of our knowledge. We differentiate between statistical LMs and
transformer-based LMs.

##### Statistical Language Modeling Perplexity

The task of LMs is to predict a word given a sequence of preceding words.
Probabilistic LMs learn from massive text corpora which words must follow each
other. A statistical LM is given by

$P_{MLE}(w_{n}|w_{1}...w_{n-1})=\frac{c(w_{1}...w_{n})}{c(w_{1}...w_{n-1})}$
(3)

where MLE refers to Maximum Likelihood Estimation and n to the order of
n-grams. For example, a bi-gram LM would predict the next word based on the
preceding word.

We can evaluate LMs with the perplexity (PPL) score, a measure from
information theory. For example, given a LM $P$ and test set $W$ consisting of
words in their sentential contexts, the word-level PPL measures the surprisal
of the model when computing the likelihood of the sequence of words:

$PPL(W)=2^{H(W)}$ (4)

where $H$ is the cross-entropy of the sequence $W$:

$H(W)=-\frac{1}{N}\sum_{i=1}^{N}\mathrm{log}_{2}P(w_{i})$ (5)

The lower the PPL score, the less surprised the model is, and hence, the
better the text quality should be for HTR results.

##### Pseudo-Perplexity

With the advent of deep learning, language modelling has considerably
increased accuracy, not least thanks to transformer-based [Vaswani et al.
(2017] methods like, e.g., BERT [Devlin et al. (2018]. BERT is an approach
that uses the transformer blocks and attention heads and learns two tasks: (1)
the prediction of tokens that are masked in the input and (2) the prediction
of whether the next sentence is a natural continuation of the input sentence.
As such, the authors designed BERT as a language representation model suitable
to be fine-tuned on many tasks like natural language understanding, question
answering, or grounded common sense inference. There is a large amount of
research striving to improve and modify masked language models (MLMs) like
BERT. One modification we will use is RoBERTa [Liu et al. (2019], which used
longer training over more and larger batches and more data, the omission of
the next sentence prediction, and other tweaks to make BERT more robust.

Whilst PPL is a suitable metric for traditional LMs, the MLM objective does
not permit a direct calculation of PPL. Instead, ?) proposed the use of
pseudo-perplexities (PPPLs). The PPPL is the exponential of the word-
normalised sums of each token’s conditional log probability in all output
sentences to evaluate. More concretely, given an input sentence, we mask each
token once, add up the conditional log probabilities of all tokens and
normalise by the number of words in the sentence. We use the MLM
scorer666https://github.com/awslabs/mlm-scoring framework proposed by the
authors for our experiments.

## 4\. Data and Models

### 4.1. HTR Data

In contrast to the evaluation methods in Section 2, we work with historical
handwritten data, more concretely, with samples from Heinrich Bullinger’s
(1504-1575) extensive correspondence. The Zurich State Archives and the Zurich
Central Library have preserved some 2,000 letters that Bullinger wrote and
10,000 letters that he received. 80% of the letters are Latin, most of the
others in Early New High German. About 3,000 letters had already been manually
transcribed, edited, and published in printed form in efforts that spanned
three decades.777They are available on http://teoirgsed.uzh.ch/. Another 5,700
letters have been transcribed by various scholars and are available as
electronic texts, albeit in uncertain quality. In our ongoing project
Bullinger digital888https://www.bullinger-digital.ch/, we make the scans and
transcriptions accessible for researchers and the public. Moreover, we will
produce automatic transcriptions for the roughly 3,000 letters for which no
transcriptions exist and thus will face model selection and HTR quality
estimation issues.

Bullinger received many more letters than he wrote, so the author distribution
is heavily skewed. Fig. 2 exemplifies this fact; over 350 contemporaries only
ever wrote once to Bullinger. This skew presents a considerable hurdle for HTR
models since the presence of many different hands makes it more difficult to
generalise. Fig. 3 shows examples of how the handwritings differ among
authors. Especially in such a setting, it can be necessary to train multiple
models with different author distributions and choose the model that performs
best.

Figure 2: Author distribution of the Bullinger correspondence. Number of
letters on the x-axis.

Our training data for these models comes from two sources. Firstly, we found
that a correspondent of Bullinger’s, Rudolf Gwalther, wrote a volume called
Lateinische Gedichte, whose scan images and partial transcriptions are
available on
e-manuscripta999https://www.e-manuscripta.ch/zuz/content/titleinfo/1111284. We
loaded the images into the Transkribus interface, applied layout recognition
to identify lines, and aligned the transcriptions with the lines. We made
minor corrections during this process, mainly concerning inconsistent
capitalisation and punctuation. The resulting “Gwalther” data set consists of
$\sim$4,000 lines and is one of our training sets (see box 1a on the right in
Fig. 1).101010This set is publicly available on
https://doi.org/10.5281/zenodo.4780947.

Secondly, we aligned the scan images that our suppliers have already delivered
with the already published transcriptions by applying Transkribus’ Text2Image
module, which resulted in an automatically aligned GT of roughly 20,000 lines
that we call “Misc. Bullinger correspondence” (see box 1a on the left in Fig.
1). An analysis of the author distribution shows a heavily skewed picture with
69 different authors, where about 4,600 lines stem from Oswald Myconius and
only two, e.g., from Georg Cassander.

We also compiled two separate test sets. On the one hand, we sampled 825 lines
from a portion of scans without transcriptions (box 1b in Fig. 1) and manually
transcribed the lines. Again, the author distribution is heavily skewed.
Moreover, 35% of the 825 lines are by authors, which we also find in the
“Misc. Bullinger correspondence”. On the other hand, we include a separate 57
lines letter from Gwalther that is not part of any training or test data. We
observed that Gwalther’s writing style differs in the poetry volume and the
letters, as Fig. 3 shows. Models trained on “Gwalther” data should
nevertheless perform better on this test set than other models.

Figure 3: Different handwriting styles. a) poetry by Rudolf Gwalther, letters
by b) Rudolf Gwalther, c) Matthieu Coignet, and d) Kaspar Wolf.

### 4.2. Language Resources

Since we worked with Latin correspondence, we collected Latin reference
material (i.e., we did not use texts in Early New High German). For the
experiments in Section 5, we downloaded the Latin part of the CC-100
corpus111111http://data.statmt.org/cc-100/, accessed: 29.09.2021. [Conneau et
al. (2019, Wenzek et al. (2019]. The raw file contains 2.5G of text, i.e., 390
million tokens. Next, we removed generic ”Lorem ipsum” text, deduplicated the
corpus and retained only lines consisting of characters in the Latin alphabet,
numbers, and a selection of punctuation marks. Finally, we tokenised and
normalised the corpus with the help of the Classical Language Toolkit CLTK
[Johnson et al. (2021] and split it into sentences. After preprocessing
$\sim$206 million tokens remain.

### 4.3. Models

#### 4.3.1. HTR Models

| Number of lines for training
---|---
| 30 | 90 | 150 | 300 | 600 | 900 | 1200 | 1500 | 1800 | 2100 | 2400 | 2700 | 3000 | 3300 | 3600
No base | 39.33 | 14.37 | 11.39 | 7.28 | 5.55 | 4.96 | 4.36 | 4.14 | 3.9 | 3.94 | 3.64 | 3.59 | 3.36 | 3.24 | 3.29
Acta-based | 13.67 | 8.38 | 7.08 | 5.5 | 4.42 | 4.03 | 3.66 | 3.43 | 3.27 | 3.13 | 3.09 | 3.2 | 2.82 | 2.8 | 2.74
| Other models (evaluated on the “Gwalther” validation set)
Bullinger | 6.99
Bullinger+Acta | 6.56
Acta_17 | 14.66
Spruchakten | 15.95

Table 1: CERs for all models trained on different training data and number of
lines on their respective test sets.

We trained all our HTR models with Transkribus and the HTR+ model architecture
for 50 epochs. For both training data sets in box 1a in Fig. 1, we set aside
10% of the corpus for validation and 90% for training. Transkribus offers the
possibility to include base models in the training process. We selected the
publicly available model “Acta_17 HTR+”.121212The data (chancery writings) and
model were compiled by Alvermann et al. from the University of Greifswald.
Period: 1580-1705, size: $\sim$600k words in Latin and Low German from 1k
different authors, epochs: 1k. A base model initialises the model’s weights
and allows for fine-tuning the model on novel data. This way, the model knows
something about handwriting before seeing the new training data, leading to
faster convergence and better performance. We trained a variant with and
without a base model for every data set.

For the “Gwalther” data set, we additionally conducted a size ablation,
starting at 300 lines (corresponding to roughly ten pages) up to the usage of
the entire training set in increments of 300 lines, following a similar
approach as detailed in ?). Moreover, we simulated an extreme low-resource
setting, in which we sampled 30, 90, and 150 lines five times and used those
as training data.

We sketched the training procedure in box 2 in Fig. 1. This training regime
results in a total of 56 models. We only considered the best performing model
per training set size for the low-resource simulation setting, thus discarding
24 models. We also evaluated the base model Acta_17 HTR+ and the
Spruchakten_M_211 model on the “Gwalther” validation set and the test
sets.131313The model was trained on a subset of the Acta_17 training set:
period: 1583-1653, size: 246k words, epochs: 1k. The reason for including this
model is to check whether the effects of applying a model on paleographically
more related data (since it’s closer to Bullinger’s time) are more pronounced
than just corpus size.

We present the results of all the models in Table 1. We note a considerable
improvement of performance the more data we add, while models trained with a
base model consistently perform better than their counterparts without a base
model. The Spruchakten model, although trained on data closer to Bullinger’s
time of living, performs worse than the Acta_17 model, which indicates that
more data helps.

#### 4.3.2. Language Models

We used the Latin corpus presented in Section 4.2 to train all LMs. We applied
a data split of 90% for training and 10% for testing and kept it constant for
all models.

##### Statistical LM

We used the KenLM package [Heafield (2011, Heafield et al. (2013] to compute a
5-gram Kneser-Ney interpolated LM on the Latin training data. It achieves a
PPL score of 23.13 on the Latin corpus test set.

##### Masked Language Models

For the evaluation with the help of MLMs, we used the pre-trained multilingual
BERT model based on ?). Its training data consists of Wikipedia articles in
104 different languages, among which we also find Latin. It is available via
the Hugging Face model hub [Wolf et al.
(2019].141414https://huggingface.co/bert-base-multilingual-cased

Since the multilingual BERT model uses WordPiece tokenisation and a shared
vocabulary of 110k subtokens, we hypothesise that a single-language MLM should
be superior to multilingual BERT. Thus, we trained a RoBERTa model with
standard settings on the training set of our Latin corpus. We make the
model151515https://huggingface.co/pstroe/roberta-base-latin-cased, as well as
the training and test
data161616https://huggingface.co/datasets/pstroe/cc100-latin available on the
Hugging Face hub.

## 5\. Experiments

### 5.1. HTR Quality Estimation Suitability

After training the HTR and language models and preparing our language
resource, we want to investigate whether the metrics introduced in Section 3.1
are suitable for HTR quality evaluation. We aim to establish a correlation
between the CERs and the different metrics. Thus, we applied each model to our
test set and computed the scores of our metrics. For evaluating PPPL, PPL, and
token ratio, we preprocessed the HTR result in the same way as our Latin
reference corpus to guarantee a fair comparison.

Our null hypothesis H0 is that the metrics’ scores on the test data and the
CERs that the models achieve on the same data do not correlate. We follow ?)’s
approach in fitting linear models with the CERs as dependent variables and the
scores of the metrics as predictors using the R package [R Core Team (2021].
The adjusted R² informs us about the goodness of fit, where a higher value
indicates the model can fit the data well. This procedure helps us determine
whether a metric reliably estimates HTR quality.

### 5.2. Ranking Ability

The ranking tells us whether our metrics capture differences in HTR models.
Our null hypothesis H0 states that there is no correlation between the
rankings of different models based on CERs and the rankings based on the
metrics. Rejecting H0 would prove the existence of such a correlation and
validate the proposed metrics. Furthermore, the strength of the correlation
between the CER and each of the metrics allows us to determine which metric
the ranking should be based on.

We compute Spearman’s rank correlation coefficient $\rho$ between the
reference and metric rankings to check for correlations. It is given by

$\rho=1-\frac{6\sum d_{i}^{2}}{n(n^{2}-1)},$ (6)

where $d_{i}$ denotes the difference between two ranks and $n$ is the number
of ranks. A comparison against critical values indicates whether the
correlation is significant, thus rejecting H0.

Applied to our data, this means that we ran all our models on the Gwalther57
and the Bullinger825 test sets in block 1b in Fig. 1, computed the
corresponding CERs, and ranked the models according to the CERs. These are our
reference rankings. We added the reference ranking for the Gwalther433
validation set in block 1a. Table 1 shows that many models perform in similar
ranges, especially for the Acta-based models. Our metrics should reflect this
by returning higher values when we apply the models to the validation set. We
then took each model output for the Gwalther433, Gwalther57, and the
Bullinger825 test set and applied our metrics to it. This produces a score for
each model and metric, which we used to rank the models. Lastly, we computed
Spearman’s $\rho$ by comparing each metric ranking to its corresponding
reference ranking. For example, we compared the model ranking of the PPL
metric on the Gwalther57 test set against the model ranking according to the
CERs on the same set.

### 5.3. Hit Top-N

The last investigation we conducted is a check for each metric on how well it
identifies a model in the Top-1, Top-3, or Top-5. As we see in Table 1, the
Top-5 Acta-based models have a difference of at most 0.35 percentage points,
which lead us to assume that for practical applications, a method that would
identify either of those five models as the best performing model was suitable
for model selection. A single factor ANOVA test on the Gwalther433 with a
p-value of 0.71 hardens our suspicions that the means between the Top-5
performing models are not significantly different.

## 6\. Results and Discussion

Table 2 shows the adjusted R² values for the models we fitted for each CER vs.
metric pair. The superscript behind each score indicates the polynomial degree
of the model. The higher the R² score, the better the model fits the data,
i.e., the higher the correlation between metric and actual CER. We exemplify
this with two metrics for the Gwalther57 test set from the Table 2 in Fig. 4.
Each black dot represents a model. We notice the linear relationship (blue
line) between metric and CER.

The big picture from HTR quality estimation suitability confirms that our
metrics generally find it easier to score the Gwalther433 set. This is only
logical since we trained most of our models on the “Gwalther” set.

Figure 4: Model fit for two measures (Token ratio left and PPL right) for the
Gwalther57 test set. Regression line in blue, 95% confidence interval in grey.

|  | Adj. R²
---|---|---
Metrics |  Gwalther433 |  Bullinger825 |  Gwalther57
| BERT | 0.99² | 0.80² | 0.92²
PPPL | RoBERTa | 0.98¹ | 0.69² | 0.88²
PPL | Statistical LM | 0.65² | 0.52¹ | 0.42¹
Token ratio | 0.99¹ | 0.954 | 0.92³
Character n-grams | 2-gram | 0.42² | -0.01¹ | 0.14²
3-gram | -0.03¹ | 0.09¹ | -0.03¹
4-gram | 0.80² | 0.35¹ | 0.06¹
5-gram | 0.96² | 0.79¹ | 0.81¹
6-gram | 0.99² | 0.96³ | 0.96²
7-gram | 0.99² | 0.99² | 0.97²

Table 2: R² values for all metrics compared to the performance of the models
on the respective validation or test sets. The superscript indicates the
polynomial degree of the model we fitted to the data.

To put our results in relation to previous work, we would like to point out
that some of our R² scores are considerably higher than what ?) reported for
their confidence vs. CER and lexicality vs. CER correlations. However, our
results confirm the importance of the lexicality of the output and that its
measuring is suitable for HTR quality estimation. As concerns character
n-grams, the general tendency is, the higher the order of the n-gram, the
better it correlates with the CER.

As for the (P)PPL metrics, especially the scoring with the multilingual BERT
model results in reliable estimates for HTR quality. We found it surprising
that a dedicated Latin model does not perform better than multilingual BERT.
However, the upside of this finding is that we can use a pre-trained model out
of the box for HTR quality estimation without having to train additional
models. Moreover, the solid performance of PPPL makes the collection of
language-specific data superfluous.

Moving on to the examination of the ranking capabilities of our metrics, we
find that most metrics’ rankings exhibit strong correlations to the reference
rankings, as Table 3 shows. Overall, the ranking correlation shows a similar
picture as the R² values do. Again, the token ratio metric shows strong
correlations. The PPL of the statistical LM cannot keep up with the other
measures, while our newly proposed application of PPPL for ranking purposes
perform equally well as the token ratio metric, at least on the Gwalther data.

| Ranking Reference (CER)
---|---
Metrics | Gwalther433 | Bullinger825 | Gwalther57
| BERT | missing0.98 | missing0.90 | missing0.90
PPPL | RoBERTa | missing0.96 | missing0.82 | missing0.85
PPL | Statistical LM | missing0.78 | missing0.65 |  missing0.71
Token ratio |  | missing0.98 | missing0.95 | missing0.91
Character n-grams | 2-gram | -0.28 | 0.28 | 0.13
3-gram | 0.01 | missing0.48 |  missing0.36
4-gram | missing0.58 | missing0.62 | 0.15
5-gram | missing0.90 | missing0.88 | missing0.70
6-gram | missing0.97 | missing0.94 | missing0.92
7-gram | missing0.99 | missing0.97 | missing0.94

Table 3: The Spearman correlation values for all measures. Significance
levels: 0.05, 0.025, 0.01, 0.005, 0.001.

Lastly, we take a look at the Top-N evaluation in Table 4. We see that metrics
select the best model in 70% of the cases for the Bullinger825 test set and in
60% of the cases for the Gwalther57 test set, respectively. This is in stark
contrast to the Gwalther433 validation set, with only 30% Top-1 hits. This is
mainly because the CERs of the models are very close to each other on the
validation set, thus making it harder for alternative metrics to pick the best
model. All in all, our metrics select one of the five best performing models
90% of the time for the Bullinger825 set and 80% for the Gwalther57 set, thus
confirming that our metrics are capable of model selection.

Metrics | Ranking reference
---|---
Gwalther433 | Bullinger825 | Gwalther57
PPPL | BERT | 1 | 1 | 1
RoBERTa | 1 | 1 | 1
PPL | Statistical LM |  | 3 |
Token ratio |  | 3 | 1 | 1
Character n-grams | 2-gram |  |  |
3-gram |  | 3 | 5
4-gram |  | 1 | 3
5-gram | 5 | 1 | 1
6-gram | 1 | 1 | 1
7-gram | 3 | 1 | 1

Table 4: Top-N values for each metric.

## 7\. Conclusion

This paper proposes metrics based on lexical features and (pseudo-)perplexity
scores for HTR quality estimation and model selection. We confirmed other
research that the lexicality of an HTR result is a strong indicator of
quality. However, lexical resources are not always readily available,
especially for low-resourced languages. While we dealt with Latin in this
paper, we would like to recall that the Bullinger correspondence contains
Early New High German texts (ENHG). We would need to assemble a big reference
corpus for our or other lexicon-based methods to work. Our experiments
confirmed that HTR quality estimation metrics based on transformers like
multilingual BERT are suitable replacements. We leave it for future work
whether pseudo-perplexity aligns with CER for historical German. After all,
ENHG is not part of multilingual BERT’s training data, and it is far more
irregular and includes many more spelling variants (especially compared to
Latin).

The pseudo-perplexity score also proved capable of ranking HTR outputs.
Applying different models to new data and evaluating the output with ground
truth-free metrics also opens up new possibilities to further improve HTR
quality. For example, when we apply the metrics to lines of different model
outputs, we can puzzle together the lines with the best scores, thus combining
the best possible outputs from different models.

Our findings are valuable for digitisation initiatives by libraries and
archives, which are concerned with the automatic transcription of handwritten
documents. Especially our suggested employment of transformers works out of
the box and provides quality estimates, based on which rankings of different
models are possible. Hence, libraries and archives could quickly produce
estimates of the HTR (or OCR) and identify quality issues. It needs further
research to investigate whether such metrics are suitable for mass
application.

## 8\. Acknowledgements

We express our gratitude to the UZH Foundation and the various sponsors of
this project171717see https://www.bullinger-digital.ch/about. We thank the
Zurich State Archive and the Zurich Central Library for providing the scans.

## 9\. Bibliographical References

## References

  * Alex and Burns (2014 Alex, B. and Burns, J.  (2014).  Estimating and rating the quality of optically character recognised text.  In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage, pages 97–102.
  * Chiron et al. (2017 Chiron, G., Doucet, A., Coustaty, M., Visani, M., and Moreux, J.-P.  (2017).  Impact of OCR errors on the use of digital libraries: Towards a better access to information.  In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL). IEEE.
  * Clausner et al. (2016 Clausner, C., Pletschacher, S., and Antonacopoulos, A.  (2016).  Quality Prediction System for Large-Scale Digitisation Workflows.  In 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pages 138–143, Santorini, Greece, April. IEEE.
  * Conneau et al. (2019 Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V.  (2019).  Unsupervised cross-lingual representation learning at scale.  arXiv preprint arXiv:1911.02116.
  * Devlin et al. (2018 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Graves and Schmidhuber (2008 Graves, A. and Schmidhuber, J.  (2008).  Offline handwriting recognition with multidimensional recurrent neural networks.  Advances in neural information processing systems, 21:545–552.
  * Graves et al. (2009 Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H., and Schmidhuber, J.  (2009).  A novel connectionist system for unconstrained handwriting recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5):855–868.
  * Hamdi et al. (2020 Hamdi, A., Jean-Caurant, A., Sidère, N., Coustaty, M., and Doucet, A.  (2020).  Assessing and minimizing the impact of ocr quality on named entity recognition.  In International Conference on Theory and Practice of Digital Libraries, pages 87–101. Springer.
  * Heafield et al. (2013 Heafield, K., Pouzyrevsky, I., Clark, J. H., and Koehn, P.  (2013).  Scalable modified kneser-ney language model estimation.  In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696.
  * Heafield (2011 Heafield, K.  (2011).  Kenlm: Faster and smaller language model queries.  In Proceedings of the sixth workshop on statistical machine translation, pages 187–197.
  * Hill and Hengchen (2019 Hill, M. J. and Hengchen, S.  (2019).  Quantifying the impact of dirty OCR on historical text analysis: Eighteenth Century Collections Online as a case study.  Digital Scholarship in the Humanities, 34(4):825–843.
  * Hodel et al. (2021 Hodel, T. M., Schoch, D. S., Schneider, C., and Purcell, J.  (2021).  General models for handwritten text recognition: Feasibility and state-of-the art. german kurrent as an example.  Journal of open humanities data, 7(13):1–10.
  * Holley (2009 Holley, R.  (2009).  How good can it get? Analysing and improving OCR accuracy in large scale historic newspaper digitisation programs.  D-Lib Magazine, 15(3/4).
  * Johnson et al. (2021 Johnson, K. P., Burns, P. J., Stewart, J., Cook, T., Besnier, C., and Mattingly, W. J. B.  (2021).  The Classical Language Toolkit: An NLP Framework for Pre-Modern Languages.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 20–29, Online, aug. Association for Computational Linguistics.
  * Kang et al. (2021 Kang, L., Riba, P., Villegas, M., Fornés, A., and Rusiñol, M.  (2021).  Candidate fusion: Integrating language modelling into a sequence-to-sequence handwritten word recognition architecture.  Pattern Recognition, 112:107790.
  * Liu et al. (2019 Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  (2019).  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * Marti and Bunke (2002 Marti, U.-V. and Bunke, H.  (2002).  The iam-database: an english sentence database for offline handwriting recognition.  International Journal on Document Analysis and Recognition, 5(1):39–46.
  * Michael et al. (2018 Michael, J., Weidemann, M., and Labahn, R.  (2018).  Htr engine based on nns p3.  Technical report.
  * Mühlberger et al. (2019 Mühlberger, G., Seaward, L., Terras, M., and 51 more authors.  (2019).  Transforming scholarship in the archives through handwritten text recognition: Transkribus as a case study.  Journal of Documentation, 75(5):954–976.
  * Neudecker et al. (2021 Neudecker, C., Baierer, K., Gerber, M., Clausner, C., Antonacopoulos, A., and Pletschacher, S.  (2021).  A Survey of OCR Evaluation Tools and Metrics.  In Proceedings of the 6th International Workshop on Historical Document Imaging and Processing (HIP’21), Lausanne, Switzerland, September 6, 2021. ACM.
  * Nguyen et al. (2019 Nguyen, T.-T.-H., Jatowt, A., Coustaty, M., Nguyen, N.-V., and Doucet, A.  (2019).  Post-ocr error detection by generating plausible candidates.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 876–881.
  * Pontes et al. (2019 Pontes, E. L., Hamdi, A., Sidere, N., and Doucet, A.  (2019).  Impact of OCR quality on named entity linking.  In International Conference on Asian Digital Libraries, pages 102–115. Springer.
  * R Core Team (2021 R Core Team, (2021).  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing, Vienna, Austria.
  * Reffle and Ringlstetter (2013 Reffle, U. and Ringlstetter, C.  (2013).  Unsupervised profiling of OCRed historical documents.  Pattern Recognition, 46:1346–1357, 05.
  * Sabir et al. (2017 Sabir, E., Rawls, S., and Natarajan, P.  (2017).  Implicit language model in lstm for ocr.  In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 7, pages 27–31. IEEE.
  * Salah et al. (2015 Salah, A. B., Moreux, J.-P., Ragot, N., and Paquet, T.  (2015).  Ocr performance prediction using cross-ocr alignment.  In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 556–560. IEEE.
  * Salazar et al. (2019 Salazar, J., Liang, D., Nguyen, T. Q., and Kirchhoff, K.  (2019).  Masked language model scoring.  arXiv preprint arXiv:1910.14659.
  * Sánchez et al. (2019 Sánchez, J. A., Romero, V., Toselli, A. H., Villegas, M., and Vidal, E.  (2019).  A set of benchmarks for handwritten text recognition on historical documents.  Pattern Recognition, 94:122–134.
  * Sarkar et al. (2001 Sarkar, P., Baird, H. S., and Henderson, J.  (2001).  Triage of ocr results using confidence scores.  In Document Recognition and Retrieval IX, volume 4670, pages 216–222. International Society for Optics and Photonics.
  * Schambach (2010 Schambach, M.-P.  (2010).  Reviewing performance metrics for handwriting recognition: Must-rejects and recognition graph scores.  In 2010 12th International Conference on Frontiers in Handwriting Recognition, pages 457–462. IEEE.
  * Springmann et al. (2016 Springmann, U., Fink, F., and Schulz, K. U.  (2016).  Automatic quality evaluation and (semi-) automatic improvement of ocr models for historical printings.  http://arxiv.org/abs/1606.05157.
  * Ströbel et al. (2020 Ströbel, P. B., Clematide, S., and Volk, M.  (2020).  How much data do you need? about the creation of a ground truth for black letter and the effectiveness of neural OCR.  In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3551–3559, Marseille, France, May. European Language Resources Association.
  * Terras (in press Terras, M.  (in press).  Inviting ai into the archives: The reception of handwritten recognition technology into historical manuscript transcription.  In Lise Jaillant, editor, Archives, Access and AI, Digital Humanities Research. Transcript Verlag.
  * van Strien et al. (2020 van Strien, D., Beelen, K., Ardanuy, M., Hosseini, K., McGillivray, B., and Colavizza, G.  (2020).  Assessing the Impact of OCR Quality on Downstream NLP Tasks.  In Proceedings of the 12th International Conference on Agents and Artificial Intelligence. SCITEPRESS - Science and Technology Publications.
  * Vaswani et al. (2017 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  (2017).  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008.
  * Wenzek et al. (2019 Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E.  (2019).  Ccnet: Extracting high quality monolingual datasets from web crawl data.  arXiv preprint arXiv:1911.00359.
  * Wolf et al. (2019 Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.  (2019).  Huggingface’s transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771.