# Optimization of Image Processing Algorithms for Character Recognition in
Cultural Typewritten Documents

Mariana Dias<EMAIL_ADDRESS>Faculty of Engineering of the University of
Porto and INESC-TECPortugal and  Carla Teixeira Lopes<EMAIL_ADDRESS>Faculty of
Engineering of the University of Porto and INESC-TECPortugal

(2023)

###### Abstract.

Linked Data is used in various fields as a new way of structuring and
connecting data. Cultural heritage institutions have been using linked data to
improve archival descriptions and facilitate the discovery of information.
Most archival records have digital representations of physical artifacts in
the form of scanned images that are non-machine-readable. Optical Character
Recognition (OCR) recognizes text in images and translates it into machine-
encoded text. This paper evaluates the impact of image processing methods and
parameter tuning in OCR applied to typewritten cultural heritage documents.
The approach uses a multi-objective problem formulation to minimize
Levenshtein edit distance and maximize the number of words correctly
identified with a non-dominated sorting genetic algorithm (NSGA-II) to tune
the methods’ parameters. Evaluation results show that parameterization by
digital representation typology benefits the performance of image pre-
processing algorithms in OCR. Furthermore, our findings suggest that employing
image pre-processing algorithms in OCR might be more suitable for typologies
where the text recognition task without pre-processing does not produce good
results. In particular, Adaptive Thresholding, Bilateral Filter, and Opening
are the best-performing algorithms for the theatre plays’ covers, letters, and
overall dataset, respectively, and should be applied before OCR to improve its
performance.

image processing, optical character recognition, parameter tuning, cultural
heritage

††copyright: rightsretained††journalyear: 2023††doi: 10.1145/3606705††journal:
JOCCH††ccs: Applied computing Optical character recognition††ccs: Information
systems Digital libraries and archives

##  1\. Introduction

Cultural heritage institutions have the role of protecting, preserving, and
facilitating access to cultural knowledge. In the current digital era,
cultural material can be digitally preserved and remotely accessed by the
public. Cultural heritage institutions have been adopting Linked Data to
improve archival descriptions and information discovery (Niu, 2016). Linked
data describes a web of data; it refers to a machine-readable structured data
network connected to and from external data sets (Bizer et al., 2009). In
cultural heritage, records’ metadata, artifacts, and concepts can be linked to
improve data exploration (Bromage, 2016).

Digitalization methodologies are increasingly being applied to allow remote
access to archival content. As digitalization in cultural heritage contents
usually results in non-machine-readable representations, an initial conversion
to a machine-readable format is needed to fully explore the potential of these
representations in Linked Data implementations. This necessity happened in
EPISA (Entity and Property Inference for Semantic Archives), a project that
explores the use of Linked Data in the Portuguese Archives. As most of the
digital representations in the Portuguese Archives are non-machine-readable,
we first needed to convert them to a machine-readable format to build a tool
(Dias and Lopes, 2022) that automatically suggests description values using
the content of digital representations.

The process of recognizing handwritten or printed text in digital images and
translating them into machine-readable documents is called Optical Character
Recognition (OCR). The success of the OCR process depends on the quality of
the scanned image, and it is common for heritage documents to suffer some
degree of degradation over time. From uneven illumination to erased characters
and angled digital representations, image processing methods can be applied
before the text recognition phase to improve image quality and overall text
extraction. Various parameters can tune image processing algorithms. Using
default parameter values may yield bad performance results compared to the
optimized combination of parameters. Therefore, the task of assigning values
to parameters is essential.

We explore several image processing algorithms, using OpenCV, an image
processing Python library with the Tesseract OCR engine. To solve the
parameters’ optimization problem, we apply a non-dominated sorting genetic
algorithm (NSGA-II) that instantiates the values of parameters of the image
processing algorithms. In the end, the algorithm generates a solution set for
each pair of image processing algorithms and type of digital representation.
In the evaluation stage, we compare the OCR performance using different image
processing algorithms with the default and the obtained parameters.

The main contributions of this article are summarized as follows:

  * •

We analyze the effectiveness of NSGA-II for the parameterization of image
processing algorithms;

  * •

We compare global parameterization (i.e., considering all the digital
representations regardless of their type) with a parameterization by digital
representation type (e.g., letters, structured reports);

  * •

We analyze the impact of image pre-processing on OCR considering global and by
typology parameterization;

  * •

We create and release a cultural heritage dataset (Dias, 2022) of Portuguese
typewritten digital representations from the 20th century and a dataset of its
manual transcriptions (Falcão, 2022);

  * •

We disclose the source code111https://github.com/feup-infolab/archmine of the
parameterization of image processing algorithms.

The article is organized as follows. The various OCR-related tools and
performance measures are explored in Section 2, while the related research is
detailed in Section 3. Section 4 explains the proposed approach. Section 5
formalizes the problem of parameter tuning of image processing algorithms as
an optimization problem and analyzes the effect of parameterization on the OCR
performance, comparing it to default parameter values. We analyze the effect
of each image processing algorithm on the OCR performance in Section 6,
comparing it to an OCR scenario where no image pre-processing is used. The
results are discussed in Section 7. Finally, conclusions are drawn in Section
8.

##  2\. Background

Optical Character Recognition is an active research field with constant new
developments where most recent works use deep learning advances. In this
section, we describe different OCR software, image processing algorithms,
approaches to parameter optimization, and the evaluation of OCR systems.

###  2.1. OCR Software

There are currently four prominent OCR software applications: ABBYY
FineReader, Google Cloud Vision API, Ocropy, and Tesseract. Both Tesseract and
Ocropy are open-source engines, while ABBYY FineReader and Google Cloud Vision
API are commercial engines. In terms of approaches, all engines use Deep
Neural Networks (DNN) to train their models, except for Ocropy which uses
shallow networks. A DNN can improve accuracy by adding training data (Drobac
and Lindén, 2020), unachievable with shallow networks.

Furthermore, while Ocropy is a language-independent engine, ABBYY FineReader,
Google Cloud Vision API, and Tesseract support dictionaries and language
modeling. It is not easy to give a general definitive comparison of the
different engines based on accuracy because the characteristics of different
datasets influence the results. Engines are also affected by languages and
font types. Moreover, recent versions of some engines have additional or
updated training models and data that improve the performance and invalidate
past research conclusions over which software is the best. In Table 1, a
comparison between the different highlighted technologies is available. One of
the comparison criteria is whether the OCR software supports the Portuguese
language because this work uses a dataset of Portuguese cultural heritage
documents.

Table 1. Comparison of four OCR software applications. OCR Software | Approach | Platforms | License | Languages
---|---|---|---|---
| ABBYY
---
FineReader
| Convolutional Neural Networks
---
(CNN) and Recurrent Neural
Networks (RNN) (FineReader PDF, 2019)
| Mac,
---
Windows,
Online
Proprietary |  | Supports Portuguese with
---
dictionary support (FineReader PDF, 2021)
| Google Cloud
---
Vision API
| CNN-based with language
---
models (Walker et al., 2018)
Online | Proprietary |  | Supports Portuguese
---
through Latin scripts (Walker et al., 2018)
Ocropy |  | One-dimensional Long
---
Short-Term Memory (LSTM)
without language models (Drobac and Lindén, 2020)
Linux | Apache |  | Supports languages that
---
use Latin scripts (Baierer, 2020)
Tesseract |  | LSTM-based engine with
---
language models
| Mac,
---
Windows,
Linux
Apache |  | Portuguese language
---
model with dictionary
package (Weil, 2021)

###  2.2. Image Processing

Image processing techniques aim to visually enhance images and make them more
suitable for human or machine interpretation. The strategies highlighted in
this paper are image binarization and noise reduction.

Binarization consists of transforming colored pixels into black and white
ones. Most recent implementations of binarization techniques are based on
thresholding (Tensmeyer and Martinez, 2020). Thresholding is a segmentation
method that sets pixels to black or white depending on whether the grey value
is above or below a threshold value to evidence contrast between the
background and foreground.

There are various noise reduction techniques, such as filtering and
morphological transformations. Filters can be grouped into smoothing and
sharpening. Morphological transformations are used to thin characters, connect
broken strokes, and smooth contours (Kumar et al., 2013).

OpenCV is an open-source computer vision Python library launched in 1999 by
Intel (Mohamad, 2015) with various image processing algorithms, specifically
for binarization, image smoothing, and morphological transformations. The
library offers four binarization operations: adaptive threshold, Otsu
threshold, simple threshold, and triangle threshold (see Tables 2 and 3); four
different smoothing operations: bilateral filter, Gaussian filter, homogeneous
blur, and median blur (see Tables 4 and 5); seven different morphological
transformation operations: black hat, closing, dilation, erosion,
morphological gradient, opening, and top hat (see Tables 6 and 7). Tables 2,
4, and 6 describe the meaning of image processing algorithms from the OpenCV
library. Tables 3, 5, and 7 describe the meaning and typology of image
processing algorithms’ parameters.

Table 2. Description of binarization algorithms in the OpenCV library. Algorithm | Description
---|---
Adaptive thresholding | Calculates the threshold value within the pixels’ neighborhood area
Otsu thresholding | Automatically selects a threshold value based on interclass variance maximization
Simple thresholding | Applies the same global threshold value to every pixel
Triangle thresholding |  | Histogram-based algorithm that automatically selects a threshold value based on
---
interclass variance maximization
Table 3. Description of binarization algorithms’ parameters in the OpenCV library. Parameter Name | Parameter Type | Meaning
---|---|---
adaptiveMethod | nominal |  | Adaptive thresholding method. Takes the values: mean (0);
---
or gaussian (1)
blockSize | discrete | Size of a pixel’s neighborhood area
constant | continuous | Arbitrary constant
maxValue | continuous |  | Maximum value assigned to pixels that exceed the threshold value
---
thresh | continuous | Threshold value
thresholdType | nominal |  | Thresholding type. Takes the values: threshold binary (0);
---
or threshold binary inverted (1)
type | nominal |  | Thresholding type. Takes the values threshold binary (0);
---
threshold binary inverted (1); truncate (2); threshold to zero (3);
or inverted (4)
Table 4. Description of image smoothing algorithms in the OpenCV library. Algorithm | Description
---|---
Bilateral filter | Uses Gaussian filter, but its weight has one more component: difference in intensity
Gaussian blur | Blurs an image using Gaussian filter
Homogeneous blur | Blurs an image using a normalized box filter
Median blur | Blurs an image by replacing each pixel with a median of its neighborhood pixels
Table 5. Description of image smoothing algorithms’ parameters in the OpenCV library. Parameter Name | Parameter Type | Meaning
---|---|---
anchor | discrete | Anchor point
borderType | nominal |  | Border type. Takes the values: constant (0); replicate (1); reflect (2);
---
default (3); or isolated (4)
d | discrete | Diameter of each pixel neighborhood
ksize | discrete | Aperture linear/Gaussian kernel size
sigmaColor | continuous | Filter sigma in the colour space
sigmaSpace | continuous | Filter sigma in the coordinate space
sigmaX | discrete | Gaussian kernel standard deviation in X direction
sigmaY | discrete | Gaussian kernel standard deviation in Y direction
Table 6. Description of morphological transformation algorithms in the OpenCV library. Algorithm | Description
---|---
Black hat | Difference between input image and closing of the image
Closing | Dilation followed by erosion
Dilation | Enlarges boundaries of regions of foreground pixels
Erosion | Erodes boundaries of regions of foreground pixels
Morphological gradient | Difference between input image and opening of the image
Opening | Erosion followed by dilation
Top hat | Difference between input image and opening of the image
Table 7. Description of morphological transformation algorithms’ parameters in the OpenCV library. Parameter Name | Parameter Type | Meaning
---|---|---
anchor | discrete | Anchor point
borderType | nominal |  | Border type. Takes the values: constant (0); replicate (1); reflect (2);
---
default (3); or isolated (4)
iterations | discrete | Number of times technique is applied
kernel | continuous |  | Structuring element that determines the shape of a pixel
---
neighborhood

###  2.3. Parameter Optimization

To select appropriate values for the algorithms’ parameters, we can use
default values or manual configurations based on literature, experience, or
trial-and-error (Probst et al., 2019). OpenCV’s documentation presents some
examples of implementations using recommended assigned values. For instance,
in thresholding algorithms, 255 is the typical value assigned to the maxValue
parameter. However, these configurations do not guarantee an optimal
parameterization of the algorithms as the parameters’ values should be
adjusted to the images’ state. As an alternative, parameter tuning can be done
using a combinatorial search approach. According to the literature, the
genetic and Hill-Climbing algorithms are the two best-tailored approaches to
handle various parameters and their values (Kotthoff, 2016).

A genetic algorithm is a randomized search algorithm inspired by Charles
Darwin’s theory of natural selection (Yang, 2021). It has the premise that
individuals pass traits to their offspring, and the individuals with the best
characteristics, tend to survive and have more offspring (Yu et al., 1998).
Reproduction is the process in which the fitness function, i.e., the objective
function, dictates the probability of having offspring for the next generation
(Roetzel et al., 2020). In a minimization problem, lower fitness values mean a
higher likelihood of offspring. Crossover is a process in which offspring are
generated by combining traits from two parents, randomly selected members of
the last population. Mutations are used to ensure a generation’s diversity
onto the next one. The hill-climbing method uses neighbors in every iteration
to evaluate a current solution candidate (Pfaffe et al., 2017). It greedily
moves towards the neighbor with the highest value until there is no better
neighbor. Only the genetic algorithm can manipulate nominal parameter types in
a meaningful way from the two algorithms. In contrast, the hill-climbing
algorithm requires a notion of neighborhood.

As some of the OpenCV image processing algorithms have nominal parameters, we
use genetic algorithms. Genetic algorithms have many advantages in dealing
with complex problems, and parallelism as multiple offspring that act as
independent agents are ideal for parallel exploration of the search space. The
main disadvantage is the difficulty in formulating the fitness function and
parameters, such as the population size, crossover, and mutation rates (Yang,
2021). In particular, we adopt the Non-dominated Sorting Genetic Algorithm
(NSGA-II) (Blank and Deb, 2020) to solve the image processing optimization
problem. NSGA-II is the standard metaheuristic for solving multi-objective
optimization problems (Nebro et al., 2022). It has been applied to various
search and optimization problems, such as scheduling problems (Yahui et al.,
2020), resource allocation problems (Gao et al., 2019), and optimal parameter
selection problems (Su et al., 2021). The algorithm modifies the processes of
mating and survival selection of individuals. Since not all individuals can
survive, they are compared by rank with a binary tournament mating selection
and selected as a solution with crowding distance that uses Manhattan
Distance. Applying a ranking scheme encourages convergence and the crowing
distance density estimator promotes diversity (Nebro et al., 2022).

###  2.4. Evaluation of OCR systems

There are two ways to evaluate OCR systems: via text or layout. A text-based
evaluation is based on the comparison of the OCR text result against the
ground truth. Recent research has four prominent text-based measures:
Character Error Rate (CER), Word Error Rate (WER), Index/Count-based Bag of
Words, and Flex Character Accuracy. In Table 8, a comparison between the
different measures is available. Traditionally, text-based metrics use the
edit distance between the ground truth and the OCR output. That is the case
for the indicators CER and WER which are also dependent on reading order.
Reading order refers to a reading route for textual elements that can be
sequentially ordered lists for simple layouts, such as, a book page, or more
complex in the case of newspapers where there isn’t necessarily a left-to-
right order (Clausner et al., 2013). Its reading dependence makes them
unreliable when applied to complex page layouts since the performance is
influenced by page segmentation and text detection (Clausner et al., 2013).

Table 8. Comparison of text-based OCR performance measures. Measure | Description | Formula
---|---|---
|  Character Error Rate (CER) (Pletschacher et al., 2015)
---
| Minimum number of character edition
---
operations (insertions, deletions, and
substitutions) needed to obtain the ground
truth based on Levenshtein distance.
$\frac{C_{\text{insertions}}+C_{\text{deletions}}+C_{\text{substitutions}}}{\text{Number
of characters in text}}$
|  Word Error Rate (WER) (Pletschacher et al., 2015)
---
| Minimum number of word edition
---
operations (insertions, deletions, and
substitutions) needed to obtain the ground
truth based on Levenshtein distance.
$\frac{W_{\text{insertions}}+W_{\text{deletions}}+W_{\text{substitutions}}}{\text{Number
of words in text}}$
|  Index-based Bag of Words (Antonacopoulos et al., 2013)
---
| Uses the number of words recognized at
---
least once and does not account for every
occurrence.
$\frac{\text{Number of correct words}}{\text{Number of words in text}}$
|  Count-based Bag of Words (Antonacopoulos et al., 2013)
---
| Considers the correct count of the number
---
of occurrences of recognized words.
$\frac{\text{Number of words with correct occurrences}}{\text{Number of words
in text}}$
|  Flex Character Accuracy (Clausner et al., 2020)
---
| Based on CER but adapted to not depend on
---
reading order. Edit operations are determined
by text alignment and line matching, where
non-matched lines are insertions or deletions.
$\frac{L_{\text{insertions}}+L_{\text{deletions}}}{\text{Number of chunks in
text}}$

In contrast, a layout-based evaluation is based on the document page’s
structure. Measures analyze the document’s segmentation process in different
regions, classification of the areas, and reading order. The most popular ones
are segmentation errors, misclassifications, and reading order errors.

There are five types of OCR error analysis from the viewpoint of misspellings:
edit operations, length effects, erroneous character positions, real-word vs.
non-word errors, and word boundaries (Nguyen et al., 2020). Edit operations
can be deletions, insertions, substitutions, and transpositions of characters
to transform one string into another. Edit distances of 1 indicate single-
error tokens, while tokens with a higher edit distance indicate multi-error
tokens. There are two types of word errors: non-word and real-world errors
(Tong and Evans, 1996). Typically caused by the misrecognition of characters,
non-word errors occur when words are converted into invalid words. In
contrast, real-world errors arise when words are changed to valid words with a
different meaning (Lyu et al., 2021). Word boundary errors are caused by the
wrong insertion or deletion of white spaces, which results in incorrect split
errors with multiple words being merged into one or a word being split into
multiple words (Nguyen et al., 2020).

With the edit operation analysis, there are three types of OCR errors
(Carrasco, 2014): misspelled characters associated with the operation of
substitution, spurious symbols attributed to the insertion of characters, and
missing text related to the deletion of characters. An example of these errors
in the context of Cultural Heritage is the transcription of a structured
report from the 20th century, “RELATÓRIO N.º 3089” (REPORT N.º 3089). The OCR
engine reads it as “RELÁTORIO Nº 3”. The letters “089” are missing. Another
excerpt of the transcription reads “Enviado pela P.I.D.E.” (Sent by P.I.D.E.),
while the OCR output is “Enviado pela P.T.D.E.”. Here, the letter “I” is
misinterpreted as “T”. Another excerpt reads “permite-se aconselhar a
juventude” (allows to advise the youth), and the OCR output is “pDermite-se
aconselhar à juventude”. There is an insertion of the character “D” in the
word “permite-se” and a substitution of the character “a” with “à”.

##  3\. Related Work

Historical documents present various challenges to the OCR process. Most
engines require additional training to recognize historical fonts and
languages. Digital representations of historical printed records are often
noisy because of the papers’ degradation due to aging, such as smears and
faded text, and other printing noise like varying kerning and leading, i.e.,
different spaces between letters and line-break hyphenation amongst others
(Gupta et al., 2007).

A popular way to improve the OCR results in historical documents is to apply
an image pre-processing step to the OCR task. In the literature, various
approaches combine image processing algorithms. Harraj and Raissouni (2015)
applied illumination adjustment, grayscale conversion, unsharp masking, and
Otsu binarization to counteract image distortions and improve OCR accuracy.
Similarly, Ntogas and Ventzas (2009) trained a model for Finnish Fraktur fonts
and analyzed different combinations of pre-processing image algorithms.
Koistinen et al. (2017) show a process for image binarization with manual
image condition classification and image processing methods. The approach has
five stages: image acquisition, image preparation, filtering methods,
threshold methods, and image refinement. Bui et al. (2017) use a Convolution
Neural Network (CNN) to automatically select image processing algorithms
according to each input document image. Most of the previously mentioned works
have historical documents as their dataset. Ntogas and Ventzas (2009) used
document images from the Holy Monastery of Dousiko, Meteora, Greece, while
Koistinen et al. (2017) used Finnish historical documents.

According to the overall results of the previously mentioned research, the use
of image processing algorithms as a pre-processing step to the OCR process
improves the accuracy of the engines. Koistinen et al. (2017) improved word
level by 27.5% compared to FineReader 7 or 8 and 9.2% for FineReader 11.
Harraj and Raissouni (2015) registered an improvement in OCR accuracy between
2% to 6.8%. Bui et al. (2017) succeeded in proving the performance of engines
with their automatic pre-processing method selection, with Tesseract OCR
having an improvement of 25%.

As previously discussed in Section 2.3, there are various approaches to the
parameter tuning of image processing algorithms. Lund et al. (2013) selected
seven values for thresholding from the range of 0 to 255. There is an equal
incremental difference between the values and an equal division of the
grayscale spectrum. This experimental method reduced the error rate to 39.1%
in comparison to the original performance of the engine. To our knowledge,
there is no literature on combinatorial search algorithms applied to tuning
parameters of image processing algorithms in OCR.

In contrast, there are some examples of implementations of machine learning
algorithms to optimize OCR accuracy. For instance, Sporici et al. (2020)
developed an adaptive image pre-processing method using a reinforcement
learning model to minimize the edit distance between recognized text and
ground truth. The technique improved the F1-score of Tesseract 4.0 from 16.3%
to 72.9%.

Table 9 presents the comparison of the different approaches, algorithms, and
results for the discussed articles.

Table 9. Comparison of research papers related to image pre-processing in OCR
or parameter tuning.

Article | Methodology |  | Image processing
---
algorithms
| Evaluation
---
dataset
Results
(Ntogas and Ventzas, 2009) |  | Analyzed the performance of a
---
pipeline of image processing
algorithms according to a manual
image condition classification
| Filtering algorithms:
---
Mean, Median, Wiener,
Bullerworth, and Gaussian
Binarization algorithms:
Bernsen, Niblack, Otsu,
and Sauvola
| Documents from
---
the Holy Monastery
of Dousiko, Meteora,
Greece
| Binarization algorithms Sauvola
---
and Niblack had the best image
quality
(Lund et al., 2013) |  | Parameter tuning of thresholding
---
image processing methods by
selecting parameter values with
basis on an even distribution in
a range
| Global thresholding,
---
Otsu, and Sauvola
| Collection of 19th
---
Century Mormon
Article
| Reduction in error rate of 39.1% in
---
comparison to the original
performance of the OCR engine
(Harraj and Raissouni, 2015) |  | Pipeline of image pre-processing
---
methods
| Illumination adjustment,
---
grayscale conversion,
unsharp masking, and
Otsu binarization
Standard dataset |  | OCR accuracy improved between
---
2% to 6.8%
(Koistinen et al., 2017) |  | Analyze the performance of the
---
combination of different
thresholding algorithms
| Otsu thresholding,
---
Gaussian blur, Sauvola,
WolfJolion, Contrast Limited
Adaptive Histogram Equalized
(CLAHE), and Linear
Normalization
| Historical Finnish
---
documents
| Combination of four methods
---
(Linear Normalization+WolfJolion,
CLAHE+WolfJolion, original image
and WolfJolion) had the best results
with an improvement of word error
of 1.91% in comparison to FineReader
11, and 7.21% in comparison to
FineReader 7 and 8
(Bui et al., 2017) |  | Automatic selection of image
---
pre-processing algorithms with
a Convolution Neural Network
(CNN)
| Otsu binarization,
---
Su et al. binarization,
Gaussian filtering, non-local
means of noise reduction, and
Signh image sharpening
| Documents of
---
supermarket receipts
captured by mobile
devices
| Selection method improved
---
performance of Tesseract by 25%.
The neural network had a
performance from 93% to 96%
(Sporici et al., 2020) |  | Reinforcement learning model
---
that minimizes edit distance for
an adaptive image pre-processing
method
Adaptive thresholding |  | Brno Mobile OCR
---
Dataset (Kiss et al., 2019)
| F1-score of Tesseract 4.0
---
improved from 16.3% to 72.9%

##  4\. Methodology

Three research questions are associated with this experiment:

  * •

RQ1. Can applying parameter tuning to image processing algorithms with NSGA-II
benefit the OCR performance of cultural heritage digital representations?

  * •

RQ2. Can applying image processing algorithms to the cultural heritage digital
representations before the text recognition task improve the OCR performance
on the overall dataset?

  * •

RQ3. Can applying image processing algorithms to the cultural heritage digital
representations before the text recognition task improve the OCR performance
on subsets by typology?

The methodology of this work can be divided into four main steps: define and
collect a dataset of cultural heritage digital representations, explored in
Section 4.1, split the dataset into parameterization and evaluation subsets in
Section 4.2, define the image processing pipeline in Section 4.3, compare the
different types of parameterization and analyze the impact of image-processing
in the OCR process in Section 4.4.

###  4.1. Cultural Heritage Dataset

The training of image processing algorithms’ parameters and evaluation of the
OCR optimization process requires a dataset to conduct various experiments. To
fit the requirements of the EPISA project, we need a Portuguese cultural
heritage dataset. However, since there are currently no such available
datasets, we built a dataset of digital representations of archival records.
In this work, we use records from the National Archives of Torre do Tombo
(ANTT), a Portuguese central archive with millions of documents that date back
to the 9th century. This archive contains more than 40 million digital
representations and their respective archival records (DGARQ, 2008) that can
be shared and accessed through an information system called
Digitarq222https://digitarq.arquivos.pt/.

To build a dataset of digital representations of archival records, we first
had to identify typewritten representations in the overall set of
representations. Exploring the database, we found that the document
classification of the digital representations into typewritten or handwritten
is usually done in the fields “Dimension and Support”, “Scope and Content”,
and “Note”. The description forms varied from mentions to the Portuguese word
for “typewritten” (“datilografado”), its abbreviations (“dact.”) or
alternative ways of spelling, i.e., forms of the word before and after the
Portuguese Language Orthographic Agreement of 1990, “dactilografado”, and
“datilografado”. We searched the database for records whose descriptions in
the previously mentioned fields contained different forms of the term
“typewritten”. The query results returned the name of the digital
representation files and its dissemination URL that was later used to retrieve
each digital representation.

The dataset (Dias, 2022) has 27,017 one-page digital representations from
8,115 records of two fonds from the 20th century: the General Administration
of National Treasury (DGFP) and the National Secretariat of Information (SNI).
DGFP was a central department of the Ministry of Finance created during the
First Portuguese Republic regime (1910-1926) that managed public assets,
including movable and fixed assets that had been under the responsibility of
the former royal family. SNI was a public agency created during the
authoritarian Estado Novo regime (1933-1974) to supervise political
propaganda, press organs, popular culture, and entertainment, and promote
national tourism.

The digital representations were classified by writing format (handwritten,
typewritten, or blank), as some archival records containing handwritten or
blank documents are not relevant to our work. The dataset has 23,794
typewritten digital representations, 1,681 handwritten digital
representations, and 1,542 blank digital representations. Furthermore, the
dataset was classified into ten types of digital representations. The
typewritten dataset has 3,264 letters, 6,560 structured reports, 1,970 non-
structured reports, 82 covers of processes, 6 covers of minutes, 1,473
contents of minutes, 19 books’ covers, 182 books’ contents, 165 theatre plays’
covers, and 8,845 theatre plays’ contents. Figure 1 shows examples of
different types of typewritten digital representations from the dataset.
Identifying digital representation typologies relevant to the OCR task was
carried out by observing existing textual descriptions of the records and
document layouts of the digital representations. The classification of the
dataset was performed manually.

(a) Letter

(b) Structured report

(c) Non-structured report

(d) Cover of a process

(e) Cover of a theatre play

Figure 1. Digital representations from the typewritten dataset.

###  4.2. Parameterization and Evaluation Samples

To conduct an unbiased evaluation of the OCR optimization process, we need to
use different samples of the cultural heritage dataset to find the best
parameter values for image processing algorithms (Parameterization Dataset)
and to assess the algorithms’ performance in the text recognition task
(Evaluation Dataset). The best-performing image processing algorithms in the
Evaluation Dataset with tuned parameters in the Parameterization Dataset will
be used in the pre-processing phase of the OCR task with the Cultural Heritage
Dataset to extract its textual content.

The dataset of documents we are working with refers to historical records with
secondary value, i.e., evidential, and informative values, that no longer
possess administrative value (dos Arquivos Nacionais/Torre do Tombo, 2006).
Historical archives consider the documents’ information as a whole, unlike
current archives that detail the information contained in each document to
facilitate quick access to the data. For this reason, from the ten types of
digital representations of the Cultural Heritage Dataset, we decided to focus
only on the five we considered most informative: letters, covers of processes,
covers of theatre plays, structured and non-structured reports. These are the
ones with more significant potential for describing historical records.

From the five digital representation types, we selected 708 typewritten
digital representations. Per typology, we randomly selected 5% of the digital
representations of each archival series. If this methodology does not fulfill
the requirement of choosing at least 60 digital representations per typology
to maximize the probability of normal distributions necessary in parametric
tests, we select digital representations of each archival series until the
sample amounts to 60.

The digital representations of the samples were manually transcribed to
provide a ground truth in the evaluation stage (Falcão, 2022). We split the
sample into two: the Parameterization and the Evaluation Datasets. The
datasets each have 81 letters, 164 structured reports, 49 non-structured
reports, 30 covers of processes, and 30 theatre plays’ covers. Table 10
characterizes the Parameterization and Evaluation Datasets in comparison to
the Cultural Heritage Dataset regarding the number of one-page digital
representations through different typologies.

Table 10. Characterization of the Parameterization, Evaluation, and Cultural
Heritage Datasets’ digital representations.

Series’ Reference Code | Series Name | Parameterization / Evaluation / Cultural Heritage Datasets
---|---|---
Letter |  | Process
---
cover
| Structured
---
report
| Theatre play
---
cover
| Non-structured
---
report
PT/PNA/DGFP/0001 | Royal Palace’s Listings | 1/2/57 | - | - | - | -
PT/PNA/DGFP/0002 |  | List of deliveries made to
---
various institutions and
personalities
4/2/122 | - | - | - | -
PT/PNA/DGFP/0003 |  | Deliveries made to the Royal
---
Family and the respective
property appraisal processes
1/0/29 | - | - | - | -
PT/PNA/DGFP/0005 |  | Work documentation related
---
to the listings
1/0/29 | - | - | - | -
PT/PNA/DGFP/0006 | Correspondence | 1/5/113 | - | - | - | -
PT/TT/SNI-DGE/1 |  | Censorship cases for theatre
---
plays
7/0/138 | 18/22/54 | 0/1/20 | 30/30/165 | 0/0/1
PT/TT/SNI-DGE/3 |  | Minutes of the Censorship
---
Commissions’ Sessions
- | - | - | - | -
PT/TT/SNI-DSC/9 | Subscription of Press Records | 16/18/699 | 8/5/17 | 0/1/6 | - | 0/2/6
PT/TT/SNI-DSC/13 |  | Information about forbidden
---
books and period publications
41/48/1,782 | 2/1/5 | 3/0/66 | - | 0/0/2
PT/TT/SNI-DSC/19 |  | Records on the apprehension
---
and prohibition of publications
0/2/34 | 0/1/2 | - | - | -
PT/TT/SNI-DSC/22 | Records on new publications | 1/0/23 | 1/0/1 | - | - | -
PT/TT/SNI-DSC/24 | Non-periodic publications | 0/1/22 | 1/1/3 | 0/0/1 | - | -
PT/TT/SNI-DSC/35 | Reports on censored books | 8/3/216 | - | 161/162/6,467 | - | 49/49/1,961
Overall |  | 81/81/3,264 | 30/30/82 | 164/164/6,560 | 30/30/165 | 49/49/1,970

###  4.3. Image Processing Pipeline

We used the Tesseract engine for the OCR task with the Pytesseract wrapper for
Python, and the available tessdata_best (Weil, 2021) training data. We did not
train a language model, since a Portuguese language model is already
available, and literature claims that the engine is prepared to handle
typewritten documents.

For image processing, the OpenCV library described in Table 2 was used. Three
image processing techniques were selected: binarization, smoothing, and
morphological. The selection of the methods was based on the state of the
images in the dataset. The images’ degradation stems mainly from contrast
variation, bleed-through, faded ink, smears, and thin text (Sulaiman et al.,
2019), e.g., there is no angled digital representation; therefore, there is no
need for a skew correction algorithm. This work’s pipeline is shown in Figure
2.

Figure 2. Overview of the different steps of the proposed methodology.

###  4.4. Evaluation

Our methodology includes two evaluations that address the Research Questions
defined above: the effect of parameter tuning of image processing algorithms
on the OCR task (RQ1) in Section 5.3 and the effect of the algorithms
themselves (RQ2 and RQ3) in Section 6.

We used performance metrics based on the CER and count-based Bag of Words
presented in Table 8. The CER-based metric is presented as character accuracy
in Equation 1 to facilitate the comparison between the measure as success
rates. The count-based Bag of Words metric is presented as an F-measure in
Equation 4, based on the notions of precision in Equation 2 and recall in
Equation 3.

(1) $\text{Character Accuracy}=(1-\text{CER})*100$

(2) $\text{Precision}=\frac{\text{number of words with correct
occurrences}}{\text{number of words in OCR output text}}$

(3) $\text{Recall}=\frac{\text{number of words with correct
occurrences}}{\text{number of words in ground truth text}}$

(4)
$\text{F1}=2*\frac{\text{precision}*\text{recall}}{\text{precision}+\text{recall}}$

The evaluation of the effect of image processing algorithms also includes an
analysis of OCR error types. We analyzed the relation of each algorithm with
OCR error types based on the OCR error analysis of edit operations described
in Section 2.4.

##  5\. Parameter Tuning of Image Processing Algorithms

This section defines the problem of parameter tuning of image processing
algorithms as an optimization problem and analyzes the effect of
parameterization on the OCR performance, comparing it to default parameter
values. Section 5.1 details the process of formalizing and solving the problem
of parameter tuning into a multi-objective optimization. We showcase the best
parameter values for image processing algorithms obtained with
parameterization in Section 5.2, and evaluate the effect of parameterization
on the OCR performance in Section 5.3.

###  5.1. Problem Formalization and Optimization

The autotuning of image processing algorithms is defined as a constrained
multi-objective problem with two objective functions, $f_{1}(x)$ and
$f_{2}(x)$, formulated by Equations 5 (Fontan et al., 2016) and 6,
respectively.

(5) $lev_{a,b}(i,j)=\left\\{\begin{array}[]{cl}max(i,j)&,\ min(i,j)=0\\\
min\left\\{\begin{array}[]{cl}lev_{a,b}(i-1,j)+1\\\ lev_{a,b}(i,j-1)+1\\\
lev_{a,b}(i-1,j-1)+1\end{array}\right.&,\ otherwise\end{array}\right.$

(6) $bow_{a,b}(i)=\left\\{\begin{array}[]{cl}1&,\ n_{a}(i)=n_{b}(i)\\\ 0&,\
otherwise\end{array}\right.$

The first objective function, $f_{1}(x)$, is the Levenshtein edit distance
between the ground truth and the output. The edit distance calculates the
minimum number of characters’ edit operations, such as insertions,
substitutions, and deletions, to transform a string a into b with lengths i
and j, respectively. In contrast, the second objective function, $f_{2}(x)$,
is the number of words with the same word occurrences between the ground truth
and the output, where i is the index of matched words between strings a and b,
and $n_{a}(i)$ is the number of occurrences of the matched word i in string a.
The goal is to find solutions for the parameter values that satisfy each image
processing algorithm’s constraints and are as good as possible for both
objective functions (Blank and Deb, 2020).

We used the Non-dominated Sorting Genetic Algorithm (NSGA-II) (Blank and Deb,
2020) to perform parameter tuning of image processing algorithms. For the
formalization of the constraints, $g(x)$, the image processing algorithms’
equality constraints are transformed into inequality constraints $g(x)\leq 0$.
All the constraints needed for the selected image processing algorithms are of
the type “a parameter must be odd”, which can be expressed through the
equality constraint $|\text{parameter}\%2|=1$ and transformed into the
inequality $|\text{parameter}\%2-1|\leq 0$, where
$g(x)=|\text{parameter}\%2-1|$. The algorithms Adaptive Thresholding,
Homogeneous Blur, Median Blur, and Gaussian Blur all have a parity constraint
on the parameter blockSize, in the case of the thresholding algorithm, and on
the parameter ksize for the rest.

###  5.2. Best Parameters Per Digital Representation Typology

We used the parameterization dataset to tune image processing algorithms. We
considered two parameterization scenarios: global and by type of digital
representation. Global parameter tuning uses the overall parameterization
sample instead of subsets of the sample by digital representation typology.
The parameter values obtained using the two parametrization scenarios are
presented in Table LABEL:tab:params, along with the default values proposed by
the OpenCV documentation. These values do not intend to be reference values.
We decided to include them to show how parameterized values differ from the
default values, how values differ between parameterization scenarios, and how
values differ between types of digital representations.

Table 11. Parameters set for image processing algorithms with default values and values tuned by NGSA-II for global set of documents and classified types of digital representations. Algorithm | Parameter | Default | Global | Type of digital representation
---|---|---|---|---
Letter |  | Process
---
cover
| Structured
---
report
| Theatre play
---
cover
| Non-structured
---
report
Adaptive Thresholding | maxValue | 255 | 217 | 24 | 72 | 13 | 152 | 32
adaptiveMethod | 0 or 1 | 1 | 0 | 0 | 0 | 1 | 0
thresholdType | 0 | 0 | 0 | 0 | 0 | 1 | 0
blockSize | 11 | 33 | 39 | 65 | 43 | 57 | 51
c | 2 | 25 | 30 | 29 | 43 | 18 | 32
Bilateral Filtering | d | 9 | 4 | 4 | 2 | 2 | 1 | 2
sigmaColor | 75 | 10 | 4 | 85 | 36 | 6 | 18
sigmaSpace | 75 | 231 | 31 | 52 | 100 | 191 | 253
Black Hat | kernel | 5 | 10 | 6 | 229 | 139 | 30 | 38
iterations | 1 | 7 | 10 | 7 | 9 | 2 | 1
borderType | 3 | 4 | 4 | 2 | 3 | 1 | 1
Closing | kernel | 5 | 1 | 1 | 1 | 1 | 1 | 1
iterations | 1 | 2 | 2 | 4 | 9 | 4 | 3
borderType | 3 | 3 | 1 | 0 | 1 | 2 | 3
Dilation | kernel | 5 | 32 | - | 1 | 1 | 1 | 1
iterations | 1 | 8 | - | 4 | 7 | 9 | 8
borderType | 3 | 2 | - | 3 | 2 | 1 | 2
Erosion | kernel | 5 | 1 | 1 | 1 | 1 | 1 | 2
iterations | 1 | 5 | 7 | 1 | 5 | 2 | 1
borderType | 3 | 3 | 2 | 2 | 1 | 3 | 1
Gaussian Blur | ksize | 5 | 3 | 3 | 3 | 3 | 3 | 3
borderType | 3 | 1 | 0 | 1 | 1 | 1 | 1
Homogeneous Blur | ksize | 5 | 1 | 1 | 3 | 1 | 3 | 3
borderType | 3 | 1 | 2 | 3 | 1 | 0 | 3
Median Blur | ksize | 5 | 3 | 3 | 3 | 5 | 3 | 3
Morphological Gradient | kernel | 5 | 2 | 2 | 2 | 2 | 105 | 2
iterations | 1 | 1 | 1 | 2 | 1 | 2 | 2
borderType | 3 | 2 | 2 | 0 | 1 | 1 | 2
Opening | kernel | 5 | 2 | 1 | 2 | 1 | 2 | 2
iterations | 1 | 2 | 2 | 1 | 9 | 1 | 1
borderType | 3 | 0 | 4 | 1 | 4 | 0 | 3
Otsu Thresholding | maxValue | 255 | 199 | 24 | 251 | 92 | 34 | 14
type | 0 | 3 | 0 | 0 | 3 | 3 | 0
Simple Thresholding | thresh | 127 | 152 | 110 | 33 | 82 | 240 | 224
maxValue | 255 | 45 | 4 | 10 | 221 | 3 | 42
type | 0 | 1 | 3 | 3 | 3 | 4 | 4
Top Hat | kernel | 5 | 90 | 51 | 217 | 233 | 139 | 204
iterations | 1 | 5 | 7 | 5 | 2 | 5 | 2
borderType | 3 | 2 | 0 | 0 | 2 | 2 | 4
Triangle Thresholding | maxValue | 255 | 15 | 74 | 248 | 220 | 174 | 238
type | 0 | 3 | 3 | 2 | 2 | 3 | 0

We performed the parameterization on a PC with 3.20 GHz, AMD Ryzen 7 5800H,
and 16GB RAM. Table 12 shows the average time per digital representation to
parameterize image processing algorithms. The table shows that parameter
tuning for each digital representation took the least time, in the range of 2
minutes, with the image processing algorithms Dilation, Erosion, and
Morphological Gradient. In contrast, the parameterization for each digital
representation took the longest with the algorithms Black Hat, Opening, and
Otsu Thresholding, within 10 minutes.

Table 12. Average time to parameterize the image processing algorithms per digital representation. Technique | Algorithm |  | Average parameter tuning time
---
(min:s)
Binarization | Adaptive Thresholding | 06:46
Otsu Thresholding | 10:52
Simple Thresholding | 06:00
Triangle Thresholding | 08:52
Image Smoothing | Bilateral Filter | 09:11
Gaussian Blur | 07:12
Homogeneous Blur | 04:33
Median Blur | 08:39
Morphological Transformation | Black Hat | 10:29
Closing | 05:31
Dilation | 02:02
Erosion | 02:59
Morphological Gradient | 02:35
Opening | 10:35
Top Hat | 07:11

###  5.3. Effect of Parameterization on OCR Performance

To evaluate the effect of default, global, and digital representation-type
parameterization on the OCR performance, we compared the performances of image
processing algorithms in OCR with default parameter values to algorithms with
parameter values obtained by our two parameterization scenarios. To compare
the character accuracy and F1-score results in the three scenarios (default,
global parameterization, and parameterization by typology), we use the
Wilcoxon signed-rank test (Noether, 1992) with the Bonferroni correction. We
used non parametric tests because the assumptions for the t-test were not
verified. Table 13 presents the results of the one-sided Wilcoxon pairwise
comparisons. We used this non parametric test because the assumptions for the
t-test were not verified.

Table 13. Performance comparison between image processing algorithms tuned
with Wilcoxon pairwise comparisons using default, global and digital
representation-type parameters.

Technique | Algorithm | Character Accuracy | F1-score
---|---|---|---
| Default vs
---
Global
| Default vs
---
Typology
| Global vs
---
Typology
| Default vs
---
Global
| Default vs
---
Typology
| Global vs
---
Typology
Binarization |  | Adaptive Thresholding (a)
---
<<< | <<< | <<< | <<< | <<< | <<<
| Adaptive Thresholding (b)
---
<<< | <<< | <<< | <<< | <<< | <<<
| Otsu Thresholding
---
<<< | <<< | >>> | <<< | <<< | >>>
| Simple Thresholding
---
>>> | <<< | <<< | >>> | <<< | <<<
| Triangle Thresholding
---
0.65 | <<< | <<< | <<< | <<< | <<<
Image smoothing |  | Bilateral Filtering
---
<<< | <<< | 0.17 | <<< | <<< | 1
| Gaussian Blur
---
<<< | <<< | 1 | <<< | <<< | 1
| Homogeneous Blur
---
<<< | <<< | 0.19 | <<< | <<< | >>
| Median Blur
---
<<< | <<< | >>> | <<< | <<< | >>>
Morphological transformation | Black Hat | <<< | <<< | 0.2 | <<< | <<< | 0.24
Closing | <<< | <<< | 0.25 | <<< | <<< | 0.26
Dilation | >>> | <<< | <<< | >> | <<< | <<<
Erosion | <<< | <<< | 0.38 | <<< | <<< | 0.38
| Morphological Gradient
---
<<< | <<< | > | <<< | <<< | 0.19
Opening | <<< | <<< | <<< | <<< | <<< | <<<
Top Hat | <<< | <<< | 1 | <<< | <<< | 0.44

(a) Adaptive Thresholding using adaptiveMethod=0;
(b) Adaptive Thresholding using adaptiveMethod=1;
For ¡ or ¿, P¡0.05; for << or >> P¡0.01; for <<< or >>> P¡0.001
Displays the p-values of Wilcoxon pairwise comparisons with symbol “¡” for
significantly worse performances and “¿” for significantly better
performances.

By analyzing the statistical outcomes, the OCR performance using digital
representation-type parameterization on the image processing algorithms shows
statistically significant improvements in both measures compared to the
parameterization using default values. Similarly, the OCR performance using
global-type parameterization also shows significant improvement in both
measures compared to using default values, except for Simple Thresholding and
Dilation where the performance is significantly worse. For most algorithms,
there is either not a significant difference between global and digital
representation-type parameterization or the specific parameterization performs
better in both measures, except for the algorithms Otsu Thresholding,
Homogeneous Blur, Median Blur, and Morphological Gradient, where it performs
worse in at least one of the measures.

##  6\. Effect of Image Pre-Processing on OCR Performance

While in the previous section we studied the effect of different types of
parameterization on the OCR performance of image processing algorithms, this
section analyzes the effect of each image processing algorithm on the OCR
performance, comparing it to an OCR scenario where no image pre-processing is
used. Due to the significantly better results of the parameterization scenario
that considered the type of digital representation reported in Section 5.3, we
conduct the current evaluation with these parameterization values. We use the
evaluation dataset described in Section 4.2 and conduct our analysis using the
overall set of documents in Section 6.1 and using subsets defined by the type
of digital representation in Section 6.2. We used the Friedman test to analyze
OCR performance differences between the image processing algorithms, and the
one-sided Wilcoxon signed-ranked test with the Bonferroni correction as a
post-hoc test to discover which algorithms perform better in relation to each
other and to not applying any. We used non parametric tests because the
assumptions for repeated measures ANOVA and t-test were not verified. We also
analyze the frequency of OCR errors with different image processing algorithms
on the overall evaluation dataset.

###  6.1. Comparison of Image Pre-processing on the Overall Dataset

This section assesses the results we obtained using the complete evaluation
dataset. We compare the performance of the several image pre-processing
algorithms in Section 6.1.1 and analyze the relation of each algorithm with
OCR error types in Section 6.1.2.

####  6.1.1. Performance Analysis

Table 14 presents the results of the statistical tests showing that, in the
overall dataset, Bilateral Filter and Opening are the best-performing
algorithms. Both algorithms significantly improve the OCR performance in terms
of character accuracy, with the Opening algorithm reaching a more significant
improvement. In contrast, Median Blur and Morphological Gradient are the
lowest-performing algorithms in character accuracy, and Morphological Gradient
is the lowest-performing algorithm in F1-score. Four algorithms worsen the
character accuracy of OCR without image pre-processing: Otsu Thresholding,
Triangle Thresholding, Median Blur, and Morphological Gradient. Similarly,
nine algorithms worsen the OCR performance in terms of F1-score: Adaptive
Thresholding, Otsu Thresholding, Simple Thresholding, Triangle Thresholding,
Gaussian Blur, Homogeneous Blur, Median Blur, Morphological Gradient, and
Black Hat.

Table 14. Summary of Friedman test results and Wilcoxon pairwise comparisons
for post-hoc analysis of OCR performance of image processing algorithms with
the overall dataset. The bold values represent the image processing algorithms
and average results that improved the performance of OCR without image pre-
processing.

|  | Character Accuracy | F1-score
---|---|---|---
Mean | Significantly worse results | # | Mean | Significantly worse results | #
None | N | 66.4 | OT | TT | MB | MG |  |  |  |  |  |  | 4 | 48.8 | AT | OT | ST | TT | GB | HB* | MB | MG | BH* | 9
Adaptive Thresholding | AT | 68.9 | TT | MB | MG |  |  |  |  |  |  |  | 3 | 47.1 | TT | MB | MG |  |  |  |  |  |  | 3
Otsu Thresholding | OT | 65.7 | TT | MB | MG |  |  |  |  |  |  |  | 3 | 45.0 | TT | MB | MG |  |  |  |  |  |  | 3
Simple Thresholding | ST | 65.8 | TT | MB | MG |  |  |  |  |  |  |  | 3 | 47.1 | OT | TT | MB | MG |  |  |  |  |  | 4
Triangle Thresholding | TT | 61.2 | MB | MG |  |  |  |  |  |  |  |  | 2 | 41.1 | MB | MG |  |  |  |  |  |  |  | 2
Bilateral Filter | BF | 68.0 | N* | OT | ST | TT | GB | HB | MB | C* | MG |  | 9 | 49.1 | AT | OT | ST | TT | GB | HB** | MB | BH** | MG | 9
Gaussian Blur | GB | 64.2 | TT | MB | MG |  |  |  |  |  |  |  | 3 | 45.9 | MB | MG | TT |  |  |  |  |  |  | 3
Homogeneous Blur | HB | 64.8 | OT | TT | MB | MG |  |  |  |  |  |  | 4 | 47.0 | OT | TT | GB** | MB | MG |  |  |  |  | 5
Median Blur | MB | 45.7 |  |  |  |  |  |  |  |  |  |  | 0 | 30.3 | MG |  |  |  |  |  |  |  |  | 1
Black Hat | BH | 68.7 | TT | MB | MG |  |  |  |  |  |  |  | 3 | 49.0 | AT | OT | TT | MB | MG |  |  |  |  | 5
Closing | C | 66.8 | OT | TT | GB* | MB | MG |  |  |  |  |  | 5 | 48.7 | AT | OT | ST** | TT | GB | HB* | MB | BH* | MG | 9
Erosion | E | 67.0 | OT | TT | MB | MG |  |  |  |  |  |  | 4 | 48.7 | AT | OT | ST* | TT | GB | MB | MG |  |  | 7
Morphological Gradient | MG | 49.3 |  |  |  |  |  |  |  |  |  |  | 0 | 30.3 |  |  |  |  |  |  |  |  |  | 0
Opening | O | 67.8 | N** | OT | ST** | TT | GB** | HB | MB | C* | E* | MG | 10 | 49.2 | AT | OT | ST | TT | GB | HB | MB | BH** | MG | 9
Top Hat | TH | 67.6 | OT* | TT | MB | MG |  |  |  |  |  |  | 4 | 48.9 | AT | OT | ST* | TT | GB** | MB | MG |  |  | 7
$\chi^{2}$ |  | 905.97 |  |  |  |  |  |  |  |  |  |  |  | 1170.5 |  |  |  |  |  |  |  |  |  |
p-value |  | <2.2e-16 |  |  |  |  |  |  |  |  |  |  |  | <2.2e-16 |  |  |  |  |  |  |  |  |  |

*P¡0.05; **P¡0.01

Looking at the results for the binarization algorithms, the Simple
Thresholding algorithm is the best-performing algorithm. However, it does not
improve the OCR performance. For the smoothing algorithms’ results, Bilateral
Filter is the best-performing smoothing algorithm. Every smoothing algorithm
except Bilateral Filter deteriorates F1-score when used to pre-process the
images before OCR. The results for the morphological transformation algorithms
show that Opening is the best-performing algorithm in comparison to not
applying an image pre-processing algorithm and other binarization, smoothing,
and morphological transformation algorithms.

####  6.1.2. Error Analysis

To better understand how image processing algorithms impact the frequency of
OCR errors, we conducted an OCR error analysis of the most common errors for
each image processing algorithm based on edit operation types. We present the
frequency of edit operations for each image processing algorithm in Figure 3.
The first graph bar in each edit operation labeled “None” corresponds to
performing OCR without image pre-processing.

Figure 3. Frequency of edit operations for image processing algorithms in OCR.
The operations of deletion, insertion and substitution are designated as del,
ins and sub, and the combinations between them as del+ins, del+sub, ins+sub
and del+ins+sub.

The results show the frequency of edit operations (deletion, insertion, and
substitution) and combinations between them with each image processing
algorithm. We conclude that among single modification error types, the
operation of deletion is the most common, and insertion is the least common.
Among the operation of deletion, Median Blur and Triangle Thresholding have
the highest occurrences. We can also observe that the algorithms Adaptive
Thresholding, Bilateral Filtering, Black Hat, Dilation, Opening, and Top Hat
had a lower frequency of deletion operations in comparison to not applying any
image processing algorithm to OCR. As for the combination of edit operation
types where multiple errors occur, we conclude that deletion and insertion,
along with deletion, insertion, and substitution have a very small occurrence
with all the algorithms. The algorithms Adaptive Thresholding and Black Hat
had a higher frequency of the operation of substitution combined with deletion
and insertion in comparison to not using image processing algorithms.

###  6.2. Comparison of Image Pre-processing on the Datasets Defined by Type
of Digital Representation

This section presents the results we obtained on algorithms comparison using
subsets of the evaluation dataset defined by the type of digital
representation.

####  6.2.1. Letters

This section presents the results regarding the letters’ typology. Table 15
presents the results of the statistical tests showing that, similarly to the
results for the overall dataset, Bilateral Filter is the best-performing
algorithm. Triangle Thresholding and Morphological Gradient are the worst-
performing algorithms. Moreover, the mean results of character accuracy and
F1-score are higher than in the overall dataset.

Table 15. Summary of Friedman test results and Wilcoxon pairwise comparisons
for post-hoc analysis of OCR performance of image processing algorithms with
the letters sample. The bold values represent the image processing algorithms
and average results that improved the OCR performance.

|  | Character Accuracy | F1-score
---|---|---|---
Mean | Significantly worse results | # | Mean | Significantly worse results | #
None | N | 77.4 | OT | ST** | TT | MB | MG |  |  |  |  |  | 5 | 60.1 | AT | OT | ST | TT | GB** | MB | MG | 7
Adaptive Thresholding | AT | 80.1 | TT | MB* | MG |  |  |  |  |  |  |  | 3 | 57.6 | TT | MG |  |  |  |  |  | 2
Otsu Thresholding | OT | 75.5 | TT | MG |  |  |  |  |  |  |  |  | 2 | 52.7 | TT | MG |  |  |  |  |  | 2
Simple Thresholding | ST | 74.4 | TT | MG |  |  |  |  |  |  |  |  | 2 | 55.3 | OT | TT | MG |  |  |  |  | 3
Triangle Thresholding | TT | 63.3 |  |  |  |  |  |  |  |  |  |  | 0 | 37.7 |  |  |  |  |  |  |  | 0
Bilateral Filter | BF | 78.6 | N* | OT | ST | TT | HB* | MB | C* | E* | MG | O* | 10 | 60.2 | AT | OT | ST | TT | GB** | MB | MG | 7
Gaussian Blur | GB | 74.3 | TT | MB* | MG |  |  |  |  |  |  |  | 3 | 60.1 | OT | TT | MB | MG |  |  |  | 4
Homogeneous Blur | HB | 77.4 | OT | ST** | TT | MB | MG |  |  |  |  |  | 5 | 60.1 | AT | OT | ST | TT | GB** | MB | MG | 7
Median Blur | MB | 71.5 | MG* | TT |  |  |  |  |  |  |  |  | 2 | 54.2 | MG | TT |  |  |  |  |  | 2
Black Hat | BH | 79.7 | TT | MB | MG |  |  |  |  |  |  |  | 3 | 60.7 | AT | OT | TT | MB** | MG |  |  | 5
Closing | C | 77.4 | OT | ST** | TT | MB | MG |  |  |  |  |  | 5 | 60.1 | AT | OT | ST | TT | GB** | MB | MG | 7
Erosion | E | 77.4 | OT | ST** | TT | MB | MG |  |  |  |  |  | 5 | 60.1 | AT | OT | ST | TT | GB** | MB | MG | 7
Morphological Gradient | MG | 66.0 |  |  |  |  |  |  |  |  |  |  | 0 | 36.6 |  |  |  |  |  |  |  | 0
Opening | O | 77.4 | OT | ST** | TT | MB | MG |  |  |  |  |  | 5 | 60.1 | AT | OT | ST | TT | GB** | MB | MG | 7
Top Hat | TH | 77.6 | TT | MG |  |  |  |  |  |  |  |  | 2 | 59.2 | AT* | OT | TT | MB* | MG |  |  | 5
$\chi^{2}$ |  | 223.76 |  |  |  |  |  |  |  |  |  |  |  | 395.56 |  |  |  |  |  |  |  |
p-value |  | <2.2e-16 |  |  |  |  |  |  |  |  |  |  |  | <2.2e-16 |  |  |  |  |  |  |  |

*P¡0.05; **P¡0.01

For the binarization methods’ results, all the algorithms, except Adaptive
Thresholding, decrease character accuracy. The results for F1-score are akin
to the overall dataset. As for the results of the smoothing methods, Bilateral
Filter is still the best-performing algorithm in character accuracy. However,
the results for the morphological transformation algorithms indicate that, in
comparison to the overall dataset, Opening no longer improves the OCR
performance.

####  6.2.2. Non-structured reports

This section presents the results regarding the non-structured reports
typology. Table 16 presents the results of the statistical tests showing that
the mean results of the OCR performance are slightly higher than in the
overall dataset and no algorithm improves the OCR performance. Black Hat and
Opening are the best-performing algorithms and Homogeneous Blur, Median Blur,
and Morphological Gradient are the worst-performing algorithms.

Table 16. Summary of Friedman test results and Wilcoxon pairwise comparisons
for post-hoc analysis of OCR performance of image processing algorithms with
the non-structured reports sample. The bold values represent the average
results that improved the OCR performance.

|  | Character Accuracy | F1-score
---|---|---|---
|  | Mean | Significantly worse results | # | Mean | Significantly worse results | #
None | N | 69.2 | OT | GB | HB | MB | MG |  |  |  | 5 | 53.8 | OT | TT* | GB | HB | MB | MG |  | 6
Adaptive Thresholding | AT | 75.8 | TT | GB* | HB* | MB** | MG |  |  |  | 5 | 53.7 | TT | MG |  |  |  |  |  | 2
Otsu Thresholding | OT | 66.2 | GB** | HB** | MB** | MG |  |  |  |  | 4 | 44.0 | MG |  |  |  |  |  |  | 1
Simple Thresholding | ST | 68.8 | OT | GB | HB | MB | MG |  |  |  | 5 | 53.1 | OT | TT* | GB | HB | MB | MG |  | 6
Triangle Thresholding | TT | 67.9 | MG |  |  |  |  |  |  |  | 1 | 46.5 | MG |  |  |  |  |  |  | 1
Bilateral Filter | BF | 69.3 | GB | HB | MB | MG |  |  |  |  | 5 | 53.8 | OT | TT* | GB | HB | MB | MG |  | 6
Gaussian Blur | GB | 58.0 | HB* |  |  |  |  |  |  |  | 1 | 44.3 | MG |  |  |  |  |  |  | 1
Homogeneous Blur | HB | 52.4 |  |  |  |  |  |  |  |  | 0 | 41.1 | MG |  |  |  |  |  |  | 1
Median Blur | MB | 57.5 |  |  |  |  |  |  |  |  | 0 | 42.2 | MG |  |  |  |  |  |  | 1
Black Hat | BH | 79.1 | OT | TT | GB | HB | MB | E* | MG | TH | 8 | 58.6 | AT** | OT | TT | GB | HB | MB | MG | 7
Closing | C | 69.2 | OT | GB | HB | MB | MG |  |  |  | 5 | 53.8 | OT | TT* | GB | HB | MB | MG |  | 6
Dilation | D | 69.2 | OT | GB | HB | MB | MG |  |  |  | 5 | 53.8 | OT | TT* | GB | HB | MB | MG |  | 6
Erosion | E | 70.7 | GB** | HB** | MB** | MG |  |  |  |  | 4 | 54.1 | OT | TT** | GB* | HB** | MB | MG |  | 6
Morphological Gradient | MG | 48.9 |  |  |  |  |  |  |  |  | 0 | 22.6 |  |  |  |  |  |  |  | 0
Opening | O | 72.8 | OT** | TT* | BF* | GB | HB | MB | MG |  | 7 | 56.0 | OT | TT** | GB | HB | MB | MG |  | 6
Top Hat | TH | 71.3 | GB* | HB** | MB* | MG |  |  |  |  | 4 | 55.4 | OT | TT | GB | HB | MB | MG |  | 6
$\chi^{2}$ |  | 161.69 |  |  |  |  |  |  |  |  |  | 199.79 |  |  |  |  |  |  |  |
p-value |  | <2.2e-16 |  |  |  |  |  |  |  |  |  | <2.2e-16 |  |  |  |  |  |  |  |

*P¡0.05; **P¡0.01

The binarization methods Otsu and Triangle Thresholding are the worst-
performing algorithms as they deteriorate the OCR performance. As for the
smoothing algorithms’ results, Bilateral Filter is the best-performing
algorithm. All the smoothing algorithms except Bilateral Filter have
significantly worse results than not using any image processing algorithm. The
results for the morphological transformation methods indicate that Black Hat
and Opening are the best-performing algorithms. In sum, most morphological
transformation algorithms improve the OCR performance in comparison to
binarization and smoothing algorithms in the non-structured reports typology.

####  6.2.3. Process covers

This section presents the results regarding the process covers’ typology.
Table 17 presents the results of the statistical tests showing that, in the
process covers sample, the mean OCR performance values are lower than in the
overall dataset, and no algorithm improved the OCR performance.

Table 17. Summary of Friedman test results and Wilcoxon pairwise comparisons
for post-hoc analysis of OCR performance of image processing algorithms with
the process covers sample. The bold values represent the average results that
improved the OCR performance.

|  | Character Accuracy | F1-score
---|---|---|---
Mean | Significantly worse results | # | Mean | Significantly worse results | #
None | N | 62.4 | MG |  |  |  | 1 | 35.3 | BH** | MG |  |  | 2
Adaptive Thresholding | AT | 63.1 | MG |  |  |  | 1 | 33.0 | MG |  |  |  | 1
Otsu Thresholding | OT | 61.6 | MG |  |  |  | 1 | 31.5 | MG |  |  |  | 1
Simple Thresholding | ST | 64.5 | BH* | MG |  |  | 2 | 36.5 | BH** | MG |  |  | 2
Triangle Thresholding | TT | 56.3 | MG* |  |  |  | 1 | 32.8 | MG |  |  |  | 1
Bilateral Filter | BF | 67.1 | BH** | MG |  |  | 2 | 38.9 | OT** | BH | MG |  | 3
Gaussian Blur | GB | 67.1 | OT** | BH | MG | TH* | 4 | 39.2 | OT** | TT* | BH** | MG | 4
Homogeneous Blur | HB | 67.4 | OT* | TT* | BH | MG | 4 | 38.8 | OT** | BH** | MG |  | 3
Median Blur | MB | 65.8 | BH* | MG |  |  | 2 | 39.3 | OT | BH** | MG |  | 3
Black Hat | BH | 54.6 | MG |  |  |  | 1 | 28.0 | MG |  |  |  | 1
Closing | C | 62.4 | MG |  |  |  | 1 | 35.3 | BH** | MG |  |  | 2
Dilation | D | 62.4 | MG |  |  |  | 1 | 35.3 |  |  |  |  | 0
Erosion | E | 62.4 | MG |  |  |  | 1 | 35.3 | BH** | MG |  |  | 2
Morphological Gradient | MG | 34.9 |  |  |  |  | 0 | 11.4 |  |  |  |  | 0
Opening | O | 65.1 | BH** | MG |  |  | 2 | 36.1 | OT* | BH** | MG |  | 3
Top Hat | TH | 62.7 | MG |  |  |  | 1 | 35.6 | MG |  |  |  | 1
$\chi^{2}$ |  | 104.27 |  |  |  |  |  | 126.29 |  |  |  |  |
p-value |  | 7.183e-16 |  |  |  |  |  | <2.2e-16 |  |  |  |  |

*P¡0.05; **P¡0.01

There are no significant differences between the binarization algorithms.
Moreover, unlike in the previous sections, binarization algorithms do not
impair the OCR performance. As for the smoothing methods, we can observe that
most algorithms have a better performance in character accuracy and F1-score
than Black Hat, Morphological Gradient, and Otsu Thresholding. Finally, the
results with morphological transformation algorithms indicate that Black Hat
and Morphological Gradient are the lowest-performing algorithms and
deteriorate the OCR performance.

####  6.2.4. Structured reports

This section presents the results regarding the structured reports typology.
Table 18 presents the results of the statistical tests showing that no
algorithm improved the OCR performance in comparison to not using any image
processing algorithm.

Table 18. Summary of Friedman test results and Wilcoxon pairwise comparisons
for post-hoc analysis of OCR performance of image processing algorithms with
the structured reports sample. The bold values represent the average results
that improved the OCR performance.

|  | Character Accuracy | F1-score
---|---|---|---
Mean | Significantly worse results | # | Mean | Significantly worse results | #
None | N | 69.7 | OT | MB | MG |  |  |  | 3 | 50.1 | AT | GB | MB | BH** | MG | 5
Adaptive Thresholding | AT | 68.5 | MB | MG |  |  |  |  | 2 | 47.2 | MB | MG |  |  |  | 2
Otsu Thresholding | OT | 69.4 | MB | MG |  |  |  |  | 2 | 49.6 | AT | MB | MG |  |  | 3
Simple Thresholding | ST | 69.0 | MB | MG |  |  |  |  | 2 | 49.1 | AT | MB | MG |  |  | 3
Triangle Thresholding | TT | 65.4 | MB | MG |  |  |  |  | 2 | 47.2 | MB | MG |  |  |  | 2
Bilateral Filter | BF | 70.7 | AT | ST* | TT** | GB** | MB | BH* | 6 | 50.3 | AT | GB | MB | BH** | MG | 5
Gaussian Blur | GB | 68.3 | MB | MG |  |  |  |  | 2 | 48.0 | MB | MG |  |  |  | 2
Homogeneous Blur | HB | 69.7 | OT** | MB | MG |  |  |  | 3 | 50.1 | AT | GB | MB | BH** | MG | 5
Median Blur | MB | 30.0 |  |  |  |  |  |  | 0 | 16.7 | MG |  |  |  |  | 1
Black Hat | BH | 69.5 | MB | MG |  |  |  |  | 2 | 49.0 | AT** | MB | MG |  |  | 3
Closing | C | 69.7 | OT** | MB | MG |  |  |  | 3 | 50.1 | AT | GB | MB | BH** | MG | 5
Dilation | D | 69.7 | OT** | MB | MG |  |  |  | 3 | 50.1 | AT | GB | MB | BH** | MG | 5
Erosion | E | 69.7 | OT** | MB | MG |  |  |  | 3 | 50.1 | AT | GB | MB | BH** | MG | 5
Morphological Gradient | MG | 52.7 | MB |  |  |  |  |  | 1 | 23.3 | MB |  |  |  |  | 1
Opening | O | 69.7 | OT** | MB | MG |  |  |  | 3 | 50.1 | AT | GB | MB | BH** | MG | 5
Top Hat | TH | 69.5 | MB | MG |  |  |  |  | 2 | 49.8 | AT | GB* | MB | MG |  | 4
$\chi^{2}$ |  | 746.29 |  |  |  |  |  |  |  | 888.56 |  |  |  |  |  |
p-value |  | <2.2e-16 |  |  |  |  |  |  |  | <2.2e-16 |  |  |  |  |  |

*P¡0.05; **P¡0.01

We can observe that from the performance results of binarization methods,
Adaptive Thresholding and Otsu Thresholding impair the OCR performance. For
the smoothing algorithms’ results, we see that Bilateral Filter is the best-
performing algorithm. However, it does not improve the OCR performance. On the
other hand, Median Blur and Gaussian Blur worsen the OCR performance. The
results for morphological transformation algorithms indicate that
Morphological Gradient and Black Hat deteriorate the performance.

####  6.2.5. Theatre plays’ covers

This section presents the results regarding the theatre plays’ covers
typology. Table 19 presents the results of the statistical tests showing a
significantly lower mean OCR performance than in the overall dataset.
Furthermore, unlike the results in the overall dataset and the previous
typologies where binarization algorithms worsened the OCR performance, in this
sample it is a binarization algorithm, Adaptive Thresholding, that is the
best-performing algorithm. In sum, three algorithms improved character
accuracy: Adaptive Thresholding, Black Hat, and Top Hat.

Table 19. Summary of Friedman test results and Wilcoxon pairwise comparisons
for post-hoc analysis of OCR performance of image processing algorithms with
the theatre plays’ covers sample. The bold values represent the image
processing algorithms and average results that improved the OCR performance.

|  | Character Accuracy | F1-score
---|---|---|---
Mean | Significantly worse results | # | Mean | Significantly worse results | #
None | N | 18.2 | MG |  |  |  |  |  |  |  |  | 1 | 16.6 | MG |  |  | 1
Adaptive Thresholding | AT | 35.6 | N | OT* | ST* | BF* | HB* | C* | D* | E* | MG | 9 | 21.8 | HB* | MB* | MG | 3
Otsu Thresholding | OT | 22.6 | MG |  |  |  |  |  |  |  |  | 1 | 14.1 | MG |  |  | 1
Simple Thresholding | ST | 21.4 | MG |  |  |  |  |  |  |  |  | 1 | 14.6 | MG |  |  | 1
Triangle Thresholding | TT | 26.7 | MG |  |  |  |  |  |  |  |  | 1 | 16.0 | MG |  |  | 1
Bilateral Filter | BF | 23.4 | MG |  |  |  |  |  |  |  |  | 1 | 16.1 | MG |  |  | 1
Gaussian Blur | GB | 21.4 | MG |  |  |  |  |  |  |  |  | 1 | 14.0 | MG |  |  | 1
Homogeneous Blur | HB | 21.2 | MG |  |  |  |  |  |  |  |  | 1 | 12.3 | MG |  |  | 1
Median Blur | MB | 22.8 | MG |  |  |  |  |  |  |  |  | 1 | 11.8 | MG |  |  | 1
Black Hat | BH | 31.8 | N* | MG |  |  |  |  |  |  |  | 2 | 22.2 | MG |  |  | 1
Closing | C | 22.4 | MG |  |  |  |  |  |  |  |  | 1 | 14.7 | MG |  |  | 1
Dilation | D | 22.4 | MG |  |  |  |  |  |  |  |  | 1 | 14.7 | MG |  |  | 1
Erosion | E | 22.4 | MG |  |  |  |  |  |  |  |  | 1 | 14.7 | MG |  |  | 1
Morphological Gradient | MG | 0.47 |  |  |  |  |  |  |  |  |  | 0 | 0 |  |  |  | 0
Opening | O | 25.7 | MG |  |  |  |  |  |  |  |  | 1 | 16.3 | MG |  |  | 1
Top Hat | TH | 29.3 | N* | MG |  |  |  |  |  |  |  | 2 | 18.9 | MG |  |  | 1
$\chi^{2}$ |  | 92.122 |  |  |  |  |  |  |  |  |  |  | 81.827 |  |  |  |
p-value |  | 1.506e-13 |  |  |  |  |  |  |  |  |  |  | 1.295e-11 |  |  |  |

*P¡0.05; **P¡0.01

###  6.3. OCR Performance Comparison Across Typologies and Overall

We present the distribution of the performance measures for the overall
dataset and the digital representation-type samples in Figure 4. We conclude
that the digital representation typology with the best results is the letters
sample, with a median character accuracy above 75% and a median F1-score above
60% for most algorithms. In contrast, the digital representation typology with
the worst performance results is the theatre plays’ covers sample, with a
median character accuracy below 25% and a median F1-score below 20%. Image
processing algorithms appear to have a greater impact on the OCR performance
of digital representations from the theatre plays’ covers sample, where the
OCR performance without applying image processing algorithms is low.

Figure 4. Distribution of OCR performance measures in the overall set of
documents, and by letters, non-structured reports, process covers, structured
reports, and theatre plays’ covers. The first row of results refers to the
character accuracy performance measure, and the second row refers to the
F1-score performance measure.

##  7\. Discussion

Our results from Sections 5.2 and 5.3 show that parameterization with the
NSGA-II algorithm influences the impact of image processing algorithms on OCR,
in response to RQ1. Not only are the parameter values very different with
parameterization, but the results are significantly better compared to using
default parameter values. We conclude that some image processing algorithms
perform significantly better with parameterization by typology than with
global parameterization. Moreover, parameterization by typology has the
advantage of being faster to process than global parameterization.

From Section 6.1, we can draw conclusions to answer RQ2. We conclude that
there are no significant improvements in the OCR performance in the overall
dataset when using binarization algorithms in an image pre-processing OCR
phase in comparison to not using them. Unlike the binarization method, the
smoothing and morphological transformation methods significantly improved the
OCR performance with Bilateral Filter and Opening, respectively. Table 20
shows the image processing algorithms we chose to apply as image pre-
processing to OCR from the results we described in Sections 6.1.1 and 6.2. In
the column “Improved algorithms” of Table 20, we list the image processing
algorithms that significantly improved the OCR performance without image pre-
processing. From these algorithms, we selected the ones that had the higher
significance levels for the overall dataset and each digital representation
typology. The two last columns of Table 20 show the mean performance values of
the selected algorithms. The OCR type error analysis in Section 6.1.2 lead us
to conclude that among single modification error types, the operation of
deletion is the most common, and insertion is the least common. Moreover,
among the combination of edit operation types, the operations of deletion and
insertion, along with deletion, insertion, and substitution have a very small
occurrence. We observe that image processing algorithms that performed the
best in the overall dataset had a lower occurrence of deletion edit
operations, and those that performed the worst had the highest occurrence.
This might signify that 1) applying image pre-processing to OCR has a bigger
impact in mitigating the error of missing text in comparison to other errors,
and 2) improving the lack of text recognition has a bigger positive influence
in improving the OCR performance than improving other errors.

Table 20. Image processing algorithms that improved the OCR performance without image pre-processing per digital representation typology and overall and were selected as the most significant. The mean OCR performance values refer to the selected image pre-processing algorithm. Typology | Improved algorithms | Selected algorithm |  | Mean character
---
accuracy
| Mean
---
F1-score
Overall | Bilateral Filter and Opening | Opening | 67.8% | 49.2%
Letters | Bilateral Filter | Bilateral Filter | 78.6% | 60.2%
| Non-structured reports
---
None | None | 69.2% | 53.8%
Process covers | None | None | 62.4% | 35.3%
Structured reports | None | None | 69.7% | 50.1%
| Theatre plays’ covers
---
| Adaptive Threshold, Black Hat
---
and Top Hat
Adaptive Threshold | 35.6% | 21.8%

Results from Section 6.2 allow us to answer RQ3. The analysis shows that it is
possible to improve the text recognition task using tuned image processing
algorithms in the subsets of letters and theatre plays’ covers. As shown in
the column “Improved algorithms” of Table 20, Bilateral Filter significantly
improved the OCR performance with image pre-processing with the subset of
letters, and Adaptive Threshold, Black Hat, and Top Hat improved the
performance with the subset of theatre plays’ covers. However, no image
processing algorithms significantly improved the OCR performance with the
samples of all the other typologies. From the column “Selected algorithm” of
Table 20, we can see that the algorithm we chose for the typology of letters
is Bilateral Filter, and for theatre plays’ covers it is Adaptive Threshold.
From the performance comparison in Section 6.3, we conclude that process
covers and theatre plays’ covers are the samples with the lowest mean OCR
performance values. They are also the samples with the most algorithms that
tend to improve the OCR performance or do so in a significant way. These
findings suggest that image processing algorithms tend to have a greater
impact on digital representation typologies that have a lower OCR performance
without an image pre-processing phase. The results also show that
morphological transformation algorithms tend to be the most beneficial to
every typology. In contrast, binarization algorithms only benefit the theatre
plays’ covers typology and tend to do so for the sample of process covers. On
the other hand, while smoothing algorithms tend to be beneficial to every
typology, like morphological transformation algorithms, only one algorithm,
Bilateral Filter, is responsible for that performance improvement.

##  8\. Conclusion

The main goal of this work is to understand how to improve the OCR performance
on cultural heritage digital representations using image processing
algorithms. We evaluated the use and parameterization of image processing
algorithms for text recognition tasks using Tesseract and OpenCV methods.

The key takeaway is that parameterization by digital representation typology
benefits the employment of image pre-processing techniques in OCR, even if
only for a small parameterization dataset. We successfully applied the NSGA-II
algorithm to perform the parameterization. The analysis of the performance of
image processing algorithms leads us to conclude that some algorithms improve
the OCR performance significantly, while several tend to improve it and it is
up to debate whether to apply them or not, and others even worsen the
performance. A previous analysis is required before applying them. On top of
that, some algorithms have more impact in some samples of the evaluation
dataset than others. Digital representation typologies influence the effect
these image processing algorithms have on the OCR performance. For this
reason, it is important to analyze typologies before applying OCR. A
binarization algorithm positively impacted the OCR performance in the sample
of theatre plays’ covers. On the other hand, smoothing algorithms improved the
performance of the overall dataset and the samples of letters and theatre
plays’ covers. Likewise, morphological transformation algorithms also improved
the OCR performance of the overall dataset and the samples of theatre plays’
covers. These findings suggest that the employment of image pre-processing
algorithms in OCR might be more suitable for typologies where the text
recognition task without pre-processing does not produce good results, as is
the case of the typology for theatre plays’ covers. Factually, the algorithms
that have been proven to improve the OCR task compared to not applying any are
Adaptive Thresholding, Bilateral Filter, Black Hat, Opening, and Top Hat.
Ultimately, there are big differences in the OCR performance by digital
representation typology, which suggest that we should adopt the optimization
of the OCR task for typologies that have weaker results without employing
image pre-processing algorithms.

The practical outcomes of this work are the source code of the
parameterization of image processing algorithms and two datasets. Both
datasets are available to consult for other scholars or researchers to use.
Likewise, we provide the source code of the parameterization of image
processing algorithms using NSGA-II.

In the future, we will analyze the effect of the combination of image
processing algorithms in the text recognition task. We would also like to
expand the current OCR error analysis to include other error types.
Furthermore, we would like to explore the detection and removal of stamps and
annotations in digital representations in the OCR performance.

###### Acknowledgements.

This work is financed by National Funds through FCT - Foundation for Science
and Technology I.P., within the scope of the EPISA project -
DSAIPA/DS/0023/2018.

## References

  * (1)
  * Antonacopoulos et al. (2013) Apostolos Antonacopoulos, Christian Clausner, Christos Papadopoulos, and Stefan Pletschacher. 2013\.  ICDAR 2013 Competition on Historical Book Recognition (HBR 2013). In _Proceedings of the 2013 12th International Conference on Document Analysis and Recognition_ _(ICDAR ’13)_. IEEE Computer Society, USA, 1459–1463.  https://doi.org/10.1109/ICDAR.2013.294
  * Baierer (2020) Konstantin Baierer. 2020\.  Models · OCROPUS/ocropy wiki.  https://github.com/ocropus/ocropy/wiki/Models
  * Bizer et al. (2009) Christian Bizer, Tom Heath, and Tim Berners-Lee. 2009\.  Linked Data: The Story so Far.  _International Journal on Semantic Web and Information Systems_ 5 (07 2009), 1–22.  https://doi.org/10.4018/jswis.2009081901
  * Blank and Deb (2020) Julian Blank and Kalyan Deb. 2020.  Pymoo: Multi-Objective Optimization in Python.  _IEEE Access_ PP (04 2020), 1–1.  https://doi.org/10.1109/ACCESS.2020.2990567
  * Bromage (2016) David Bromage. 2016\.  Linked Data for Libraries, Archives and Museums: How to Clean, Link and Publish Your Metadata.  _Archives and Manuscripts_ 44, 3 (Oct. 2016), 172–174.  https://doi.org/10.1080/01576895.2016.1233606
  * Bui et al. (2017) Quang Anh Bui, David Mollard, and Salvatore Tabbone. 2017\.  Selecting Automatically Pre-Processing Methods to Improve OCR Performances. In _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , Vol. 01. IEEE Computer Society, Kyoto, Japan, 169–174.  https://doi.org/10.1109/ICDAR.2017.36
  * Carrasco (2014) Rafael C. Carrasco. 2014\.  An Open-Source OCR Evaluation Tool. In _Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage_ (Madrid, Spain) _(DATeCH ’14)_. Association for Computing Machinery, New York, NY, USA, 179–184.  https://doi.org/10.1145/2595188.2595221
  * Clausner et al. (2013) Christian Clausner, Stefan Pletschacher, and Apostolos Antonacopoulos. 2013\.  The Significance of Reading Order in Document Recognition and Its Evaluation. In _2013 12th International Conference on Document Analysis and Recognition_. IEEE Computer Society, Washington, DC, USA, 688–692.  https://doi.org/10.1109/ICDAR.2013.141
  * Clausner et al. (2020) Christian Clausner, Stefan Pletschacher, and Apostolos Antonacopoulos. 2020\.  Flexible character accuracy measure for reading-order-independent evaluation.  _Pattern Recognition Letters_ 131 (2020), 390–397.  https://doi.org/10.1016/j.patrec.2020.02.003
  * DGARQ (2008) DGARQ. 2008.  Arquivo Nacional Torre do Tombo.  https://digitarq.arquivos.pt/
  * Dias (2022) Mariana Dias. 2022\.  Typewritten Digital Representations of Portuguese Cultural Heritage Documents from the 20th century.  Data set.  https://doi.org/10.25747/ZC25-1531
  * Dias and Lopes (2022) Mariana Dias and Carla Teixeira Lopes. 2022.  Mining Typewritten Digital Representations to Support Archival Description. In _Proceedings of the 26th International Conference on Theory and Practice of Digital Libraries \- Workshops and Doctoral Consortium_ _(CEUR Workshop Proceedings, Vol. 3246)_ , Leonardo Candela and Gianmaria Silvello (Eds.). CEUR-WS.org, Padua, Italy, 70–76.  https://ceur-ws.org/Vol-3246/09_Paper2.pdf
  * dos Arquivos Nacionais/Torre do Tombo (2006) Instituto dos Arquivos Nacionais/Torre do Tombo. 2006.  Orientações para a gestão de documentos de Arquivo no contexto de uma reestruturação da Administração Central do Estado.  , 55 pages.
  * Drobac and Lindén (2020) Senka Drobac and Krister Lindén. 2020.  Optical character recognition with neural networks and post-correction with finite state methods.  _International Journal on Document Analysis and Recognition (IJDAR)_ 23 (12 2020), 1–17.  https://doi.org/10.1007/s10032-020-00359-9
  * Falcão (2022) Margarida Falcão. 2022\.  Manual Transcriptions of Typewritten Digital Representations of Portuguese Cultural Heritage Documents from the 20th Century.  Data set.  https://doi.org/10.25747/WPNA-JE39
  * FineReader PDF (2019) FineReader PDF. 2019\.  How AI powers PDF Software & Technology Trends: FineReader Blog.  https://pdf.abbyy.com/blog/finereader-powered-by-ai/
  * FineReader PDF (2021) FineReader PDF. 2021\.  Technical Specifications and System Requirements — FineReader PDF.  https://pdf.abbyy.com/specifications/
  * Fontan et al. (2016) Lionel Fontan, Isabelle Ferrané, Jérôme Farinas, Julien Pinquier, and Xavier Aumont. 2016\.  Using Phonologically Weighted Levenshtein Distances for the Prediction of Microscopic Intelligibility. In _Interspeech 2016, 17th Annual Conference of the International Speech Communication Association_. ISCA, San Francisco, CA, United States, 650–654.  https://doi.org/10.21437/Interspeech.2016-431
  * Gao et al. (2019) Yayu Gao, Xinmin Zhang, Xiaoyou Zhang, Duan Li, Min Yang, and Jinhua Tian. 2019\.  Application of NSGA-II and Improved Risk Decision Method for Integrated Water Resources Management of Malian River Basin.  _Water_ 11, 8 (2019), 27 pages.  https://doi.org/10.3390/w11081650
  * Gupta et al. (2007) Maya Gupta, Nathaniel Jacobson, and Eric Garcia. 2007\.  OCR binarization and image pre-processing for searching historical documents.  _Pattern Recognition_ 40 (02 2007), 389–397.  https://doi.org/10.1016/j.patcog.2006.04.043
  * Harraj and Raissouni (2015) Abdeslam Harraj and Naoufal Raissouni. 2015.  OCR Accuracy Improvement on Document Images Through a Novel Pre-Processing Approach.  _Signal & Image Processing : An International Journal_ 6 (09 2015).  https://doi.org/10.5121/sipij.2015.6401
  * Kiss et al. (2019) Martin Kiss, Michal Hradis, and Oldrich Kodym. 2019\.  Brno Mobile OCR Dataset.  _CoRR_ abs/1907.01307 (2019), 6 pages.  http://arxiv.org/abs/1907.01307
  * Koistinen et al. (2017) Mika Koistinen, Kimmo Kettunen, and Tuula Pääkkönen. 2017\.  Improving Optical Character Recognition of Finnish Historical Newspapers with a Combination of Fraktur & Antiqua Models and Image Preprocessing. In _Proceedings of the 21st Nordic Conference on Computational Linguistics_. Association for Computational Linguistics, Gothenburg, Sweden, 277–283.  https://aclanthology.org/W17-0238
  * Kotthoff (2016) Lars Kotthoff. 2016\.  _Algorithm Selection for Combinatorial Search Problems: A Survey_.  Springer International Publishing, Cham, 149–190.  https://doi.org/10.1007/978-3-319-50137-6_7
  * Kumar et al. (2013) Gaurav Kumar, Pradeep Bhatia, and Indu. 2013.  Analytical Review of Preprocessing Techniques for Offline Handwritten Character Recognition.  _International Journal of Advances in Engineering Sciences_ 3 (07 2013), 14–22.  https://doi.org/10.13140/RG.2.1.3896.7842
  * Lund et al. (2013) William Lund, Douglas Kennard, and Eric Ringger. 2013\.  Combining Multiple Thresholding Binarization Values to Improve OCR Output. In _Document Recognition and Retrieval XX_ , Vol. 8658. SPIE, Burlingame, California, USA.  https://doi.org/10.1117/12.2006228
  * Lyu et al. (2021) Lijun Lyu, Maria Koutraki, Martin Krickl, and Besnik Fetahu. 2021.  Neural OCR Post-Hoc Correction of Historical Corpora.  _CoRR_ abs/2102.00583 (2021), 15 pages.  https://arxiv.org/abs/2102.00583
  * Mohamad (2015) Mumtazimah Mohamad. 2015\.  A Review on OpenCV.  https://doi.org/10.13140/RG.2.1.2269.8721
  * Nebro et al. (2022) Antonio J. Nebro, Jesús Galeano-Brajones, Francisco Luna, and Carlos A. Coello Coello. 2022.  Is NSGA-II Ready for Large-Scale Multi-Objective Optimization?  _Mathematical and Computational Applications_ 27, 6 (2022), 17 pages.  https://doi.org/10.3390/mca27060103
  * Nguyen et al. (2020) Thi-Tuyet-Hai Nguyen, Adam Jatowt, Mickael Coustaty, Nhu-Van Nguyen, and Antoine Doucet. 2020\.  Deep Statistical Analysis of OCR Errors for Effective Post-OCR Processing. In _2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_ _(JCDL ’19)_. IEEE Press, Champaign, Illinois, USA, 29–38.  https://doi.org/10.1109/JCDL.2019.00015
  * Niu (2016) Jinfang Niu. 2016\.  Linked Data for Archives.  _Archivaria_ 82 (12 2016), 83–110.  https://archivaria.ca/index.php/archivaria/article/view/13582
  * Noether (1992) Gottfried E. Noether. 1992\.  _Introduction to Wilcoxon (1945) Individual Comparisons by Ranking Methods_.  Springer New York, New York, NY, 191–195.  https://doi.org/10.1007/978-1-4612-4380-9_15
  * Ntogas and Ventzas (2009) Nikolaos Ntogas and Dimitrios Ventzas. 2009.  Binarization of pre-filtered historical manuscripts images.  _International Journal of Intelligent Computing and Cybernetics_ 2 (03 2009), 148–174.  https://doi.org/10.1108/17563780910939282
  * Pfaffe et al. (2017) Philip Pfaffe, Martin Tillmann, Sigmar Walter, and Walter Tichy. 2017.  Online-Autotuning in the Presence of Algorithmic Choice. In _2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)_. IEEE Computer Society, Lake Buena Vista, FL, 1379–1388.  https://doi.org/10.1109/IPDPSW.2017.28
  * Pletschacher et al. (2015) Stefan Pletschacher, Christian Clausner, and Apostolos Antonacopoulos. 2015.  Europeana Newspapers OCR Workflow Evaluation. In _Proceedings of the 3rd International Workshop on Historical Document Imaging and Processing_ _(HIP ’15)_. Association for Computing Machinery, New York, NY, USA, 39–46.  https://doi.org/10.1145/2809544.2809554
  * Probst et al. (2019) Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. 2019\.  Tunability: Importance of Hyperparameters of Machine Learning Algorithms.  _J. Mach. Learn. Res._ 20, 1 (jan 2019), 1934–1965.
  * Roetzel et al. (2020) Wilfried Roetzel, Xing Luo, and Dezhen Chen. 2020.  _Chapter 6 - Optimal design of heat exchanger networks_.  Academic Press, 231–317.
  * Sporici et al. (2020) Dan Sporici, Elena Cușnir, and Costin-Anton Boiangiu. 2020\.  Improving the Accuracy of Tesseract 4.0 OCR Engine Using Convolution-Based Preprocessing.  _Symmetry_ 12 (05 2020), 715.  https://doi.org/10.3390/sym12050715
  * Su et al. (2021) Zhidong Su, Weihua Sheng, and Senlin Zhang. 2021.  Convolutional Neural Network Optimization Using Modified NSGA-II. In _2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER)_. IEEE Computer Society, 466–471.  https://doi.org/10.1109/CYBER53097.2021.9588230
  * Sulaiman et al. (2019) Alaa Sulaiman, Khairuddin Omar, and Mohammad Faidzul Nasrudin. 2019\.  Degraded Historical Document Binarization: A Review on Issues, Challenges, Techniques, and Future Directions.  _Journal of Imaging_ 5 (04 2019).  https://doi.org/10.3390/jimaging5040048
  * Tensmeyer and Martinez (2020) Chris Tensmeyer and Tony Martinez. 2020.  Historical Document Image Binarization: A Review.  _SN Computer Science_ 1 (05 2020).  https://doi.org/10.1007/s42979-020-00176-1
  * Tong and Evans (1996) Xiang Tong and David A. Evans. 1996.  A Statistical Approach to Automatic OCR Error Correction in Context. In _Fourth Workshop on Very Large Corpora, VLC@COLING 1996_. Association for Computational Linguistics, Copenhagen, Denmark, 88–100.  http://www.aclweb.org/anthology/W/W96/W96-0108.pdf
  * Walker et al. (2018) Jake Walker, Yasuhisa Fujii, and Ashok C. Popat. 2018\.  A Web-Based OCR Service for Documents. In _13th IAPR International Workshop on Document Analysis Systems (DAS)_ , Vol. 1. Vienna, Austria, 21–22.
  * Weil (2021) Stefan Weil. 2021\.  Tesseract-OCR/langdata: Source training data for Tesseract for lots of Languages.  https://github.com/tesseract-ocr/langdata
  * Yahui et al. (2020) Wang Yahui, Shi Ling, Zhang Cai, Fu Liuqiang, and Jin Xiangjie. 2020.  NSGA-II algorithm and application for multi-objective flexible workshop scheduling.  _Journal of Algorithms & Computational Technology_ 14 (2020), 1748302620942467\.  https://doi.org/10.1177/1748302620942467
  * Yang (2021) Xin-She Yang. 2021\.  _Chapter 6 - Genetic Algorithms_ (second edition ed.).  Academic Press, 91–100.  https://doi.org/10.1016/B978-0-12-821986-7.00013-5
  * Yu et al. (1998) M. Yu, Nawapak Eua-anant, A. Saudagar, and Lalita Udpa. 1998\.  Genetic algorithm approach to image segmentation using morphological operations. In _1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269)_ , Vol. 3. IEEE Computer Society, Chicago, Illinois, USA, 775–779.  https://doi.org/10.1109/ICIP.1998.999063