# An end-to-end Optical Character Recognition approach for ultra-low-
resolution printed text images

Julian Gilbey  Department of Applied Mathematics and Theoretical
Physics, University of Cambridge
Email<EMAIL_ADDRESS>
ORCID iD: 0000-0002-5987-5261 Carola-Bibiane Schönlieb  Department of Applied
Mathematics and Theoretical
Physics, University of Cambridge
Email<EMAIL_ADDRESS>
ORCID iD: 0000-0003-0099-6306

###### Abstract

Some historical and more recent printed documents have been scanned or stored
at very low resolutions, such as 60 dpi. Though such scans are relatively easy
for humans to read, they still present significant challenges for optical
character recognition (OCR) systems. The current state-of-the art is to use
super-resolution to reconstruct an approximation of the original high-
resolution image and to feed this into a standard OCR system. Our novel end-
to-end method bypasses the super-resolution step and produces better OCR
results. This approach is inspired from our understanding of the human visual
system, and builds on established neural networks for performing OCR.

Our experiments have shown that it is possible to perform OCR on 60 dpi
scanned images of English text, which is a significantly lower resolution than
the state-of-the-art, and we achieved a mean character level accuracy (CLA) of
99.7% and word level accuracy (WLA) of 98.9% across a set of about 1000 pages
of 60 dpi text in a wide range of fonts. For 75 dpi images, the mean CLA was
99.9% and the mean WLA was 99.4% on the same sample of texts. We make our code
and data (including a set of low-resolution images with their ground truths)
publicly available as a benchmark for future work in this field.

##  I Introduction

The current generation of optical character recognition (OCR) software is
designed for recognising printed text at 300 dpi. For lower resolutions down
to about 150 dpi, the image can be enlarged to 300 dpi prior to performing the
OCR step using bicubic interpolation, and the results are still excellent.
(For example, when testing this on our scanned text dataset, described in
section IV-A, we found that on average, reducing the 300 dpi images by a
factor of 2 and then enlarging them back to 300 dpi using bicubic
interpolation resulted in slightly _better_ recognition results when using the
standard Tesseract OCR software. An analysis of this behaviour is beyond the
scope of this paper.)

Document images at resolutions less than about 150 dpi offer much greater
challenges, as much less information is present in the image. (See, for
example [1], [2] and the other sources discussed in section II. We give
illustrative numerical results for our dataset in section IV below.) This
might happen if images are stored at a very low resolution for space purposes,
or if photographs of scenes containing text have been captured from a distance
so the text within them is very small. In this work, we have focused our
attention on scanned documents, but there are clear potential extensions of
our technique to other use cases.

It is astonishing that humans are able to read such low-resolution images with
relative ease when printed at this low resolution. However, things are very
different when they are enlarged. For example, figure 1 shows a word in a 60
dpi document, enlarged so that the individual pixels are visible. It is
difficult to visually determine what the word is and even where the character
boundaries are. In spite of this, when the image is held at a distance (or
made smaller), the characters become quite distinct and reasonably easy to
read. In contrast to this, figure 2 shows a different word at 300 dpi,
enlarged to the same size; this is easily readable from both nearby and from a
distance. A standard OCR system trained to recognise 300 dpi images of 10
point text will have significant difficulty with figure 1, even if it is
enlarged to 300 dpi using bicubic interpolation (as in Figure 4).

Figure 1: A word from a 60 dpi scan, enlarged so that the individual pixels
are visible. Figure 2: A simulated word from a 300 dpi scan, enlarged so that
the individual pixels are visible.

In this work, we have taken a novel approach to this problem, building on
certain existing ideas, but introducing a new human-vision-inspired technique.
Our main contributions are:

  * •

Performing OCR on 60 dpi scanned printed text, and achieving outstanding
character and word recognition rates; OCR on such ultra-low-resolution images
has not been previously reported.

  * •

Performing OCR on 75 dpi images, and achieving character and word recognition
rates significantly in excess of previously reported results for some fonts.

  * •

Demonstrating how sensitive OCR of low-resolution images can be to the font
used, suggesting that it might be very hard to develop an OCR system which
works well on all low-resolution images.

We note that we cannot directly compare our results with those appearing in
the literature, as there is no standard benchmark dataset for this problem and
none of the software developed is publicly available. We are therefore limited
to quoting the results appearing in those papers. Also, as we note later, low-
resolution OCR is very sensitive to the font used and to the precise nature of
the images, making it even harder to directly compare our results. To help
remedy this situation, we have prepared a test dataset that could be used for
future benchmarking, as described in section IV-A below.

The rest of the paper is structured as follows. In section II, we briefly
survey related work. In section III, we discuss one aspect of human vision and
how it is related to our problem, and then explain our approach in detail. We
present and analyse our results in section IV. A summary of our algorithm is
displayed in figures 5 to 7.

##  II Related work

One early approach for performing OCR on low-resolution text images was
presented in [3] and [4] in the context of images captured by low-resolution
cameras. The issues they faced were primarily around blurriness; the image
resolution was on the effective order of 150–200 dpi. They used a neural
network to recognize individual characters, followed by a separate word
recognition phase.

More recently, there has been renewed effort to address low-resolution images
in the 75–100 dpi range. Several authors consider using multiple thresholds
for binarisation of the images and then using an ensemble method on the OCR
output produced from each (such as [2], [5] and [6]); these approaches are
suitable for OCR systems that work by initially thresholding a greyscale image
to produce a binary image before performing segmentation and feature
extraction.

A second popular approach has been to use super-resolution (for example, [7],
[1], [8], [9] and [10]). This approach was also perhaps partially spurred on
by the ICDAR 2015 Competition on Text Image Super-Resolution ([11], [12] and
[13]), which aimed to recognise text from frames extracted from video streams.
The task was to reconstruct a high-resolution image from a low resolution one,
and to then use that to perform OCR. (It is important to note that super-
resolution is not simply upscaling using bicubic interpolation or similar, but
rather an attempt to match the original image. The quality of a super-
resolution reconstruction is typically measured using a metric such as PSNR,
comparing the reconstruction with the ground truth high-resolution image.)
There has been considerable success with this method. For example, Pandey et
al. [9] reported OCR results for 75 dpi English text as 75.1% CLA (character
level accuracy) and 48% WLA (word level accuracy). They achieved significantly
better results on 75 dpi Tamil (90.8% CLA and 54.3% WLA) and Kannada (95.16%
WLA and 70.80% WLA). A more powerful super-resolution technique used by Lat
and Jawahar [10], and their quoted results are the best we are aware of. Their
OCR results depended upon the OCR tool being used: the commercial ABBYY
achieved 97.88% CLA and 95.20% WLA on their swd dataset, while the open source
Tesseract version 3.x achieved 85.96% CLA and 89.70% WLA; ABBYY also achieved
99.65% CLA and 98.90% WLA on their end dataset.

There are some issues with Lat and Jawahar’s [10] methodology, which makes it
difficult to directly compare their results with other quoted results. Though
the precise texts chosen for their end test set are different from the
training texts, the generation process was presumably the same, and so this
only shows (the still impressive result) that their GAN super-resolution
scheme can very accurately reconstruct high-resolution images from low-
resolution images if they are of a very similar nature to the training images.
However, with real-life scanners, different scanners may produce different
behaviours from simulations, and so it is necessary to test any proposed
system with real scanners. Also, for their swd dataset, it seems from their
Figure 7 that these images are captured at closer to 100 dpi than 75 dpi, so
it is unsurprising that they give relatively good results.

None of these researchers reported attempting OCR with 60 dpi images: whilst
even 50 dpi images were improved using super-resolution (for example, Pandey
and Ramakrishnan [9] report an average PSNR of 16.89 for super-resolved 50 dpi
images and 19.36 for 75 dpi images), the OCR systems are relatively poor at
extracting text from such noisy images, and so the accuracy rate would be even
lower for super-resolved 60 dpi images. We also note at this point that 60 dpi
images contain only about 60% as many pixels as the corresponding 75 dpi
images, making the task significantly harder.

Super-resolving the image is of interest to humans, who would like to see a
higher-resolution image, but our primary goal here is to extract text from the
image. Furthermore, as is clear from figure 1 above, it can be challenging to
even determine whether such a low-resolution image is even of a sans serif or
a serif font. A super-resolved 300 dpi image starting from this would be quite
likely to hallucinate much of the detail, and even if it did perform this step
reasonably well, a standard OCR system might well struggle with the resulting
image.

One possible approach is to create an end-to-end loss function, where a single
neural network performs super-resolution followed by OCR, and the loss
function is determined by the OCR accuracy. Another approach, which is close
to the one we have taken in this paper, is to skip the super-resolution step
entirely, and to perform OCR directly on the original image. Such an approach
was, to the best of our knowledge, first taken by Wang and Singh [14], who
designed a RNN-LSTM neural network to replace Tesseract 3.02’s feature-based
character recognition system, and trained a network on a collection of domain-
specific text lines generated at 72 dpi, 100 dpi, 150 dpi, 200 dpi and 300
dpi. This RNN-LSTM network had an input height of 32 pixels, so they scaled
the images of text lines, both during training and recognition, to this height
using spline interpolation. (We emphasise that this scaling is not an attempt
to perform super-resolution.) Their quoted result for 72 dpi images is a CLA
of 91.95%. Since then, Tesseract 4.0.0 has been released, which uses a similar
neural network for recognition, but is only trained on 300 dpi images.

##  III Our human-vision inspired approach

###  III-A Human vision

Figure 3: Figure 1, enlarged to 300 dpi with nearest-neighbour interpolation
and then filtered with a low-pass spatial Gaussian filter. Figure 4: Figure 1,
enlarged to 300 dpi using bicubic interpolation.

It is remarkable that most people are able to read the word in figure 1 when
they hold it at a distance. The cause of this was researched in detail by
Majaj et al. [15]. At a far distance (or at a small size, which is effectively
the same), our visual system primarily processes the low-frequency information
(the gross shape of the letters), whereas at a near distance (or large size),
we process higher-frequency information (the edges of the letters). For a low-
resolution text image such as this, the high-frequency (edge) information is
mostly noise, so it is impossible to identify letters from this, whereas the
low-frequency information still contains approximately enough information to
identify the letters. (The concept that different frequencies are perceived at
different distances has also been successfully developed by Oliva et al [16]
to create hybrid images, for example making a piece of text visible from
nearby but invisible at a distance; a well-known example of this phenomenon is
the Marilyn Monroe and Albert Einstein hybrid.)

One way we could therefore hope to improve the performance of an OCR system on
these low-resolution images is to upscale them to 300 dpi (the resolution on
which the OCR systems are trained) using nearest-neighbour interpolation and
then blur the images using a small Gaussian filter. This has the effect of
reducing the high-frequency components and making the edges less distinct. A
similar effect results from using bilinear or bicubic interpolation. As a
result, the word becomes easier to read; results of doing this to the word in
figure 1 are shown in figure 3 and figure 4.

###  III-B Current OCR systems

Figure 5: A schematic illustration of our low-resolution OCR ensembling
approach: (1) a low-resolution image is upscaled and blurred in multiple
different ways; (2) each upscaled image is recognised using an LSTM trained
for that type of upscaling, the results of which include confidences for each
word; (3) the results are merged to produce the best option for each word,
taking a dictionary into account. An example of the merging algorithm in
action is shown in figure 7. Figure 6: A schematic illustration of our
modified version of Tesseract for low-resolution images, where the green
arrows indicate new steps and the red arrow indicates a replaced step: (1) the
original low-resolution image is upscaled and blurred to make a 300 dpi image;
(2) the page is segmented and the text lines extracted; (3) (original
Tesseract only) each text line is scaled to 36 pixels high; (4) each 36-pixel
high scaled text line is produced using upscaling and blurring directly from
the original image; (5) the text line is passed to an LSTM network with a CTC;
(6) textual output and confidences for each word are produced. Figure 7:
Merging different results, the final step of the pipeline in figure 5. (a) The
words identified when performing OCR using one type of upscaling; the colours
are a visual representation of the confidence level, and the textual results
are shown below it (with a number such as [16] meaning 16% confidence). (b)
The results using a different upscaling. (c) Merger of (a) and (b) using our
algorithm. The small bands of colour above and below the images are from
adjacent lines of text. Note that the bounding boxes of the identified words
differ significantly between the two images.

State-of-the-art OCR systems now use trained bidirectional-LSTM neural
networks to perform the character recognition step. Probably the best-known
open source OCR system available today is Tesseract [17], which has used such
a network since version 4.0.0 (see, for example, [18], section 6 of [19] and
[20]). In brief, Tesseract first segments the page into areas of text, which
are further subdivided into paragraphs and then lines. Each line is passed
into the trained network, which uses a final softmax layer attached to a CTC
(connectionist temporal classifier) ([21], [19]) to determine the most likely
sequence of letters and spaces; these are then further processed to produce a
sequence of words. Each letter and each word also carries with it a confidence
score, based on the output of the classifier, and Tesseract can be instructed
to output this information.

###  III-C Modifying Tesseract

Tesseract is designed to be trainable, and so we would like to train it to
recognise upscaled images. A small issue arises in training it directly on
upscaled images, so we slightly modified the program to handle them, as we now
explain. A sketch of Tesseract’s pipeline (version 4.1.1 of the software) is
shown in figure 6, together with our modifications. We choose an upscaling
method and apply it consistently for training and recognition. The upscalings
that we used in this work were: upscaling using nearest neighbour
interpolation followed by a spatial Gaussian filter with standard deviation 0,
0.5, 1, 1.5 or 2 pixels in the 300 dpi upscaled image; bilinear interpolation
(with no filtering), and bicubic interpolation (also with no filtering). (In
our implementation, we have referred to nearest neighbour interpolation as box
interpolation; for upscaling, these are equivalent.) Tesseract is given the
low-resolution image and upscales it to 300 dpi in the manner specified. It
then performs its page segmentation as normal, and extracts a text line. At
this point, Tesseract would scale the text line to make it the correct height
for the trained LSTM network (which happens to be 36 pixels), using linear
interpolation and unsharp masking. Since our image has already been upscaled
once, this would potentially lead to the loss of information, so we instead
produce the required region by upscaling the original image. (This is why we
chose to modify the software rather than directly training on upscaled
images.) During training, we use this image to train the LSTM network for this
type of upscaling, and during recognition, we use that specifically-trained
network.

For the training phase, we used transfer learning, beginning with the
Tesseract ‘best’ English network (available from the Tesseract repository). We
generated lines of low-resolution text in a variety of fonts at a variety of
font sizes (from 9pt to 12pt), adding small amounts of Gaussian noise to the
results, and also randomly rotating half of them by a small angle (normally
distributed with mean $0^{\circ}$ and standard deviation $0.5^{\circ}$) to
mimic the effect of scans being slightly askew. If we were training for a
specific type of document, it would make sense to train using lines of text
from that genre of document, as the LSTM will learn patterns of languages.
(This is what is done in [14].) However, we have chosen here to train our
system for a wide variety of texts, so we have used the Tesseract training
texts, which consists of about 170 000 lines of random English ‘words’ gleaned
from a variety of sources. It is not clear how many lines of text it is best
to train on to avoid over-fitting, so for our experiments, we trained on 10%
of them and also on 100% of them; we compare the results below. For each of
these, we randomly chose 90% of the lines to be used for training and 10% for
validation (‘evaluation’ in Tesseract’s terminology). For the training on the
entire 170 000 text lines, we performed this in 10 separate steps to allow the
possibility of evaluating the performance on different sizes of training data.
We separately trained Tesseract for 60 dpi images and for 75 dpi images.

Figure 8: The proportion of words in 20 000 random lines of text (from the
Tesseract training texts) that are correctly recognised, plotted against the
confidence that Tesseract has in the word. This is the average across all
seven upscaling methods produced for 60 dpi images. The sizes of the dots are
proportional to the frequency of words with this confidence, and words with
confidence scores less than 95 are binned together.

###  III-D An ensemble method

Initial small experiments showed that there was little consistency about which
type of upscaling gave the best recognition results across texts; furthermore,
this varied on a word-by-word basis. We therefore decided to use an ensemble
method, combining the Tesseract outputs from different upscalings to give the
best possible output, at the cost of running time. As noted earlier,
ensembling approaches to OCR have been explored by a number of researchers.
However, we have more information than they did: Tesseract can be instructed
to provide a confidence score and bounding box for each word. (The confidence
score is determined from the final softmax layer and the CTC in a somewhat
complex way whose details do not concern us here.) We therefore wish to take
account of the confidence of the different outputs, not just their counts.
There are two key complexities to address here:

  1. (a)

Determining when two words output by Tesseract represent the same piece of
text; as can be seen in Figure 7, the bounding boxes determined for the same
word in different recognition passes can be very different, and one word can
even appear as two words or be entirely missing in a different pass.

  2. (b)

Deciding how to perform a majority vote taking confidences into account.

There are several difficulties in implementing point (a) above. For example,
if a word appears at a particular location in one output but no word appears
at that location in another, what should we do? If we do decide to use that
word, where in the text should it be inserted? (We must recall that our
primary goal is to recognise the original text, and the order of words is a
critical part of this, not just their location on the page.) This is further
compounded when Tesseract’s page segmentation algorithm splits up the page in
different ways with different upscalings. Several attempts to address this
were unsuccessful, and for this reason, we opted for a simpler approach which
works well most of the time. We designate one output as the master output, and
compare all other outputs to that. For every word in the master output, we
consider all the words in the corresponding locations in the other outputs
whose bounding box is approximately the same as the master output’s bounding
box, and ensemble the words so identified. This means that the number of words
being ensembled at a particular point may be fewer than the number of
Tesseract outputs; in the example shown in Figure 7, taking output (a) as the
master output, the final word ‘Payment." ’ does not match anything in output
(b), and the words ‘Pa’ and ‘l" ’ in output (b) are ignored. Finally, the
choice of the master output was determined by the performance of the modified
Tesseract on a large sample of lines of text drawn from the training set. It
turned out that for 60 dpi, nearest neighbour plus a Gaussian filter with a
standard deviation of 1 pixel was best, while bicubic interpolation was best
for 75 dpi; these were therefore used as the respective master outputs.

For point (b), a further complication is the non-linearity of the confidences
in relation to the accuracy when working with low-resolution images. Figure 8
shows the case for a large set of 60 dpi images: while the word confidence
scores for 300 dpi images turn out to be a fairly good estimate of the
probability of the word being correctly identified, this is not the case for
60 dpi images. There is also a lot of variation in this between different
fonts (not shown here). We therefore work with a _modified confidence_ score,
which is a simple piecewise linear function giving a very rough approximation
to this data:

$c_{\text{mod}}=\begin{cases}0.5c+30&\text{if $c<80$}\\\ 1.7c-65&\text{if
$c\geq 80$}\end{cases}$

(Though a confidence of 100 would give a modified confidence of over 100,
Tesseract appears to never give a confidence score greater than 97.) A similar
function is used for 75 dpi images based on the equivalent of Figure 8.

We now wish to perform a majority vote on each word, taking the modified
confidence scores into account. There does not seem to be a principled way to
do this, as we do not know what probability each network would give to other
possible outputs. So we resort to the following: let the output words under
consideration be $w^{(1)}$, …, $w^{(n)}$ with respective modified confidences
$c_{\text{mod}}^{(1)}$, …, $c_{\text{mod}}^{(n)}$, and let $W$ be the set of
distinct output words. Then we define the best output word to be

$w_{\text{best}}=\operatorname*{arg\,max}_{w\in
W}\frac{\sum_{\\{i:w^{(i)}=w\\}}c_{\text{mod}}^{(i)}}{\sqrt{|\\{i:w^{(i)}=w\\}|}}$

This is somewhere between the sum of all modified confidences for a given
possible output word and their mean, so that a repeated output with modified
confidences of 40 and 41 do not override a single output with modified
confidence of 80. Prior to doing this, we also reduce the modified confidence
of outputs that are not dictionary words (after removing any punctuation) by
30; this gives some extra weight to dictionary words. The choice of 30 was
experimentally determined as being a reasonable value on a small dataset, but
slightly better results might be obtained by using a different penalty. We
then replace the master output word with the ensemble output to produce the
final output.

##  IV Results and discussion

###  IV-A Test data preparation

We tested our method on a large set of scanned images, none of which had been
used in the training or algorithm development process. Furthermore, the
training process generated images by simulation, while the test data was
produced using a real scanner. To prepare the images, we chose 11 different
texts: ten were taken from a variety of books in the Project Gutenberg [22]
collection, including fiction, non-fiction and poetry, while one was taken
from a recent U.S. Supreme Court opinion. (Details of the texts used are
included in the public dataset we have made available.) Five pages of text
were extracted from the start of each of these, totalling 124 961 characters
and 25 656 words. Each page of text was typeset in 18 different fonts (so
identical text appeared on each corresponding page). These fonts included:
serif and sans serif fonts; monospaced and variable width fonts, and fonts
which had been trained on and fonts which had not been used for training. The
font sizes were all 10 pt or 11 pt, with the font size chosen so that the
corresponding pages would each have about the same area of paper covered with
text. The fonts used are listed in Table I. The resulting 990 pages were then
printed at 600 dpi and scanned at 300 dpi. These scans were then down-sampled
to 60 dpi and 75 dpi using box interpolation (which takes the average value of
each $5\times 5$ or $4\times 4$ box of pixels respectively).

This variety allowed us to both test the effectiveness of our algorithm and to
independently assess its performance on different types of text and on
different fonts.

Font | Size | Serif | Mono | Train | Test
---|---|---|---|---|---
Arial | 11 pt | no | no | yes | yes
Avenir | 11 pt | no | no | yes | no
Baskerville | 11 pt | yes | no | yes | yes
Bookman Old Style | 11 pt | yes | no | no | yes
Calibri | 11 pt | no | no | yes | yes
Cambria | 11 pt | yes | no | no | yes
Century Gothic | 10 pt | no | no | no | yes
Chalkboard | 11 pt | no | no | yes | no
Comic Sans | 10 pt | no | no | yes | yes
Courier New | 10 pt | yes | yes | yes | yes
FrankRuehlCLM | 11 pt | yes | no | yes | no
Franklin Gothic Book | 11 pt | no | no | yes | yes
Futura | 11 pt | no | no | yes | no
Garamond | 11 pt | yes | no | no | yes
Gill Sans | 11 pt | no | no | yes | yes
Helvetica | 11 pt | no | no | yes | yes
Lucida Console | 11 pt | no | yes | yes | no
Lucida Sans | 10 pt | no | no | yes | yes
Menlo | 10 pt | no | yes | yes | yes
Optima | 11 pt | no | no | no | yes
Palatino | 11 pt | yes | no | yes | yes
Tahoma | 11 pt | no | no | yes | no
Trebuchet | 11 pt | no | no | yes | no
Times New Roman | 11 pt | yes | no | yes | yes
Verdana | 10 pt | no | no | yes | yes
TABLE I: Fonts used for training and testing

The character level accuracy (CLA) and word level accuracy (WLA) are
calculated by finding the Levenshtein distance $d$ between our algorithm’s
output and the ground truth; the percentage accuracy is then calculated as
$(1-\frac{d}{L})\times 100\%$, where $L$ is the number of characters or words
in the ground truth. (For the CLA, we first collapse all sequences of white
space into a single space character.)

One particular difficulty that quickly became apparent in initial experiments
is that it is almost impossible to identify the type of quote marks used at
these low resolutions; an open quote mark (‘), a straight quote mark (') and a
close quote mark (’) all appear almost identical at 60 dpi. So in calculating
the CLA and WLA, a decision was made to treat all three quote marks as
equivalent (and similarly for double quotes and for dashes/hyphens).

###  IV-B Numerical results

| 60 dpi plain | 60 dpi lores | 75 dpi plain | 75 dpi lores | 300 dpi
---|---|---|---|---|---
Font | CLA | WLA | CLA | WLA | CLA | WLA | CLA | WLA | CLA | WLA
Arial | 99.4% | 97.0% | 99.8% | 99.2% | 99.8% | 99.0% | 99.9% | 99.3% | 99.9% | 99.2%
Baskerville | 99.1% | 96.3% | 99.6% | 99.0% | 99.8% | 98.8% | 99.8% | 99.4% | 99.9% | 99.5%
Bookman Old Style | 99.7% | 98.3% | 99.9% | 99.7% | 99.9% | 99.5% | 100.0% | 99.8% | 99.9% | 99.6%
Calibri | 99.4% | 96.8% | 99.8% | 99.2% | 99.8% | 98.8% | 99.9% | 99.4% | 99.9% | 99.1%
Cambria | 99.6% | 97.8% | 99.8% | 99.0% | 99.9% | 99.1% | 99.9% | 99.7% | 99.9% | 99.4%
Century Gothic | 96.6% | 84.9% | 99.2% | 96.1% | 99.2% | 95.7% | 99.7% | 98.7% | 99.7% | 98.5%
Comic Sans | 99.1% | 95.1% | 99.9% | 99.4% | 99.7% | 98.3% | 99.9% | 99.6% | 99.9% | 99.3%
Courier New | 98.9% | 94.7% | 99.8% | 99.3% | 99.7% | 98.5% | 99.9% | 99.6% | 99.9% | 99.3%
Franklin Gothic Book | 99.1% | 95.6% | 99.8% | 99.2% | 99.7% | 98.5% | 99.9% | 99.3% | 99.9% | 99.2%
Garamond | 97.1% | 91.2% | 98.8% | 97.6% | 99.7% | 98.6% | 99.7% | 99.2% | 99.9% | 99.3%
Gill Sans | 99.0% | 95.4% | 99.7% | 98.9% | 99.8% | 98.7% | 99.8% | 99.2% | 99.9% | 99.2%
Helvetica | 99.1% | 96.1% | 99.8% | 99.1% | 99.8% | 98.6% | 99.8% | 99.2% | 99.8% | 98.9%
Lucida Sans | 99.5% | 97.5% | 99.8% | 99.2% | 99.8% | 98.7% | 99.9% | 99.3% | 99.9% | 99.1%
Menlo | 99.5% | 97.3% | 99.9% | 99.6% | 99.9% | 99.3% | 99.9% | 99.7% | 99.9% | 99.6%
Optima | 99.6% | 97.9% | 99.9% | 99.3% | 99.8% | 99.0% | 99.9% | 99.3% | 99.9% | 99.3%
Palatino | 99.0% | 97.8% | 98.5% | 98.0% | 99.8% | 99.3% | 99.7% | 99.3% | 100.0% | 99.7%
Times New Roman | 99.5% | 97.4% | 99.8% | 99.4% | 99.8% | 99.2% | 99.9% | 99.7% | 100.0% | 99.7%
Verdana | 99.5% | 97.4% | 99.9% | 99.6% | 99.9% | 99.4% | 99.9% | 99.7% | 100.0% | 99.7%
TABLE II: The average results on each font: the ‘plain’ results are with
standard Tesseract on images upscaled with bicubic interpolation, while the
‘lores’ results are those obtained using our system. The final two columns
give the results for plain Tesseract on the original 300 dpi images.

We present results for the 18 fonts tested in Table II, averaged across all 55
pages printed in that font. The results are shown for the networks trained on
100% of the Tesseract training lines; the results when the networks were
trained on only 10% of the lines are very slightly worse. The table also shows
the performance of plain Tesseract on the images when they have been upscaled
to 300 dpi using bicubic interpolation. Our results show a significant
improvement for 60 dpi images and a good improvement for 75 dpi images (for 60
dpi, the overall character error rate dropped by about 64% and the word error
rate by about 73%; for 75 dpi, the corresponding figures are 35% and 51%). The
only font which performs noticeably worse with our system is Palatino, and
there is no clear reason for this.

We note at this point that even for 60 dpi images, these results are
significantly better than the previous state-of-the-art’s quoted performance
on 75 dpi images, and that the performance on both 60 dpi and 75 dpi images is
close to that of plain Tesseract on 300 dpi images; for some fonts, our method
even does slightly _better_ on the 60 dpi images than plain Tesseract on the
corresponding 300 dpi images!

Inspired by this, we also considered what would happen if we started with the
300 dpi scans and used our ensemble technique by taking the plain Tesseract
run as the master output, and the results of downsampling the image to 75 dpi
and using our modified Tesseract to produce the other outputs. Preliminary
results indicate that the mean WLA improves somewhat, but the CLA declines.
This is an area for further study.

We next look at the impact of different texts on the performance of this
system. We use Cambria to illustrate this, as its performance is approximately
in the middle of the fonts tested and it was not used during the training.
Furthermore, we use the error rate (100% minus the accuracy rate) to make it
easier to read the figures.

| 60 dpi | 75 dpi
---|---|---
Font | C err | W err | C err | W err
Around the World | 0.07% | 0.35% | 0.01% | 0.08%
Best Poetry | 0.33% | 1.45% | 0.20% | 0.77%
David Copperfield | 0.41% | 1.84% | 0.15% | 0.82%
Engineering | 0.06% | 0.33% | 0.01% | 0.04%
English Church | 0.11% | 0.48% | 0.01% | 0.03%
Fire Prevention | 0.13% | 0.64% | 0.02% | 0.07%
Flatland | 0.16% | 0.71% | 0.03% | 0.15%
Practical Mechanics | 0.34% | 1.47% | 0.12% | 0.59%
Reflections | 0.18% | 0.89% | 0.05% | 0.20%
Supreme Court | 0.38% | 1.81% | 0.22% | 0.86%
Wordsworth | 0.34% | 1.75% | 0.06% | 0.44%
TABLE III: The performance of our system on a variety of texts printed in
Cambria

It is pleasing to see that the performance was fairly consistent across these
different text genres, ranging from 19th century poetry (Best Poetry and
Wordsworth), through 19th century novels (Around the World in Eighty Days, for
example) and non-fiction works (Fire Prevention, for example), to a recent
opinion of the United States Supreme Court. There is no obvious pattern to
which texts which performed better; one might have anticipated poetry doing
very poorly, but that has not turned out to be the case. We might have
achieved better results for specific texts if we had fine-tuned the network
using related lines of text (as Wang and Singh [14] did), as the LSTM would
have learnt to recognise word patterns, but then it might have performed more
poorly on other texts. This remains a possible way to improve our system.

##  V Conclusion

We have presented a novel end-to-end ensembling approach for performing
optical character recognition on very-low-resolution scanned documents. We
have made use of what is known about human perception of letters in order to
improve our training approach over the existing state-of-the-art, and
demonstrated that our approach has very high average accuracy levels on a
large set of images. We were able to successfully perform OCR on lower
resolution images than previously reported, achieving character and word level
accuracies of over 99% for most fonts scanned at 60 dpi, and also to achieve
an accuracy rate on 75 dpi scans similar to that on the equivalent 300 dpi
images, far exceeding the existing state-of-the-art performance. We noted that
different fonts give different results with our system, and highlighted the
need for a standard benchmark dataset for this type of problem.

There are many open questions remaining. On the technical side, what is the
best number of training iterations to perform? How could we improve the
ensembling algorithm? An improvement would be to perform the page segmentation
using just one upscaling, and to use this segmentation for each of the line
recognition runs; we did not attempt to implement this in our experiments. We
could also use fewer upscalings or different upscalings from the ones we
chose: it is unclear how to best choose them, and our choice was a somewhat
arbitrary selection of filters for reducing the high frequency components of
the images.

Finally, it would be very useful to adapt this approach to tackle a challenge
such as the ICDAR 2015 Competition on Text Image Super-Resolution [11], which
used very different types of image from the ones we studied in this work.

## Software and dataset availability

The modified version of Tesseract 4.1.1 used in these experiments can be found
at https://github.com/juliangilbey/tesseract/tree/lores-v1.0. (This URL points
to the exact version used in these experiments.) Supporting scripts used are
available are https://github.com/juliangilbey/ocr-lowresolution. The scanned
image dataset is available at http://doi.org/10.5281/zenodo.3945525 and the
trained Tesseract networks are available from
http://doi.org/10.5281/zenodo.3945949.

## Acknowledgements

We thank Cognizant, in particular Indranil Mitra, Aditya Joshi and Shishir
Bharti, for posing this industrial problem and for funding this work through
research grant RG299289. We also thank Linda Bowns for pointing us in the
direction of spatial frequency channels, which led us to Majaj et al’s work
[15].

## References

  * [1] C. Dong, X. Zhu, Y. Deng, C. C. Loy, and Y. Qiao, “Boosting Optical Character Recognition: A Super-Resolution Approach,” _arXiv:1506.02211 [cs]_ , Jun. 2015, arXiv: 1506.02211. [Online]. Available: http://arxiv.org/abs/1506.02211
  * [2] I. Habeeb, S. Azmi, S. A. Mohd Yusof, and F. Ahmad, “Improving Optical Character Recognition Process for Low Resolution Images,” _International Journal of Advancements in Computing Technology (IJACT)_ , vol. 6, pp. 13–21, May 2014.
  * [3] C. E. Jacobs, J. R. Rinker, P. Y. Simard, and P. A. Viola, “Low resolution OCR for camera acquired documents,” US Patent US7 499 588B2, Mar., 2009, library Catalog: Google Patents. [Online]. Available: https://patents.google.com/patent/US7499588B2/en
  * [4] C. Jacobs, P. Y. Simard, P. Viola, and J. Rinker, “Text recognition of low-resolution document images,” in _In Proc. ICDAR_ , 2005, pp. 695–699.
  * [5] I. Q. Habeeb, Z. Q. Al-Zaydi, and H. N. Abdulkhudhur, “Enhanced Ensemble Technique for Optical Character Recognition,” in _New Trends in Information and Communications Technology Applications_ , ser. Communications in Computer and Information Science, S. O. Al-mamory, J. K. Alwan, and A. D. Hussein, Eds. Cham: Springer International Publishing, 2018, pp. 213–225. [Online]. Available: https://doi.org/10.1007/978-3-030-01653-1_13
  * [6] W. B. Lund, “Ensemble Methods for Historical Machine-Printed Document Recognition,” Ph.D., Brigham Young University, Provo, Utah, 2014\. [Online]. Available: https://scholarsarchive.byu.edu/etd/4024/
  * [7] D. Ma and G. Agam, “A super resolution framework for low resolution document image OCR,” _Proceedings of SPIE - The International Society for Optical Engineering_ , vol. 8658, Feb. 2013. [Online]. Available: https://doi.org/10.1117/12.2008354
  * [8] R. K. Pandey, S. R. Maiya, and A. G. Ramakrishnan, “A new approach for upscaling document images for improving their quality,” in _2017 14th IEEE India Council International Conference (INDICON)_ , Dec. 2017, pp. 1–6. [Online]. Available: https://doi.org/10.1109/INDICON.2017.8487796
  * [9] R. K. Pandey and A. G. Ramakrishnan, “Efficient document-image super-resolution using convolutional neural network,” _Sādhanā_ , vol. 43, no. 2, p. 15, Mar. 2018. [Online]. Available: https://doi.org/10.1007/s12046-018-0794-1
  * [10] A. Lat and C. V. Jawahar, “Enhancing OCR accuracy with super resolution,” in _2018 24th International Conference on Pattern Recognition (ICPR)_. IEEE, Aug. 2018, pp. 3162–3167. [Online]. Available: https://doi.org/10.1109/ICPR.2018.8545609
  * [11] C. Peyrard, M. Baccouche, F. Mamalet, and C. Garcia, “ICDAR2015 competition on Text Image Super-Resolution,” in _2015 13th International Conference on Document Analysis and Recognition (ICDAR)_ , Aug. 2015, pp. 1201–1205. [Online]. Available: https://doi.org/10.1109/ICDAR.2015.7333951
  * [12] ICDAR, “ICDAR 2015 Competition on Text Image Super-Resolution,” 2015\. [Online]. Available: https://projet.liris.cnrs.fr/sr2015/
  * [13] C. Peyrard, M. Baccouche, F. Mamalet, and C. Garcia, “The ICDAR2015-TextSR dataset,” Apr. 2020, original-date: 2019-10-08T10:06:23Z. [Online]. Available: https://github.com/piclem/ICDAR2015-TextSR
  * [14] S. Wang and M. K. Singh, “Systems and Methods for Optical Character Recognition for Low-Resolution Documents,” US Patent US20 180 101 726A1, Apr., 2018, library Catalog: Google Patents. [Online]. Available: https://patents.google.com/patent/US20180101726A1/en?q=Systems+Methods+%22Optical+Character+Recognition%22+Low-Resolution+Documents&oq=Systems+and+Methods+for+%22Optical+Character+Recognition%22+for+Low-Resolution+Documents
  * [15] N. J. Majaj, D. G. Pelli, P. Kurshan, and M. Palomares, “The role of spatial frequency channels in letter identification,” _Vision Research_ , vol. 42, no. 9, pp. 1165–1184, Apr. 2002. [Online]. Available: https://doi.org/10.1016/S0042-6989(02)00045-7
  * [16] A. Oliva, A. Torralba, P. G. Schyns, A. Oliva, A. Torralba, and P. G. Schyns, “Hybrid images,” in _ACM Transactions on Graphics (TOG)_ , vol. 25. ACM, Jan. 2006, pp. 527–532. [Online]. Available: https://doi.org/10.1145/1141911.1141919
  * [17] R. Smith, “Tesseract open source OCR.” [Online]. Available: https://github.com/tesseract-ocr/tesseract
  * [18] A. Graves and J. Schmidhuber, “Offline handwriting recognition with multidimensional recurrent neural networks,” 2009, pp. 545–552. [Online]. Available: https://papers.nips.cc/paper/3449-offline-handwriting-recognition-with-multidimensional-recurrent-neural-networks
  * [19] R. Smith, “Tesseract tutorial at DAS, Santorini,” 2016. [Online]. Available: https://github.com/tesseract-ocr/docs/tree/master/das_tutorial2016
  * [20] A. Ul-Hasan, “Generic text recognition using long short-term memory networks,” Ph.D. dissertation, University of Kaiserslautern, 2016. [Online]. Available: https://kluedo.ub.uni-kl.de/frontdoor/deliver/index/docId/4353/file/PhD_Thesis_Ul-Hasan.pdf
  * [21] A. Graves, S. Fernández, and F. Gomez, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _In Proceedings of the International Conference on Machine Learning, ICML 2006_ , 2006, pp. 369–376.
  * [22] “Project Gutenberg.” [Online]. Available: www.gutenberg.org


11institutetext: Max Planck Institute for Astronomy, Königstuhl 17, D-69117
Heidelberg, Germany
11email<EMAIL_ADDRESS>22institutetext: Institute of Astronomy, Russian
Academy of Sciences, Pyatnitskaya str. 48, Moscow, 119017, Russia
33institutetext: Department of Chemistry, Ludwig Maximilian University,
Butenandtstr. 5-13, D-81377 Munich, Germany

# Machine learning-accelerated chemistry modeling of protoplanetary disks

Grigorii V. Smirnov-Pinchukov 11 Tamara Molyarova 22 Dmitry A. Semenov 11 3 3
Vitaly V. Akimkin 22 Sierk van Terwisga 11 Riccardo Francheschi 11 Thomas
Henning 11

(Received ; accepted)

###### Abstract

Aims. With the large amount of molecular emission data from (sub)millimeter
observatories and incoming James Webb Space Telescope infrared spectroscopy,
access to fast forward models of the chemical composition of protoplanetary
disks is of paramount importance.

Methods. We used a thermo-chemical modeling code to generate a diverse
population of protoplanetary disk models. We trained a K-nearest neighbors
(KNN) regressor to instantly predict the chemistry of other disk models.

Results. We show that it is possible to accurately reproduce chemistry using
just a small subset of physical conditions, thanks to correlations between the
local physical conditions in adopted protoplanetary disk models. We discuss
the uncertainties and limitations of this method.

Conclusions. The proposed method can be used for Bayesian fitting of the line
emission data to retrieve disk properties from observations. We present a
pipeline for reproducing the same approach on other disk chemical model sets.

###### Key Words.:

astrochemistry – methods: numerical – protoplanetary disks – stars: pre-main
sequence – ISM: molecules – submillimeter: planetary systems

##  1 Introduction

Time-dependent gas-grain chemical kinetics codes are widely used in the
astrochemical modeling of the ISM, protoplanetary disks, and even exoplanetary
atmospheres. A typical computational time to calculate the chemical evolution
of an isolated volume over $10^{5}-10^{6}$ years usually takes about 0.1–10
seconds on a single CPU core, depending on the complexity of the chemical
network and variability of the physical conditions. More complex networks –
that include deuterium or carbon isotopologues (Albertsson et al., 2013; Yu et
al., 2016) or that separately treat reactions on dust surface and in the bulk
of icy mantles (Vasyunin et al., 2009; Garrod, 2013) – increase the number of
reactions and slow down the calculations drastically. For a typical grid size
of a 2D protoplanetary disk model of $100\times 100$, a session of chemical
kinetics modeling takes at least 15 minutes up to several days on a single CPU
core. This is a reasonable timeframe for forward modeling but too long for any
retrieval of disk physical parameters based on a fitting of molecular data.
Recently, Keil et al. (2022) proposed a new method for deriving the physical
parameters of uniform gas clouds using Markov chain Monte Carlo sampling with
realistic chemical and radiative transfer modeling and successfully applied it
to the L1544 data. However, protoplanetary disks exhibit more complex
structures with strong gradients of physical conditions and another approach
is thus needed.

With the recent large programs on protoplanetary disk chemistry at the Atacama
Large Millimeter/submillimeter Array (ALMA) (MAPS: Öberg et al. (2021);
2021.1.00128.L/AGE-PRO PI: Ke Zhang, in progress; see also Guzmán et al.
(2021); Ilee et al. (2021)) and Northern Extended Millimeter Array (NOEMA)
(L19ME/PRODIGE, PI: P. Caselli, Th. Henning., in progress, see Semenov et al.,
in prep.), as well as large, partially spatially unresolved surveys of
circumstellar disk populations (ODISEA: Cieza et al., 2019; Ansdell et al.,
2016), the amount of available data for analysis expands rapidly. In addition,
high angular- and frequency-resolution observations of molecular emission
lines in individual disks are now common (e.g., Pegues et al., 2020; Garufi et
al., 2020; Pegues et al., 2021). Analyses of such data can be vastly improved
if all the lines are taken into account simultaneously (Fedele & Favre, 2020;
Holdship & Viti, 2022). However, protoplanetary disk model retrieval in a
Bayesian sense is still in its infancy.

Our goal is to build a function (an estimator) to return the reasonably
accurate chemical composition using the smallest possible subset of local
physical conditions characterizing the protoplanetary disk. Applications of
machine learning techniques are emerging in all fields of astronomy and
physics (e.g., Dieleman et al., 2015; Dunjko & Briegel, 2018; Carleo et al.,
2019; Ribas et al., 2020; Ardévol Martínez et al., 2022), providing a robust,
human-independent, and flexible way to find dependencies and correlations
within a data set. The approach is also being applied to astrochemistry: among
others, Lee et al. (2021) searched for similarities between molecules to
propose possible detectable molecules, Grassi et al. (2021) suggested a method
to reduce the number of species for chemical kinetics, Holdship et al. (2021)
explored a large parameter space to provide time-dependent chemistry, and,
most recently, Villadsen et al. (2022) presented a way to predict binding
energies of molecules on dust surfaces.

In this work, we present a solution to the problem of chemical model
performance. We computed a grid of 540 thermo-chemical protoplanetary disk
models, containing more than two million physical bins in various disk
environments, but at fixed elemental and dust compositions and at the same
age, which we describe in Section 2. We used a K-nearest neighbors (KNN)
machine learning (ML) algorithm (Goldberger et al., 2005) to create a robust
interpolation between local physical conditions in the disk and the abundances
of molecular and atomic species, described in detail in Section 3. Once the
estimator is trained, the chemistry can be predicted in milliseconds per disk
model, making it much faster than the next bottleneck, namely, the line
radiative transfer. In Section 4, we demonstrate the performance and
limitations of the method on a small set of chemical species. In Section 5, we
summarize our findings.

##  2 Thermo-chemical protoplanetary disk model grid

Chemical kinetics models, such as ALCHEMIC (Semenov et al., 2010), NAUTILUS
(Ruaud et al., 2016), UCLCHEM (Holdship et al., 2017), and KROME (Grassi et
al., 2014) can be very flexible and include dozens to tens of thousands of
parameters. Some of these parameters describe local physical conditions, such
as gas and dust density and temperature as well as the local ultraviolet (UV)
radiation field. Others are global parameters, for instance, the ionization
rate by radioactive nuclides or such details as the probability modificator of
a gas particle sticking to a dust grain after a collision, which affect a
large fraction of the considered chemical reaction network. The initial
elemental or molecular composition of the matter make up another set of
important input parameters. In addition, the chemical network itself contains
an extensive data set of reaction rates, with only $\sim 30\%$ of the rate
values known to an adequate level of accuracy.

Classical chemical kinetics codes can utilize any combination of these
parameters, but most parameters are fixed in real applications. Also, some
parameters could correlate between various chemical calculations. For example,
it is reasonable to expect that the low-density, high-temperature regions of
protoplanetary disks lie in the disk atmosphere, where the chemistry is
dominated by a limited set of photo- and gas-phase reactions. On the other
hand, cold and dense disk regions are typical for the midplane, where
photochemistry is much less important and gas-grain interactions and surface
reactions play a major role. This simple observation has led to the idea of
constructing simplified chemical models that allow for quick computations that
are nonetheless feasible to establish the abundances of simple species such as
CO, without the need for full chemical modeling (Williams & Best, 2014).
Unfortunately, such approaches cannot be easily generalized to predict disk
chemical composition for a larger set of observed molecules or important
coolants.

To create a reference data set of protoplanetary disks with known chemical
structures, we used the ANDES astrochemical model of a 2D axisymmetric
hydrostatic disk. It employs a chemical network based on the ALCHEMIC network
(Semenov & Wiebe, 2011), with deuterium-bearing molecules and deuterium
fractionation included following Albertsson et al. (2013, 2014). The network
describes 1247 species and 38347 reactions, including gas-phase and surface
two-body reactions, adsorption and reactive desorption, photoreactions, and
ionization or dissociation by X-rays, cosmic rays, and radioactive nuclides
(Akimkin et al., 2013; Molyarova et al., 2017, 2018). The rates of surface
reactions are adjusted to mimic the chemical inactivity of the bulk icy
mantles; they are multiplied by a factor equal to the fraction of the upper
layers in the total number of surface particles. Following Eistrup et al.
(2016), we adopted an icy molecular initial composition based on the
abundances of prestellar cores (Öberg et al., 2011). We ran the time-dependent
chemical evolution for the typically assumed age of 1 Myr (Willacy et al.,
1998; Aikawa et al., 2002).

Table 1: ANDES input grid Parameter | Value
---|---
$M_{\rm star}$ | $0.2,\ 0.3,\ 0.4,\ 0.5,\ 0.8,\ 1.0\ M_{\sun}$
$M_{\rm disk}$ | $0.001,\ 0.003,\ 0.01,\ 0.03,\ 0.1\ M_{\rm star}$
$R_{c}$ |  $20,\ 50,\ 80,\ 100,\ 130,\ 170\ $au
$L_{X}$ |  $10^{29},\ 10^{30},\ 10^{31}\ $erg s-1
$T_{\rm star}$, $R_{\rm star}$ |  $f(M_{\rm star})$ Yorke & Bodenheimer (2008)
UV excess | $B_{\nu}(T_{\rm eff}=10000\ \mathrm{K})$
Accretion rate |  $1\text{\times}{10}^{-8}$ $M_{\sun}\rm{yr}^{-1}$
Dust opacity | Draine & Lee (1984)
Mean dust grain size |  $3.7\text{\times}{10}^{-5}$ cm
Dust-to-gas mass ratio | 0.01
Grazing angle | 0.05
Density power-law slope | $-1$
Table 2: Initial chemical abundances Species | Abundance | Species | Abundance
---|---|---|---
|  $n$(X) / $n$(H) |  |  $n$(X) / $n$(H)
ortho-H2 | 0.375 | CH4 ice | $1.8\text{\times}{10}^{-5}$
para-H2 | 0.125 | CO ice | $6.0\text{\times}{10}^{-5}$
HD | $1.55\text{\times}{10}^{-5}$ | CH3OH ice | $4.5\text{\times}{10}^{-5}$
H | $5.0\text{\times}{10}^{-7}$ | CO2 ice | $6.0\text{\times}{10}^{-5}$
He | 0.098 | CH4 ice | $1.8\text{\times}{10}^{-5}$
Cl | $1.0\text{\times}{10}^{-9}$ | H2O ice | $3.0\text{\times}{10}^{-4}$
Si | $8.0\text{\times}{10}^{-9}$ | H2S ice | $6.6\text{\times}{10}^{-6}$
Fe | $3.0\text{\times}{10}^{-9}$ | N2 ice | $2.1\text{\times}{10}^{-5}$
Mg | $7.0\text{\times}{10}^{-9}$ | NH3 ice | $2.1\text{\times}{10}^{-5}$
Na | $2.0\text{\times}{10}^{-9}$ | P | $2.0\text{\times}{10}^{-10}$

The disk physical structure in our models is defined through stellar mass,
$M_{\rm star}$, disk mass, $M_{\rm disk}$, and the characteristic radius,
$R_{\rm c}$; these parameters define the distribution of density, temperature,
and radiation field in the (R, z) plane. The stellar mass governs the stellar
temperature and luminosity, which are calculated at the age of 1 Myr using the
evolutionary models by Yorke & Bodenheimer (2008). The X-ray radiation field
is calculated using Bruderer et al. (2009). The interstellar cosmic ray
ionization rate was calculated according to Padovani et al. (2018). We create
an ensemble of 540 models with different stellar mass $M_{\rm star}$, disk
mass, $M_{\rm disk}$, critical radius, $R_{\rm c}$, and stellar X-ray
luminosity, $L_{X}$, to cover a wide range of physical conditions typical for
protoplanetary disks: $M_{\rm star}=0.2...1.0\,M_{\sun}$, $M_{\rm
disk}=0.1...10\,\%\,M_{\rm star}$, $R_{c}=20...170$ au, and
$L_{X}=10^{29...31}\ $erg s-1. Each model includes 50 logarithmically spaced
radial points in the range of $R=0.1...1000$ au and 80 vertical points in the
range of $z/R=0...1$. The dust size distribution is described by a power law
with a $-3.5$ exponent between $5\text{\times}{10}^{-7}$ and
$2.5\text{\times}{10}^{-3}$ cm. The UV radiation field for photoreactions is
calculated using dust opacities based on Draine & Lee (1984). An averaged
grain size of $3.7\text{\times}{10}^{-5}$ cm is adopted for surface reaction
rates. The UV excess from accretion is defined as $L_{\rm acc}=1.5\
G\dot{M}_{\rm acc}M_{\rm star}/R_{\rm star}$. The effective temperature of the
accretion region is assumed to be 10000 K. A summary of these parameters is
presented in Table 1. With 4000 spatial points in each model, we have
$2\,160\,000$ points in total, sampling the chemical model output in
conditions typical of protoplanetary disks. The total computing resource usage
for the data generation took about 1 core year.

Figure 1: Example of the re-normalization of $\log_{10}X_{\rm CO}$ abundance
prior to interpolation. The peak at $-4$ in the unnormalized data corresponds
to the condition when all the available C and O form CO.

##  3 KNN estimator training

($\log_{10}X_{c}$) (bias, dex) (std, dex) (histogram)
(a) (b) (c) (d)

Figure 2: Performance of ML-accelerated chemistry predictions for CO. (a):
mean $\log_{10}$ predicted relative abundance as a function of local
temperature, gas density, and ionization rate. Darker areas correspond to
larger relative (to H atoms) abundance. (b): median of the difference between
the predicted values and test set data (bias, dex), in dex, as a function of
temperature and density. Gray areas correspond to an unbiased fit. (c): the
standard deviation between the predicted values and test set data, in dex
(std, dex). (d): relative density (histogram) of species within the data
points, with contours, which are also present on other panels. Various regions
of the protoplanetary disk are described on panel (a). A detailed description
of the processes leading to this figure is in the main text. Other molecules
are shown in Fig. 3.

First, we built a data frame using pandas (Reback et al., 2021), which
contains a subset of local physical parameters and a set of selected chemical
species. For demonstration purposes, we chose CO, HCO+, DCO+, and electrons,
relevant to ionization studies (Smirnov-Pinchukov et al., 2020; Aikawa et al.,
2021). We also demonstrate the possible applications to other species in
Appendix 4. We have selected observationally-relevant disk positions with gas
number density above $10^{4}$ cm-3, resulting in $1\,183\,212$ data points.
The subsequent analysis was performed using the Scikit-learn python library
(Pedregosa et al., 2011). The physical quantities (input features) and the
chemical abundances normalized to the total number of H atoms (output
features) were renormalized to a uniform distribution in order to make the
parameter space more uniformly sampled (see the example in Fig. 1). We split
the data into a training set (432 disks, 946586 points) and a test set (108
disks, 236626 points). All the points from a single disk model must appear
either in the training or a test set to avoid overfitting. The algorithm
should predict the new points based on points from the other disks rather than
interpolating nearby points of the same disk.

We used the KNeighborsRegressor estimator (Goldberger et al., 2005). The
algorithm finds the $k$ nearest data points (in the input feature space) and
interpolates the output feature values between them. For robustness, we choose
the median value of these data points. The value of $k$ should be chosen based
on the data. If $k=1$, it is a classical ”nearest” interpolation. This sort of
interpolation is very sensitive to outliers (overfitting) which, in our case,
can represent a rare combination of parameters or even a numerical failure of
the original chemical kinetics solver within ANDES. If $k$ is too large, then
the local behavior of chemistry cannot be properly caught. In the worst case,
where $k$ equals the number of data points, the solution would simply
represent their median value.

We used cross-validation to choose the value of $k$ and ensure the quality of
the interpolation. In this approach, the training set is divided into 10 parts
(splits), with all points from each disk being in the same split. The
estimator is trained on 9/10 of the training data set and its performance is
benchmarked against the remaining part, using the square sum of errors metric
in the renormalized space to quantify the quality of the fit. This is repeated
for each split, and the average performance is estimated. Then the same
procedure repeats for another value of $k$, and this way, the best-performing
value of $k$ is found. The estimator with the best $k$ is fitted again
afterward on the entire training set. The results can be saved as a python
binary (”pickle”) file and used as a fast callable function in other
applications.

##  4 Results and discussion

First, we chose the local gas density, dust temperature, and ionization rate
(the minimum set of key physical parameters for chemistry) as input features.
We provide an example of the performance of the fit in Fig. 2, using the gas-
phase CO molecule. As seen in panel (a), CO is abundant in dense regions with
a temperature above 30 K. Typically, the CO snowline should be at around 20 K.
However, in our modeling, CO is also absent in the gas at higher temperatures
due to the chemical transformation into CO2 on dust grain surfaces (Molyarova
et al., 2017; Bosman et al., 2018). At lower densities, which correspond to
the outer disk, photodesorption by the interstellar UV and ionizing radiation
maintain some amount of CO in the gas phase, enough for self-shielding. Low-
density and high-temperature areas belong to the disk atmosphere, where the
UV-radiation destroys CO. The fit reproduces this general behavior, showing
insignificant systematic error (bias, dex) on the panel (b). The fit scatter
(panel c) is especially low ($<0.5$ dex) for the inner disk and midplane. Most
of the disk CO gas is present in the inner disk and, hence, the fit reproduces
the majority of the gas-phase CO in disks with a reasonable level of accuracy.
Moreover, the fit correctly predicts low CO abundances outside the CO snowline
and in the disk atmosphere. Significant scatter (above 1 dex) is present only
at the radiation-sensitive transition zone between the atmosphere and the rest
of the disk in the low-density area. The top-left corner (very high density,
very low temperature) is not covered by the original data set, as regions with
such conditions never appeared in the disk model grid.

Notably, we can reproduce deuterium fractionation. In Fig. 3, we show other
species included in the estimator’s output: HCO+, DCO+, and $e^{-}$. While
electron density is fitted almost perfectly, the HCO+ and DCO+ fits show
$\sim$1 dex scatter in the transition zone between the inner disk and the
photodissociation area, where just a small amount of gas is present. Overall,
with just three inputs’ feature set, the estimator is able to predict the disk
chemistry with a good accuracy below 0.5 dex for most of the parameter space.

The addition of more input parameters to the input features further increases
the quality of the fit. In Fig. 4, we show the model after adding local UV
radiation intensity to the set of the input features. We can see a
significantly lower scatter in the whole parameter space. Temperature and
density remain the best predictors of the disk chemistry, explaining the major
variations of $\gg 5$ dex, with the local ionization rate contributing to
$\sim 1-2$ dex and ultraviolet field contributing to $\sim 1$ dex in relevant
disk regions. The impact of ionizing radiation is more important than the UV
for the molecules, as they reside in the deeper layers of the disk, while
X-ray and cosmic rays penetrate deeper towards the midplane. Even in the
three-parameter fit, it is important to take the local UV radiation field into
account for the data generation process. Nevertheless, as the UV field is
correlated with the combination of other input parameters, it is not necessary
to have it as an input feature for the KNN algorithm.

##  5 Conclusions

We applied, for the first time, a machine-learning estimator to physical-
chemical protoplanetary disk models to predict chemical abundances much more
quickly than traditional ”full” chemical calculations. Our estimator uses a
small and easily-calculated set of local physical parameters as input
features: the dust temperature, density, and ionization rate, with the
possible addition of a local UV radiation strength. We applied this method to
a pre-computed database of 540 protoplanetary disks of various masses, sizes,
and stellar properties, including X-ray luminosities. We studied the
effectiveness and limitations of the method due to the small input feature
set, demonstrating how the addition of a local UV field improves the accuracy
for four species in the gas: CO, HCO+, DCO+, and $e^{-}$.

This approach is general and can be applied not only to this set of species
and ANDES thermo-chemical disk models, but also to other species and
astrochemical applications. For such purposes, the outputs of other
astrochemical models (e.g., Bruderer et al., 2009; Woitke et al., 2009) with
relevant physical parameters and desired chemical species abundances can be
processed with the same approach. We publish the ANDES-generated data and the
Jupyter notebook to reproduce our results on GitHub
111https://github.com/SmirnGreg/diskchef_chemistry. These results will be used
in an upcoming series of papers to fit the molecular data obtained in the
framework of the large observing program on NOEMA, L19ME (PI: Th. Henning). We
use this method to rapidly calculate chemical composition for the Bayesian
retrieval of disk physical parameters using CO isotopologues (Francheschi et
al., in prep.) and a combined fit of CO and HCO+ isotopologues (Smirnov-
Pinchukov et al., in prep).

###### Acknowledgements.

The authors acknowledge the contribution of the Python open-source community
for providing high-quality data analysis tools. All figures were created using
matplotlib (Hunter, 2007). GSP thanks Morgan Fouesneau, Ivelina Momcheva, and
Markus Schmalzl for discussions about machine learning at MPIA. TM and VA were
supported by the grant 075-15-2020-780 (N13.1902.21.0039) of Ministry of
Science and Higher Education of the Russian Federation. TH and DS acknowledge
support from the European Research Council under the Horizon 2020 Framework
Program via the ERC Advanced Grant Origins 83 24 28.

## References

  * Aikawa et al. (2021) Aikawa, Y., Cataldi, G., Yamato, Y., et al. 2021, ApJS, 257, 13
  * Aikawa et al. (2002) Aikawa, Y., van Zadelhoff, G. J., van Dishoeck, E. F., & Herbst, E. 2002, A&A, 386, 622
  * Akimkin et al. (2013) Akimkin, V., Zhukovska, S., Wiebe, D., et al. 2013, ApJ, 766, 8
  * Albertsson et al. (2014) Albertsson, T., Semenov, D., & Henning, T. 2014, ApJ, 784, 39
  * Albertsson et al. (2013) Albertsson, T., Semenov, D. A., Vasyunin, A. I., Henning, T., & Herbst, E. 2013, ApJS, 207, 27
  * Ansdell et al. (2016) Ansdell, M., Williams, J. P., van der Marel, N., et al. 2016, ApJ, 828, 46
  * Ardévol Martínez et al. (2022) Ardévol Martínez, F., Min, M., Kamp, I., & Palmer, P. I. 2022, A&A, 662, A108
  * Bosman et al. (2018) Bosman, A. D., Walsh, C., & van Dishoeck, E. F. 2018, A&A, 618, A182
  * Bruderer et al. (2009) Bruderer, S., Doty, S. D., & Benz, A. O. 2009, ApJS, 183, 179
  * Carleo et al. (2019) Carleo, G., Cirac, I., Cranmer, K., et al. 2019, Reviews of Modern Physics, 91, 045002
  * Cieza et al. (2019) Cieza, L. A., Ruíz-Rodríguez, D., Hales, A., et al. 2019, MNRAS, 482, 698
  * Dieleman et al. (2015) Dieleman, S., Willett, K. W., & Dambre, J. 2015, MNRAS, 450, 1441
  * Draine & Lee (1984) Draine, B. T. & Lee, H. M. 1984, ApJ, 285, 89
  * Dunjko & Briegel (2018) Dunjko, V. & Briegel, H. J. 2018, Reports on Progress in Physics, 81, 074001
  * Eistrup et al. (2016) Eistrup, C., Walsh, C., & van Dishoeck, E. F. 2016, A&A, 595, A83
  * Fedele & Favre (2020) Fedele, D. & Favre, C. 2020, A&A, 638, A110
  * Garrod (2013) Garrod, R. T. 2013, ApJ, 765, 60
  * Garufi et al. (2020) Garufi, A., Podio, L., Codella, C., et al. 2020, A&A, 636, A65
  * Goldberger et al. (2005) Goldberger, J., Hinton, G. E., Roweis, S., & Salakhutdinov, R. R. 2005, in Advances in Neural Information Processing Systems, ed. L. Saul, Y. Weiss, & L. Bottou, Vol. 17 (MIT Press)
  * Grassi et al. (2014) Grassi, T., Bovino, S., Schleicher, D. R. G., et al. 2014, MNRAS, 439, 2386
  * Grassi et al. (2021) Grassi, T., Nauman, F., Ramsey, J. P., et al. 2021, arXiv e-prints, arXiv:2104.09516
  * Guzmán et al. (2021) Guzmán, V. V., Bergner, J. B., Law, C. J., et al. 2021, ApJS, 257, 6
  * Holdship & Viti (2022) Holdship, J. & Viti, S. 2022, A&A, 658, A103
  * Holdship et al. (2021) Holdship, J., Viti, S., Haworth, T. J., & Ilee, J. D. 2021, A&A, 653, A76
  * Holdship et al. (2017) Holdship, J., Viti, S., Jiménez-Serra, I., Makrymallis, A., & Priestley, F. 2017, AJ, 154, 38
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90
  * Ilee et al. (2021) Ilee, J. D., Walsh, C., Booth, A. S., et al. 2021, ApJS, 257, 9
  * Keil et al. (2022) Keil, M., Viti, S., & Holdship, J. 2022, ApJ, 927, 203
  * Lee et al. (2021) Lee, K. L. K., Patterson, J., Burkhardt, A. M., et al. 2021, ApJ, 917, L6
  * Molyarova et al. (2018) Molyarova, T., Akimkin, V., Semenov, D., et al. 2018, ApJ, 866, 46
  * Molyarova et al. (2017) Molyarova, T., Akimkin, V., Semenov, D., et al. 2017, ApJ, 849, 130
  * Öberg et al. (2011) Öberg, K. I., Boogert, A. C. A., Pontoppidan, K. M., et al. 2011, ApJ, 740, 109
  * Öberg et al. (2021) Öberg, K. I., Guzmán, V. V., Walsh, C., et al. 2021, ApJS, 257, 1
  * Padovani et al. (2018) Padovani, M., Ivlev, A. V., Galli, D., & Caselli, P. 2018, A&A, 614, A111
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., et al. 2011, Journal of Machine Learning Research, 12, 2825
  * Pegues et al. (2021) Pegues, J., Öberg, K. I., Bergner, J. B., et al. 2021, ApJ, 911, 150
  * Pegues et al. (2020) Pegues, J., Öberg, K. I., Bergner, J. B., et al. 2020, ApJ, 890, 142
  * Reback et al. (2021) Reback, J., McKinney, W., jbrockmendel, et al. 2021, pandas-dev/pandas: Pandas 1.2.4
  * Ribas et al. (2020) Ribas, Á., Espaillat, C. C., Macías, E., & Sarro, L. M. 2020, A&A, 642, A171
  * Ruaud et al. (2016) Ruaud, M., Wakelam, V., & Hersant, F. 2016, MNRAS, 459, 3756
  * Semenov et al. (2010) Semenov, D., Hersant, F., Wakelam, V., et al. 2010, A&A, 522, A42
  * Semenov & Wiebe (2011) Semenov, D. & Wiebe, D. 2011, ApJS, 196, 25
  * Smirnov-Pinchukov et al. (2020) Smirnov-Pinchukov, G. V., Semenov, D. A., Akimkin, V. V., & Henning, T. 2020, A&A, 644, A4
  * Vasyunin et al. (2009) Vasyunin, A. I., Semenov, D. A., Wiebe, D. S., & Henning, T. 2009, ApJ, 691, 1459
  * Villadsen et al. (2022) Villadsen, T., Ligterink, N. F. W., & Andersen, M. 2022, arXiv e-prints, arXiv:2207.03906
  * Willacy et al. (1998) Willacy, K., Klahr, H. H., Millar, T. J., & Henning, T. 1998, A&A, 338, 995
  * Williams & Best (2014) Williams, J. P. & Best, W. M. J. 2014, ApJ, 788, 59
  * Woitke et al. (2009) Woitke, P., Kamp, I., & Thi, W. F. 2009, A&A, 501, 383
  * Yorke & Bodenheimer (2008) Yorke, H. W. & Bodenheimer, P. 2008, in Astronomical Society of the Pacific Conference Series, Vol. 387, Massive Star Formation: Observations Confront Theory, ed. H. Beuther, H. Linz, & T. Henning, 189
  * Yu et al. (2016) Yu, M., Willacy, K., Dodson-Robinson, S. E., Turner, N. J., & Evans, Neal J., I. 2016, ApJ, 822, 53

##  Appendix A Additional figures

($\log_{10}X_{c}$) (bias, dex) (std, dex) (histogram)

Figure 3: Performance of ML-accelerated chemistry predictions for HCO+, DCO+,
and electrons.

($\log_{10}X_{c}$) (bias, dex) (std, dex) (histogram)

Figure 4: Effect of adding UV radiation strength to the set of input features.
The panels show the performance of ML-accelerated chemistry predictions, as in
Fig. 2. Adding UV radiation strength increases the accuracy of the fit, but
only slightly in the areas of the parameter space dominated by the selected
molecules. Depending on the molecular species and constraints on calculation
time, it is not necessary to use UV as a parameter.

Fig. 3 demonstrates the application of the method on HCO+, DCO+, and electrons
for three input features: local temperature, gas density, ionization rate. On
the Fig. 4 we present the result for the same molecules and CO with the local
UV radiation strength added to the features list. In addition, we used our
study to demonstrate the summary of the same method application to a larger
number of different species, with four input features in Fig. 5.

($\log_{10}X_{c}$) (std, dex) ($\log_{10}X_{c}$) (std, dex)

Figure 5: Application of the same method to a broader set of species.
Performance of ML-accelerated chemistry prediction, same as in Fig. 2, based
on four input features.

Figure 3: continued.

Figure 3: continued.