# Chart-RCNN: Efficient Line Chart Data Extraction from Camera Images

Shufan Li
Orka Labs Inc.
<EMAIL_ADDRESS>Congxi Lu
Orka Labs Inc.
<EMAIL_ADDRESS>
Linkai Li
Orka Labs Inc.
<EMAIL_ADDRESS>
Haoshuai Zhou
Orka Labs Inc.
<EMAIL_ADDRESS>Corresponding Author

###### Abstract

Line Chart Data Extraction is a natural extension of Optical Character
Recognition where the objective is to recover the underlying numerical
information a chart image represents. Some recent works such as ChartOCR
approach this problem using multi-stage networks combining OCR models with
object detection frameworks. However, most of the existing datasets and models
are based on ”clean” images such as screenshots that drastically differ from
camera photos. In addition, creating domain-specific new datasets requires
extensive labeling which can be time-consuming. Our main contributions are as
follows: we propose a synthetic data generation framework and a one-stage
model that outputs text labels, mark coordinates, and perspective estimation
simultaneously. We collected two datasets consisting of real camera photos for
evaluation. Results show that our model trained only on synthetic data can be
applied to real photos without any fine-tuning and is feasible for real-world
application.

Figure 1: An overview of our method.  We train our model on synthetic data of
chart images and perform inference on real camera photos. Our main
contributions are 1. A synthetic dataset pipeline. 2. A benchmark dataset of
camera photos. 3. An end-to-end trainable model.

##  1 Introduction

Line charts are widely used to represent data in a wide range of documents
such as financial forecasts, research publications, and medical reports [13,
26]. Automatic analysis of line charts can facilitate document mining, medical
diagnosis, and help the visually impaired understand documents. However, when
line charts are published in images, the raw data is lost. Recovering the
underlying information of line charts will improve the performance of existing
chart classification and question-answering systems such as [9, 10, 14]. It is
trivial to identify the maximum value in a line chart given the raw data, but
not so given only an image. Additionally, some analysis can only be performed
when all the raw data is available. For example, when an audiologist fits a
hearing aid for a patient given an audiogram ( a line chart representing one’s
hearing loss level at different frequencies), he or she must read the exact
values from the chart. Therefore, Line Chart Data Extraction is a task of
great significance as it improves the accuracy of qualitative analysis such as
classification and question-answering, and makes quantitative applications
such as automatic hearing-aid fitting possible.

There have been numerous methods [5, 28] based on manually designed features
and rules, however, they failed to generalize a diverse range of line chart
designs. Many recent works such as [11, 16] achieved better performance using
deep neural networks. But these systems are difficult to train in that the
input image is typically passed through a sequence of independent networks
performing different tasks such as OCR and object detection, each requiring a
separate training pipeline.

One major limitation of these works is that they focus on extracting data from
”clean” chart images such as scrapped images from the Internet, Excel sheets,
and PDF documents [11, 16, 21]. We argue that extracting chart data from real-
world camera photos is equally important, if not more. There is an analogy
between scene text recognition as an extension of optical character
recognition(OCR) and chart data extraction from camera images as an extension
of chart data extraction from ”clean” images. Just as scene text recognition
powers real-time documentation translation and text-to-speech services, chart
data extraction from camera images can power applications such as helping the
visually impaired to read charts in printed documents and automatic tuning of
hearing aids using audiograms photos taken by cellphone cameras.
Unfortunately, this natural extension of chart data extraction hasn’t been
fully explored.

We believe there are several reasons that this area is currently under-
explored. First, Deep Neural Networks can heavily overfit small datasets,
which is why current deep methods such as [16, 11] are trained on large-scale
datasets of scrapped or generated images. However, there are no large-scale
datasets of camera photos of line charts available. Additionally, collecting
and annotating such photos can be costly and time-consuming. Second, existing
methods rely on pre-trained OCR models. Camera photos introduce additional
skew, rotation, and color perturbations to texts which lead to more noise in
OCR results and make the task more challenging. Third, existing methods rely
on bounding box estimations to find chart regions and assume the x and y-axis
of the chart are perfectly aligned with that of the image. However, in real
camera photos, the chart is projected onto the image plane through a
homography transformation, which means additional components need to be added
to the system to predict such transformation. Given many of these systems are
already very complicated, this is no trivial task.

In this paper, we tackle the problem of chart data extraction from camera
images by directly addressing these challenges. Our strategy is to develop a
system that pretrains on synthetic data only but can perform inference on real
camera photos. Fig. 1 is an illustration of our proposed system. Our main
contribution can be summarized as follows. 1) We propose Chart-RCNN, an end-
to-end trainable network that extracts line chart data from camera photos. 2)
We propose a synthetic data generation pipeline that can be used to train
models capable of performing inference in real camera photos. 3) We collected
datasets of camera photos of both real and synthetic charts with annotations
of raw data. These datasets can be used to benchmark the performance of future
works in this area

##  2 Related Works

###  2.1 Line Chart Datasets

PMC [1] dataset contains real charts extracted from Open-Access publications
found in the PubMedCentral. (PMC). It contains 7,401 and 3,155 line charts in
the training and test sets respectively. However, only 1486 images contain
annotations of raw data.

FigureQA [10] is a synthetic dataset of over 100,000 images originally
purposed for question-answering tasks. They also include the metadata used
during the generation which makes it possible to use it as a chart data
extraction dataset. However, they used only five line styles and the charts do
not have any marks on them.

ExcelChart400K is published alongside ChartOCR [16]. It contains Images
generated via crawled Excel sheets and annotation of keypoints. However, they
do not provide ground truth OCR annotation nor the raw data used to generate
the line charts.

All these datasets contain only clean images. Our datasets on the contrary
contain real camera photos. We also provide annotations of raw data and a
diverse range of chart styles.

###  2.2 Line Chart Data Extraction

In typical rule-based systems such as [6, 18], histogram-based techniques and
spatial filtering are used to locate coordinate axes and data points, while
OCR is employed to detect textblocks. Then a set of handicraft algorithms is
applied to filter false detections and generate a final output. However, these
algorithms fail to generalize to the diverse range of chart designs. To
address this issue, some systems such as ChartSense [8] introduce an
interactive process where a human agent can make corrections when necessary.
However, this makes the data extraction process time-consuming and difficult
to scale.

Deep-learning-based methods such as [16, 11] outperform rule-based methods by
training on large-scale datasets. ChartOCR [16] uses CornerNet to find
keypoint proposals and Microsoft OCR API to perform text recognition. They
also include a separate QUERY network to predict if two points belong to the
same line. [11] uses a segmentation network that produces segmentation masks
for different lifestyles and uses linear programming to perform line tracing.
Text recognition is performed using STAR-Net. However, these methods assume
clean, aligned images as input and cannot handle camera photos. Additionally,
unlike our methods which train an OCR network alongside the detection network,
they rely on generic OCR models that do not take into account the prior
distributions of tick numbers.

##  3 Dataset

Dataset | Image Source | Raw Data | Style | Alignment | Background
---|---|---|---|---|---
PMC [1] | Crawled | Partial* | Many | Aligned | White
FigureQA [10] | Generated | ✓ | 5 styles | Aligned | White
ExcelChart400K[16] | Crawled | ✗ | Many | Aligned | White
SYN-Train | Generated | ✓ | 30+ styles | Perspective | Synthetic
SYN-Camera | Camera Photo | ✓ | 30+ styles | Perspective | Print
Audiogram | Camera Photo | ✓ | 2 styles | Perspective | Print
Audiogram (Scanned) | Scanned | ✓ | 2 styles | Aligned | Print
Table 1: A comparison between our dataset and existing ones. The most
significant difference is that our dataset contains more than clean images
that are crawled or generated. Figure 2: Samples from Our Proposed Dataset.
SYN-Train is an online synthetic dataset with aggressive augmentation to mimic
the look of camera photos. SYN-Test is a dataset of pretrained images for
evaluation. SYN-Camera are real camera photos of charts in SYN-Test. Audiogram
datasets consist of data generated by Noah 4, a software widely used by
audiologists. There is a camera version and a scanned version.

We propose two datasets with variants. SYN dataset consists of synthetic data
generated using matplotlib [7]. It has various mark and line styles. SYN-Train
is an online dataset with infinitely many images. SYN-Test is a fixed subset
of SYN-Train with 4,000 images. SYN-Camera contains 200 camera photos of
charts in SYN-Test. Audiogram dataset consists of 420 camera photos of
audiogram created using Noah 4[4], a software system widely used in the
hearing care industry for such purposes. 223 images have ground truth
annotation of bounding boxes of labels and marks. This dataset is used to
evaluate how our model performs in real-world scenarios and compare models
trained on synthetic datasets with those trained on manually labeled data.
Samples from all these datasets are shown in Fig. 2.

Our datasets differ from existing ones mentioned in Sec. 2 in that they
consist of camera photos or images that are visually similar to camera photos
with annotations of raw data. SYN also have a larger variety in line styles
compared with existing synthetic datasets such as [10]. Tab. 1 illustrates a
comparison between our datasets and existing ones.

Figure 3: Proposed Chart-RCNN architecture. A standard Faster-RCNN with
ResNet-FPN backbone is used to detect tick labels and marks, the out put is
then passed to Text Decoder and Mark Grouping head for text recognition and
mark clustering. A downsampled feature map from FPN is passed to Homography
Prediction head to predict the perspective transforms of the image.

###  3.1 Synthetic

We use matplotlib to randomly generate line charts for SYN dataset. The images
have a dimension of 720x720. Each of the axes has a random number of ticks
between 5 and 10. The tick labels are randomly selected from a set of common
intervals such as [0, 1], [0, 100], and [1000, 10000]. The stride between two
consecutive labels is randomly selected among a set of common values such as
0.1, 0.2, 1, 10, and 100. Numbers less than 1 are randomly converted to
percentile. Numbers larger than 1000 are randomly converted to human-readable
notations such as 5k, and 10k. The grid lines are randomly added. Each line
chart has a random number of lines between 1 and 3 with random combinations of
mark types, mark colors, and line styles. Tab. 2 illustrates all the
parameters used in data generation.

After a clean image is generated, we apply a set of aggressive augmentations
to make it visually similar to a camera photo. We first apply random Thin-
Plate-Spline (TPS) [2] using a random grid to create small-scale permutations.
We then apply a random perspective transform with a distortion scale of 0.5.
We add random Gaussian noise at multiple scales. We randomly apply color
jittering to the resulting image, and finally, randomly add a blur effect
using Gaussian Blur or Motion Blur. Figure Fig. 4 is an illustration of this
pipeline. As shown in the figure, the clean image looks visually similar to a
camera photo after these transformations.

We generate 4,000 clean images without transforms as SYN-Test dataset. We
print 200 of them on A4 paper and then took camera photos as the SYN-Camera
dataset.

We propose two datasets with variants. SYN dataset consists of synthetic data
generated using matplotlib . It has various mark and line styles. SYN-Train is
an online dataset with infinitely many images. SYN-Test is a fixed subset of
SYN-Train with 4,000 images. SYN-Camera contains 200 camera photos of charts
in SYN-Test. Audiogram dataset consists of 420 camera photos of audiogram
created using Noah 4 [4], a software system widely used in the hearing care
industry for such purposes. 223 images have ground truth annotation of
bounding boxes of labels and marks. This dataset is used to evaluate how our
model performs in real-world scenarios and compare models trained on synthetic
datasets with those trained on manually labeled data.

All these datasets have raw data available. All except for SYN-Camera also
have annotations of bounding boxes of labels and marks, alongside their
values. a

Parameter | Range
---|---
Number of Ticks | 5-10
Number of Lines | 1-3
Grid | on, off
Aspect Ratio | 0.25-1.0
Mark Style | circle, triangle, cross, diamond, …
Line Style | solid, dot, dash, dash-dot
Line Color | random
Table 2: Randomized Parameters used in data generation.

###  3.2 Audiogram

We first collected raw audiogram images generated by Noah 4 System used by
audiologists. These audiograms were printed on standard A4-sized paper. We
took a total of 420 images of these printed audiograms under various angles
and lighting conditions as the Audiogram dataset. In some photos, there are
cast shadows of other objects on the paper. In some photos, obstructions such
as pens are purposely placed on the audiogram region. In some photos, the
paper is bent or folded. These intentional occlusions and distortions make the
dataset particularly challenging. To evaluate the impact of these distortions,
we generated an additional set of 30 audiograms, printed them, and scanned
them using a laser scanner to form Audiogram-Scanned Dataset.

All audiograms have their raw data available. We picked 223 camera photos and
manually added annotations of ground truth bounding boxes of labels and marks.

##  4 Method

We propose Chart-RCNN, an extension of Faster-RCNN [20] with additional heads
attached to the ResNet-FPN backbone performing OCR, mark grouping and
homography prediction. The network is trained jointly end-to-end. As is shown
in Fig. 3, an input image first go through standard Faster-RCNN blocks which
generate bounding box predictions and classification for labels and marks. We
use RoI Align to extract features in regions corresponding to text labels and
marks, which are then passed to Text Decoder for OCR and Mark Grouping Head
for clustering. The whole feature map from FPN backbone is downsampled and
passed to a Homography Predictor to predict the perspective transforms.

Figure 4: Transforms of SYN Dataset. We visualize transformed used in data
generation process. TPS and Perspective produce local and global geometric
distortions respectively, while the Color and Noise change the visual
appearance of the image.

###  4.1 Chart Element Detection

We employ the standard Faster-RCNN head following [20]. The RoIAlign generates
a 7x7 feature map which is passed to a Two-Layer MLP with a hidden dimension
of 1024. The output is then passed to a linear layer for classification. We
employ cross-entropy loss on the classification output.

###  4.2 Homography Estimation

We estimate the perspective transformation of an image by detecting four
corners of the chart region. These four corners uniquely determine a
projective transform that sends these four points respective to (0,0), (0,1),
(1,1), (1,0) in the canonical coordinate system. The parameter of such a
transformation can be determined by solving a linear system following [17] We
explore two approaches to predicting the corners: Mask-based ones and
keypoint-based ones.

The mask-based approach pass the downsample feature maps to an FCN head [22]
consisting of two Convolution layers of size 3 and stride 1 with RELU
activation, followed by a 1x1 convolution for pixel-wise classification. We
extract polygon contours from the mask image following [24]. The polygon is
subsequently reduced to a quadrilateral using a modified version of
Douglas‐Peucker Algorithm [25] (See appendix for more implementation details).

The keypoint-based approach directly predicts the coordinates of four points
following the design of Keypoint-RCNN [3]. The architecture is similar to the
mask-based approach. The only difference is that the prediction target is a
4-channel heatmap corresponding to 4 corners.

We employ cross-entropy loss in both options.

###  4.3 Text Recognition

We explore two architecture for OCR: Transformer and CRNN. For Transformer, we
followed the design of TrOCR[15]. RoiAlign generates a feature map of 3x9 for
each text region, which are then treated as 3x9 patches and passed to the
Transformer block. For CRNN, we follow the design of [23]. The text regions
are pooled to a 32x96 feature map. The corresponding RGB information of the
original image is added to this feature map through a Residual connection. We
employ cross entropy loss for Transformer and connectionist temporal
classification(CTC) loss for CRNN.

###  4.4 Mark Clustering

The Mark Grouping Head projects the features to a space where the embedding of
the marks from the same line are close to each other. It consists of a series
of convolution blocks followed by a 2-layer MLP (See appendix for more
implementation details). We then calculate the cosine similarity of all mark
features. We assume the likelihood of two points belonging to the same line is
defined by

$\mathbb{P}(G(x,y)=1)=exp(\frac{f(x)^{T}f(y)}{\tau\lVert f(x)\rVert\lVert
f(y)\rVert})$ (1)

Where $G(x,y)$ is an indicator function that equals to 1 if and only if mark
$x$ and $y$ belongs to the same line. $f$ is the mark grouping head. $\tau$ is
a hyperparameter.

This formulation transforms the clustering problem to a binary classification
problem and naturally leads to the following loss:

$\mathcal{L}_{Group}=-\sum_{i\neq j}[G(x_{i},x_{j})log\mathbb{P}(G(x,y)=1)\\\
+log(1-\mathbb{P}(G(x_{i},x_{j})=1))]$ (2)

The final loss of the entire Chart-RCNN model is simply the sum of loss of
each module.

###  4.5 Post Processing

We gather the output from different modules. We first remap the detected label
and mark coordinates to the canonical coordinate system with four corners of
the chart remapped to (0,0), (0,1), (1,1), and (1,0). We group the labels by
their proximity to chart borders. Labels close to the top and bottom border
are considered as labels for the x-axis, whereas those close to the left and
right border are considered as labels for the y-axis. We retrieve the text
output from Text Decoder and convert them to floats. Then we fit two RANSAC
linear models to obtain a transformation from the canonical coordinate system
to the raw data values. We apply such transformations to the coordinates of
detected maps to get the raw data points. When necessary, we perform
threshold-based hierarchical clustering [19] on mark features generated from
the Mask head. When evaluating on datasets with a known number of mark
classes, we use a linear classification head to directly predict the classes
of marks and group them into lines accordingly.

##  5 Experiment Results

###  5.1 Metric

We evaluate the performance on data extraction using F1 score which is defined
as

$F1=200\times\frac{precision\times recall}{precision+recall}$ (3)

where precision and recall are calculated under a tolerance of 5% relative
error for SYN dataset and an absolute error of 5dB for Audiogram dataset.

When multiple lines are present, the precision and recall are calculated by
iterating over all possible correspondences between detected lines and ground
truth and taking the best match.

###  5.2 Data Extraction on Camera Images

We train our ChartRCNN model using a global batch size of 16 across 4 GPUs for
100 epochs. We use Adam [12] optimizer with a learning rate of $10^{-4}$ and a
weight decay of 1.0e-6. The momentum is 0.9. The learning rate is linearly
scaled up in the first 3 warmup epochs, then follows a cosine decay schedule.

We perform evaluations on 4 datasets: SYN-Test, SYN-Camera, Audiogram, and
Audiogram-Scanned. All models are trained using SYN dataset. For SYN-Test and
SYN-Camera evaluation, we trained on the online version of the SYN dataset
with infinitely many images. For Audiogram and Audiogram-Scanned, we train our
model on a subset of SYN dataset. We limit the range of randomly generated
data and the styles to match the audiogram. For example, the x-axis is
generated in a log scale instead of a linear one as the general SYN dataset.
All our methods are never trained on real camera images.

To evaluate the effectiveness of our synthetic data generation pipeline, we
compared our results with models trained on labeled images in Audiogram
dataset. This model uses a standard Faster-RCNN to detect and classify marks
and labels. The OCR is performed through a classification head by considering
a set of possible common tick labels in audiograms as separate classes. We
report the results in Tab. 3. To the best of our knowledge, there are no
comparable works capable of extracting line chart data from ill-aligned non-
clean images. Hence we were unable to compare against other works.

Figure 5: SYN as a Universal Approximation. We find several typical use-cases of line charts in different domains: 1. Material Science Paper 2. Financial Forecast 3. Audiogram 4. Specs of Electronic Components. We observe that there are some images in our randomly generated SYN-Camera dataset that share certain similarities with these images. Both images in the leftmost column contain solid lines with square and circle marks. Both images in the second most left column contain no top and right border and have a dashed line. In the third column, the style of the audiogram is an exact match for the style of a line in the bottom image. In the rightmost column, both images have lines of similar trends presented in a grid.  Dataset | Ours | Supervised
---|---|---
SYN-Test | 90.2 | -
SYN-Camera | 85.3 | -
Audiogram | 87.3 | 84.3
Audiogram-Scanned | 97.1 | 98.9
Table 3: Data Extraction on Clean and Camera Images. Numbers are F1 scores.
Our model defeats the supervised baseline trained on camera photos of
audiograms, and is comparable on scanned images.

As is shown in Tab. 3. Our model defeats the supervised baseline (+3.0)
trained on real camera photos of audiograms without using any manually
annotated data. The only human labor required is to enter a prior distribution
of data ranges and line styles. Hence, it is an effective method for such
tasks. Our model also achieved comparable high performance (97.1 vs 98.9) on
scanned audiograms. For SYN dataset, we did not compare our method against a
supervised baseline because labeling camera photos are time-consuming. We
believe in using audiograms which are generated using Noah 4 [4] that are used
by audiologists across the world in clinical practices can better demonstrate
the effectiveness of our method in practical application scenarios. Hence we
chose to label the audiograms.

###  5.3 Generalizability and Limitations

We argue that our experiments on the SYN dataset provide good intuition on how
our model generalizes on real data because of its variation in styles. To show
this, we demonstrate in Fig. 5 that charts from different domains are proper
subsets of SYN under certain parametrization. We find line charts in different
domains have corresponding images in the randomly generated SYN-Camera dataset
that are reasonably similar. Hence, the SYN-Camera dataset is a good dataset
to evaluate the performance of our model. This is also validated on a domain-
specific dataset: Audiogram, where our method outperforms the supervised
method trained on human-annotated labels. Moreover, by the law of large
numbers, the model is likely to encounter charts that look more similar
because these real-world charts are in the parameter space of our random
generation process. To validate this, we test our model pretrained on the
general SYN dataset for 200 epochs (2x schedule) without any prior knowledge
of the style nor label distribution of Audiograms. The F1 score is 71.6 on
Audiogram, suggesting a limited zero-shot transfer capability.

However, while the SYN dataset can approximate line charts in various domains,
our method still has the following limitations. First, we use mark detection
as the basis of line recognition, which is not applicable to line charts that
do not have any marks. Second, we assume the chart region has distinguishable
corners which can be used to perform homography estimation. As we have
demonstrated, in Fig. 5, a wide range of charts from different domains satisfy
our assumptions. In domain-specific applications such as audiograms or spec
sheets of electronic components, charts have a uniform style and these
assumptions are easily satisfied.

##  6 Ablation and Analysis

In this section, we evaluate the contribution of different components to the
performance of our proposed system. For all ablation experiments, we chose a
shorter schedule (20 epochs) than the full evaluation (100 epochs) due to
limited computing resources. The experiments are evaluated on Audiogram
Dataset.

###  6.1 Transforms

Transforms | F1
---|---
Baseline | 14.3
\+ Perspective | 17.7
\+ Color | 76.1
\+ Blur and Noise | 82.1
\+ TPP | 84.8
Table 4: Ablation Results of Different Transforms. All proposed transforms
lead to a positive increase in F1 scores. Color seems to be the most important
as it simulates different white balance settings of a camera.

We employ a wide range of transforms in the data transformation process as
described in Sec. 3.1 and illustrated in Fig. 4. To evaluate the importance of
each transform, we start with a baseline in which the model is trained on only
clean images just like most existing works. Then we gradually add different
transforms into the data generation process. The results are shown in Tab. 4.
All proposed transforms lead to a positive increase in F1 scores among which
Color has the most significant impact. We believe this is likely due to the
fact that it emulates the different white balancing settings of a camera and
different lighting conditions. Without such transforms, the model could easily
overfit to data with a pure white background which is rather rare in real
camera photos. Blur and Noise are the second most important transform because
it emulates the change in focal length and real-world noises captured by
camera sensors. We believe it also mitigates the overfitting problem caused by
the model attending to specific local structure that is unique to certain
fonts or the way in which a line or circle is drawn by the computer. By adding
high-frequency noises, the local structure is perturbed.

###  6.2 Number of Images

Num of Images | F1
---|---
100 | 76.7
1,000 | 81.4
10,000 | 81.9
$\infty$ | 84.8
Table 5: Ablation Results on the Number of Training Images. The performance
improves with more images.

We employ an online dataset of infinitely many images which is rare in related
works. The stochastic process also makes the training process indeterministic.
To evaluate whether this is necessary, we explored alternative options where
the model is only trained on a fixed number of pre-generated images. As is
shown in Tab. 5, the performance improves with more images. We believe that an
online dataset solves the overfitting problem because it is hard to overfit a
dataset with infinitely many images. Additionally, we observe no significant
increase in data time. Hence our method is desirable.

###  6.3 OCR

Unlike [16] which uses external OCR API or [11] which uses a trained OCR, we
fine-tune an OCR model jointly during the training. We believe this is
necessary for that it can help the model learn a prior distribution of text
labels present in specific types of charts, which can compensate for the
additional difficulty caused by noises and perspective projections in camera
photos. To validate our hypothesis, we plug in a CRNN model pretrained for
number recognition into our trained model and compare how it affects the
performance. We also benchmarked a transformer-based Text Decoder proposed in
Sec. 4.3 against CRNN.

Text Decoder | Homography Predictor | F1
---|---|---
Online CRNN | Keypoint | 84.8
Online Transformers | Keypoint | 81.9
Pretrained CRNN | Keypoint | 79.4
Online CRNN | Mask | 84.3
Table 6: Ablation Results for Text Decoder and Homography Predictor. Online
CRNN has the best performance with a lead of +2.9. Both methods for homography
estimation have comparable performance. Due to the complexity of the mask-
based method, we picked the keypoint-based one as our final design.

As shown in Tab. 6, Online CRNN has the best performance with a lead of +2.9
against the Transformer and a lead of +5.4 against trained CNN. This confirms
our hypothesis that online OCR training is necessary. While several benchmarks
showed [27, 15] that Transformers tend to outperform CNN-based architecture,
they are harder to train and require more delicate tuning. This could be why
we failed to observe an expected lead in Transformers. Due to limited
computing resources, we were unable to explore more variants of transformer-
based OCR methods.

###  6.4 Homography Estimation

We estimate homography transforms by first predicting the four corners of the
chart region. We proposed a mask-based approach and a keypoint-based approach
in Sec. 4.2. We benchmarked their performance. The results are shown in Tab.
6. While both methods achieved similar performance, the mask-based approach
requires complicated postprocessing. Hence, we find the keypoint-based
approach which is more intuitive and straightforward is more desirable.

###  6.5 Error Analysis

So far all the analyses are based on the F1 score which tests the overall
performance of our method as an integrated system. To further ablate the
source of errors, we replace part of the model output with ground truth
annotations and evaluate the performance of the remaining system. These
experiments are performed on images sampled from a manually annotated training
set of the Audiogram dataset because it requires ground truth annotations of
bounding boxes. This is ok because our models are never exposed to human
labels. However, this set of images is not used in our other experiments
because it is exposed to a supervised baseline. Hence, the numbers are not
directly comparable with our other experiments. The results are shown in Tab.
7

Architecture | F1
---|---
Chart-RCNN | 86.3
\+ GT Detection | 87.3
\+ GT OCR | 87.3
All GT | 100.0
Table 7: Error Analysis. Adding ground truth bounding boxes improves the
result while further adding ground truth OCR annotations leads to no
significant improvements.

We discovered that adding ground truth bounding boxes improves the result
while further adding ground truth OCR annotations leads to no significant
improvements. This suggests our online fine-tuning process combined with a
robust RANSAC regressor can effectively handle noises in OCR output. The
results also suggest that most error comes from homography estimation. This is
reasonable since we are setting a small tolerance of 5dB, and small errors in
keypoint estimations may propagate to the estimated transformation matrix,
which in turn translates to reprojected coordinates. This process involves
inverting and multiplying several matrices and in this process, the errors may
be amplified in a super-linear fashion.

##  7 Conclusion

One of the most significant challenges of chart data extraction from camera
photos is the cost to obtain annotated training samples. We proposed a
customizable synthetic dataset based on a matplotlib with rich styles and can
approximate chart styles in specific domains. We introduce a series of
transforms that make a clean chart looks visually similar to a camera photo.
We proposed Chart-RCNN, a unified architecture to extract data from camera
photos of line charts. We demonstrated through experiments a cost-effective
way of training such a model using no annotated data which still achieves
comparable results with models trained on human-annotated data. Evaluations
suggest our model is feasible for real-world applications such as audiogram
interpretation.

Ablation analysis suggests most errors come from the perspective estimation
process. Our future works will attempt to incorporate some of the state-of-
the-art architecture beyond 4-point-based methods. We will also explore more
possible applications such as helping the visually impaired to read charts in
printed documents and books.

## References

  * [1] Kenny Davila, Chris Tensmeyer, Sumit Shekhar, Hrituraj Singh, Srirangaraj Setlur, and Venu Govindaraju.  Icpr 2020 - competition on harvesting raw tables from infographics.  In ICPR Workshops, 2020.
  * [2] Jean Duchon.  Splines minimizing rotation-invariant semi-norms in sobolev spaces.  In Walter Schempp and Karl Zeller, editors, Constructive Theory of Functions of Several Variables, pages 85–100, Berlin, Heidelberg, 1977. Springer Berlin Heidelberg.
  * [3] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask r-cnn.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:386–397, 2020.
  * [4] HIMSA.  Noah system 4 – himsa.
  * [5] Weihua Huang, Ruizhe Liu, and C-L Tan.  Extraction of vectorized graphical information from scientific chart images.  In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 1, pages 521–525. IEEE, 2007.
  * [6] Weihua Huang and Chew Lim Tan.  A system for understanding imaged infographics and its applications.  In Proceedings of the 2007 ACM Symposium on Document Engineering, DocEng ’07, page 9–18, New York, NY, USA, 2007. Association for Computing Machinery.
  * [7] John D Hunter.  Matplotlib: A 2d graphics environment.  Computing in science & engineering, 9(03):90–95, 2007.
  * [8] Daekyoung Jung, Wonjae Kim, Hyunjoo Song, Jeongin Hwang, Bongshin Lee, Bo Hyoung Kim, and Jinwook Seo.  Chartsense: Interactive data extraction from chart images.  Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017.
  * [9] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan.  Dvqa: Understanding data visualizations via question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2018.
  * [10] Samira Ebrahimi Kahou, Adam Atkinson, Vincent Michalski, Ákos Kádár, Adam Trischler, and Yoshua Bengio.  Figureqa: An annotated figure dataset for visual reasoning.  ArXiv, abs/1710.07300, 2018.
  * [11] Hajime Kato, Mitsuru Nakazawa, Hsuan-Kung Yang, Mark Chen, and Björn Stenger.  Parsing line chart images using linear programming.  2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2553–2562, 2022.
  * [12] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [13] Ichiro Kobayashi.  Toward text based information processing: with an example of natural language modeling of a line chart.  In IEEE SMC’99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No. 99CH37028), volume 5, pages 202–207. IEEE, 1999.
  * [14] Cem Kosemen and Derya Birant.  Multi-label classification of line chart images using convolutional neural networks.  SN Applied Sciences, 2(7):1–20, 2020.
  * [15] Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Zhoujun Li, and Furu Wei.  Trocr: Transformer-based optical character recognition with pre-trained models.  ArXiv, abs/2109.10282, 2021.
  * [16] Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin.  Chartocr: Data extraction from charts images via a deep hybrid framework.  In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). The Computer Vision Foundation, January 2021.
  * [17] Wei Mou, Han Wang, Gerald Seet, and Lubing Zhou.  Robust homography estimation based on non-linear least squares optimization.  In 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 372–377. IEEE, 2013.
  * [18] Jorge Poco and Jeffrey Heer.  Reverse‐engineering visualizations: Recovering visual encodings from chart images.  Computer Graphics Forum, 36, 2017.
  * [19] Pradeep Rai and Shubha Singh.  A survey of clustering techniques.  International Journal of Computer Applications, 7(12):1–5, 2010\.
  * [20] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:1137–1149, 2015.
  * [21] Manolis Savva, Nicholas Kong, Arti Chhajta, Li Fei-Fei, Maneesh Agrawala, and Jeffrey Heer.  Revision: Automated classification, analysis and redesign of chart images.  In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 393–402, 2011.
  * [22] Evan Shelhamer, Jonathan Long, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431–3440, 2015.
  * [23] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:2298–2304, 2017.
  * [24] Satoshi Suzuki and Keiichi Abe.  Topological structural analysis of digitized binary images by border following.  Comput. Vis. Graph. Image Process., 30:32–46, 1985.
  * [25] Mahes Visvalingam and J. Duncan Whyatt.  The douglas‐peucker algorithm for line simplification: Re‐evaluation through visualization.  Computer Graphics Forum, 9, 1990.
  * [26] Samuel J Wang, Blackford Middleton, Lisa A Prosser, Christiana G Bardon, Cynthia D Spurr, Patricia J Carchidi, Anne F Kittler, Robert C Goldszer, David G Fairchild, Andrew J Sussman, et al.  A cost-benefit analysis of electronic medical records in primary care.  The American journal of medicine, 114(5):397–403, 2003.
  * [27] Krzysztof Wróbel.  Ocr correction with encoder-decoder transformer.  2021\.
  * [28] Yanping Zhou and Chew Lim Tan.  Chart analysis and recognition in document images.  In Proceedings of Sixth International Conference on Document Analysis and Recognition, pages 1055–1058. IEEE, 2001.

## Appendix

##  A.1 Implementation Details

###  A.1.1 Keypoint-based Homography Estimation

We use a modified version of keypoint prediction head of [3]. Since the local
structures of the four corners of a chart are similar, and some charts do not
have a border at corners, we add an attention mechanism so that the model can
access global positional information but mere local structures. The output of
the ResNet-FPN backbone is downsampled to a 14x14 feature map. This is
considered as a sequence of length 196. We add 2D positional encoding and pass
the sequence through 3 transformer layers. The sequence is then reshaped back
to a 14x14 feature map, which is then passed through 4 additional convolution
layers. To get the final output of a 4-channel keypoint heatmap, we use a
ConvTranspose layer of kernel size 4. The exact architecture is shown in Fig.
A1

Figure A1: Architecture of Keypoint Head . The output of ResNet-FPN is
downsampled to a 14x14 feature map. 2D Positional Encoding is then added. We
consider the feature map as a sequence of length 196 and pass it through three
transformer layers whose Multi-Head Attention Block has 8 heads and a
dimension of 256. The output of transformer layers is then reshaped back to a
feature map. It is then passed through 4 convolution layers with kernel size 3
and ReLU activation. The final output is generated by a ConvTranspose layer of
kernel size 4.

###  A.1.2 Mask-based Homography Estimation

Figure A2: Comparison Between Ours and Douglas-Peucker . The Douglas-Peucker
is typically used to simplify polygons. However, it only reduces a polygon to
a subset of the original vertices. In this example, the white line is the
polygon generated by predicted masks, and the colored regions are reduced
quadrilaterals. The four corners of the chart are actually outside the
original polygon, hence Douglas-Peucker fails to recover them.

We use the same mask prediction head following [3]. However, it is non-trivial
to recover a quadrilateral from predicted binary masks because of noises in
predictions. Typically, such tasks are performed by Douglas‐Peucker Algorithm
[25] which reduces complicated polygons to simple ones. However, it is flawed
because the vertices of the simplified polygon are always a subset of the
vertices of the original polygon. This can give sub-optimal approximations
when the four corners of a chart are not the vertices of the predicated mask.
As shown in Fig. A2, our method overcomes this problem.

The details of this procedure are described in Algorithm A1. Given an input
binary mask $M$, we first convert it to a set of contours $C$. Then we find
$c\in C$ which is the contour with the largest enclosed area. Let $h$ be its
convex hull, we set the threshold of Douglas‐Peucker as $0.5\%$ of the
circumference of $h$ and perform standard Douglas‐Peucker. This gives us a
reduced polygon $c_{approx}$. Let $E$ be the set of its edges. We want to find
a subset $E_{4}$ containing 4 edges that will be the edges of the final
reduced quadrilateral. We initialize $E_{4}$ as an empty set, then
repetitively adding the longest edge in $E$ that is not too close to an
existing edge in $E_{4}$. In particular, we consider two edges $e_{1},e_{2}$
as too close if the angle between them is less than $20$ degrees and the
minimum distance between three points (two ends and the midpoint) of one line
and an arbitrary point on the other line is less than a $1/28$ of image
dimension. With four edges, we find all their intersections and filter out
those outside the image region. We then sort the remaining points by the angle
of the vector from the centroid of all these points to a specific point. This
recovers an ordered set of four points that can be used to solve a projective
transformation matrix.

Procedure A1 Homography Estimation on Binary Mask

$M\in\\{0,1\\}^{H\times W}\,\,\,\,\text{Binary Mask}$

$p_{1},p_{2},p_{3},p_{4}\in\mathbb{R}^{2}\,\,\,\,\text{Keypoints}$

$C\leftarrow\textsc{cv2.FindContours}(M)$

$c\leftarrow\textsc{LargestContourByArea}(C)$

$h\leftarrow\textsc{cv2.ConvexHull}(c)$

$\epsilon\leftarrow 0.005\cdot length(h)$

$c_{approx}\leftarrow\textsc{DouglasPeucker}(c,\epsilon)$

$E\leftarrow\textsc{GetEdges}(c_{approx})$

$E_{4}\leftarrow\\{\\}$

while $|E_{4}|<4$ do

$e\leftarrow\textsc{GetLongestLine}(E)$

$E_{4}.\text{add}(e)$

$E.\text{pop}(e)$

$E.\text{removeCloseLines}(e)$

end while

$P\leftarrow\textsc{GetIntersections}(E_{4})$

$p_{1},p_{2},p_{3},p_{4}\leftarrow\textsc{SortByAngle}(P)$

###  A.1.3 Mark Grouping Head

Figure A3: Architecture of Mark Grouping Head . The mark features are first
projected to a different space by an MLP. We then compute the pair-wise cosine
similarity of all mark features. We consider the resulting matrix as log-
likelihood of an random variable indicating if two marks belong to the same
line and apply cross-entropy loss on it following Eq. 2.

We illustrate the architecture of our Mark Grouping Head in Fig. A3. The mark
features are first projected to a different space by an MLP. We then compute
the pair-wise cosine similarity of all mark features. We consider the
resulting matrix as the log-likelihood of a random variable indicating if two
marks belong to the same line and apply cross-entropy loss on it following Eq.
2. In Eq. 2, we introduce a temperature hyperparameter $\tau$ because the
cosine similarity is bounded in the interval $[-1,1]$. $\tau$ scales this
interval to a larger range. In batched training, we apply a mask and the loss
is only applied to the similarity between marks in the same image. We also
masked out the diagonal entries of the similarity matrix because they are
trivial. In order to add more flexibility to the output range so that the
similarity is not always centered at zero, we add a learnable bias term that
is added to the cosine similarity matrix.

##  A.2 F1 Score under Different Error Tolerance

Figure A4: F1 under difference tolerance. To further evaluate our method, we
plot the F1 score of our model on different datasets across different error
tolerance. In audiogram datasets, the tolerance is measured by dB. In SYN
dataset, the tolerance is measured by percentage.

To further evaluate our method, we plot the F1 score of our model on different
datasets across different error tolerance in Fig. A4. We noticed that in
Audiogram-Camera, we can achieve an F1 score of $95.1$ with only a $5$dB
tolerance, which is half the length of the gap between two consecutive ticks.
This suggests most errors are within a margin of $5$dB. In SYN datasets, the
F1 continues to increase as the error tolerance gets larger, suggesting a
wider distribution of errors. This is likely caused by the varying chart
styles in the dataset with different levels of difficulty.

##  A.3 More Visualization of Datasets

In Fig. A5 we provide more visualization of samples in the datasets we used.
In particular, columns SYN-Train, SYN-Camera, and Audiogram-Camera demonstrate
the similarity of synthetic data and camera photos.

Figure A5: More Visualization of Samples in Datasets. The leftmost three
columns are SYN-Train, SYN-Test, and SYN-Camera. The rightmost two columns are
Audiogram-Camera and Audiogram Scanned.


* [69] G. R. Sell, Topological dynamics and ordinary differential equations, (1971).
  * [70] J. M. Smith, Mathematical ideas in biology, CUP Archive, 1968.
  * [71] R. J. Standish, R. J. Hobbs, M. M. Mayfield, B. T. Bestelmeyer, K. N. Suding, L. L. Battaglia, V. Eviner, C. V. Hawkes, V. M. Temperton, V. A. Cramer, et al., Resilience in ecology: Abstraction, distraction, or where the action is?, Biological Conservation, 177 (2014), pp. 43–51.
  * [72] S. H. Strogatz, Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering, CRC press, 2018.
  * [73] E. H. Van Nes and M. Scheffer, Slow recovery from perturbations as a generic indicator of a nearby catastrophic shift, The American Naturalist, 169 (2007), pp. 738–747.
  * [74] T. L. Vincent and L. R. Anderson, Return time and vulnerability for a food chain model, Theoretical Population Biology, 15 (1979), pp. 217 – 231\.
  * [75] B. Walker, C. S. Holling, S. R. Carpenter, and A. Kinzig, Resilience, adaptability and transformability in social–ecological systems, Ecology and society, 9 (2004).
  * [76] J. R. Webster, J. B. Waide, and B. C. Patten, Nutrient recycling and the stability of ecosystems, in Mineral Cycling in Southeastern Ecosystems; Proceedings of a Symposium, 1975.
  * [77] S. Wieczorek, P. Ashwin, C. M. Luke, and P. M. Cox, Excitability in ramped systems: the compost-bomb instability, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 467 (2011), pp. 1243–1269.
  * [78] D. A. Wiley, S. H. Strogatz, and M. Girvan, The size of the sync basin, Chaos: An Interdisciplinary Journal of Nonlinear Science, 16 (2006), p. 015103.