# ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of
Long and Short Summaries

Raian Rahman1, Rizvi Hasan1, Abdullah Al Farhad1, Md Tahmid Rahman Laskar2,3,
Md. Hamjajul Ashmafee1, Abu Raihan Mostofa Kamal1
1Department of Computer Science and Engineering, Islamic University of
Technology, Bangladesh
2School of Information Technology, York University, Canada
3Dialpad Canada Inc., Canada
{raianrahman, rizvihasan, alfarhad, ashmafee<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Automatic chart to text summarization is an effective tool for the visually
impaired people along with providing precise insights of tabular data in
natural language to the user. A large and well-structured dataset is always a
key part for data driven models. In this paper, we propose ChartSumm: a large-
scale benchmark dataset consisting of a total of 84,363 charts along with
their metadata and descriptions covering a wide range of topics and chart
types to generate short and long summaries. Extensive experiments with strong
baseline models show that even though these models generate fluent and
informative summaries by achieving decent scores in various automatic
evaluation metrics, they often face issues like suffering from hallucination,
missing out important data points, in addition to incorrect explanation of
complex trends in the charts. We also investigated the potential of expanding
ChartSumm to other languages using automated translation tools. These make our
dataset a challenging benchmark for future research.

##  1 Introduction

Automatic chart summarization is a task where the goal is to describe
important data points and trends in a chart in natural language. Chart
summaries are helpful to better interpret the chart, making it useful for the
visually impaired people as well as to improve the performance of different
information retrieval algorithms (Obeid and Hoque, 2020; Carenini et al.,
2013; Li et al., 2013).

Scarcity of large scale well defined datasets with chart image, metadata and
well described summaries is a major challenge in automatic chart
summarization. To our best knowledge, there are only four datasets (Obeid and
Hoque, 2020; Zhu et al., 2021a; Hsu et al., 2021; Kanthara et al., 2022)
available for the chart to text summarization task, making this task a low
resource problem. Among these datasets, three of them (Obeid and Hoque, 2020;
Zhu et al., 2021b; Kanthara et al., 2022) contain chart images with metadata
and well defined summaries while the SciCAP dataset (Hsu et al., 2021) only
contains chart images and captions.

In this work, we address the scarcity of public datasets in the automatic
chart summarization task. We propose “ChartSumm”, a large scale dataset for
chart to text summarization comprising of $84,363$ chart images with
corresponding chart metadata and summaries. (see Figure 1 for an example). We
also propose two test sets based on the summary length. In this paper, our
major contributions are summarized below:

Figure 1: An example chart-summary pair from our proposed dataset.

(i) Proposing a new benchmark dataset for the automatic chart summarization
task. To our best knowledge, our ChartSumm dataset is currently the largest
dataset proposed for this task. Meanwhile, we also introduce two different
test sets to separately compare the performance on generating short and long
summaries.

Dataset Name | Task | Data Source | Formulation | Summary Type | Example Count
---|---|---|---|---|---
SciCap (Hsu et al., 2021) | Image → Text caption | Scientific Papers |  | Chart Image
---
Captions
Short captions from scientific paper figures | 290,000
Chart2Text (Obeid and Hoque, 2020) | Table → Text description | Statista |  | Chart Image
---
Metadata
Description
Descriptive human written summary | 8,305
AutoChart (Zhu et al., 2021a) | Table → Text description | Different public repository |  | Chart Image
---
Metadata
Description
Automatic template generated summary | 23,543
Chart-To-Text (Kanthara et al., 2022) |  | 1\. Table → Text description
---
2\. Image → Extracted metadata using OCR → Text description
| Statista
---
Pew
| Chart Image
---
Metadata
Description
| Descriptive human written summary from
---
pew and statista
44,085
ChartSumm (ours) | Table → Text description |  | Statista
---
Knoema
| Chart Image
---
Metadata
Description
| System generated short summary from Knoema
---
and descriptive human written summary from staista
84,363

Table 1: Comparison between existing datasets and our proposed dataset.

(ii) Conducting a series of experiments using strong baselines to demonstrate
how models trained on our dataset have better generalization capability than
other existing datasets. In addition, we identify the limitations of the
state-of-the-art models in our proposed dataset. Furthermore, we also explore
the scope of expanding our dataset to other languages through translation and
evaluate the performance in a human-annotated test set in the Bengali
language. To our best knowledge, this is the first work that investigated the
Chart Summarization task in any languages except English. The dataset and
codes are available at https://github.com/pranonrahman/ChartSumm.

##  2 Related Work

Existing Chart-To-Text summarization systems generate summaries from either
the chart image Hsu et al. (2021) or the chart metadata Gong et al. (2019);
Obeid and Hoque (2020); Kanthara et al. (2022). Before the advent of deep
learning, most early work utilized a two stage approach that applied content
selection using different statistical tools in the first step followed by
generating summaries using pre-defined templates (Reiter, 2007; Zhu et al.,
2021b). However, predefined template-based architectures frequently lack
generality and fail to capture complex trends in data. In recent years, deep
learning-based techniques have gained significant attention (Gong et al.,
2019; Obeid and Hoque, 2020; Gajbhiye and Lopes, 2021; Zhu et al., 2021a; Hsu
et al., 2020; Zhou et al., 2021; Dadhich et al., 2021; Sreevalsan-Nair et al.,
2021; Luo et al., 2021; Chai et al., 2021; Kanthara et al., 2022) due to their
superior performance over the template-based approaches. Nonetheless, due to
the lack of Chart-To-Text summarization datasets, not only that the models
proposed for this task require improvements, the generalized effectiveness of
these models is also yet to be investigated.

Among the four benchmark Chart-To-Text summarization datasets that are
publicly available, the Chart2Text (Obeid and Hoque, 2020) summarization
dataset is the first dataset proposed for this task that includes 8,305
samples collected from the Statista111https://www.statista.com/. However, the
size of this dataset is quite small and so effective data-driven methods
cannot be trained on top of that. Later, the SciCAP (Hsu et al., 2021) data
was proposed for the chart captioning task from chart images. Thus, it is not
suitable for methods that can only generate summary from metadata. The
recently proposed AutoChart Zhu et al. (2021b) dataset is based on some
predetermined templates and so this dataset does not contain much variance in
the chart descriptions. More recently, Kanthara et al., Kanthara et al. (2022)
proposed the Chart-To-Text dataset that consists of chart images, with their
corresponding metadata and human written descriptions. Though this dataset is
currently the largest dataset available for this task containing 44,085 charts
that were collected from the Statista website and the Pew website, our
proposed ChartSumm dataset is almost double in size than the Chart-To-Text
dataset.

##  3 The ChartSumm Dataset

This section describes how we compile a large-scale dataset consisting of
$84,363$ examples from the Knoema222https://knoema.com/atlas and the Statista
website for the chart to text summarization task along with their analysis.

###  3.1 Dataset Construction

Knoema: It is a statistical service-based online platform that contains the
economic indicator of more than 200 countries. Knoema provides a short
description for each statistic generated by its digital data assistant named
Yodatai333https://yodatai.knoema.com/ that summarizes basic information about
datasets. To construct our dataset, we first crawl over 1,10,000 statistics
from Knoema. Then, we filter out the statistics where the source of data is
not publicly available, resulting in 43,179 publicly available statistics.
Afterward, we collect the chart metadata and their corresponding short
descriptive captions. Since the statistics in Knoema are shown with respect to
the year, we classify each chart as a simple line chart. The title and caption
of the chart are then tokenized while we remove white spaces and newlines
using stemming. We also normalize the numerical entities.

Statista: It is also an online platform where statistics on a wide range of
topics are published along with a short human-written description of the
statistics. Topics in Statista include economics, marketing, industry, and
opinion research. For dataset creation, at first, we crawl over $750,000$
available pages in Statista research to collect a list of $41,184$ publicly
available charts along with summaries and chart metadata. Then we classify the
data into simple and complex charts depending on the number of columns in the
chart. Similar to the Knoema dataset, we also apply tokenization and stemming.
Since many examples in the Statista dataset did not contain the x_label, we
apply the following heuristic rules to automatically classify the x_labels as
Year, Month, Day, Quarter, Country, City, and Continent:

  * •

Year: If all x values were integers less than 2050 and greater than 1800 we
set the x label to “year”.

  * •

Month: If all x values were names of months, we set the x label to “month”.

  * •

Day: If all x values were names of days(saturday, sunday …), we set the x
label to “month”.

  * •

Quarter: If most x values were Q1/Q2/Q3/Q4 and optionally followed by an
integer(year) we set x label as “quarter”.

  * •

Country: If more than 30% x values contained values from the list of all
countries collected from Wikipedia, we labeled the x label as “country”

  * •

City: If more than 30% x values contained values from the list of cities
collected from World City Database, we labeled the x label as “city”

  * •

Area: If the x labels contained the names of general areas like continent
names, sub continent names, etc, we set the x label as “area”.

  * •

NER: We also used named entity recognition to identify some other named types
such as companies, social medias etc.

For charts where the x_labels could not be automatically determined, the value
for the x_label is set to “x_label”.

To classify the charts into different types (bar/line/pie), we use
ChartReader444https://github.com/Cvrane/ChartReader. The charts are then
divided into simple and complex categories. We need to find missing x labels
since some of the scrapped data have missing x labels. So we manually identify
them using the following methods.

Figure 2: Topic distribution of ChartSumm

###  3.2 Dataset Analysis

In this section, we analyze our proposed ChartSumm dataset. At first, we
compare our ChartSumm dataset with some existing datasets in Table 1. We find
the both Chart2Text (Obeid and Hoque, 2020) and Chart-To-Text (Kanthara et
al., 2022) collected their data from a source called Statista in which the
summaries are bit descriptive and longer in length. Whereas our dataset
contains both long and short summaries.

Note that our proposed dataset contains line charts, bar charts, and pie
charts. For Statista. bar chart is the most common type ($64.70\%$ in
statista) followed by line chart ($33.76\%$ in statista) followed by pie chart
($1.54\%$ in statista). For knoema, all charts are line charts. In Figure 2,
we show the topic distribution of our dataset. For topic modeling, we perform
Latent Dirichlet allocation (LDA) (Blei et al., 2003) by creating a topic per
document and a word per topic model, both of which are based on Dirichlet
distributions. We find that our proposed ChartSumm dataset covers a large
spectrum of topics including Economy & politics ($21.60\%$), Society & Science
($13.03\%$), Internet & Media ($11.43\%$), Public life & Health ($10.42\%$),
Sports & Entertainment ($9.14\%$), Consumer Goods ($7.71\%$), Retail & Trade
($5.35\%$), Education ($5.32\%$), etc.

Source | Average Cell Count | Average Summary Length | Average Title Length
---|---|---|---
Token | Chars | Token | Chars
Knoema | 55.44 | 34.76 | 207.69 | 8.86 | 57.18
Statista - simple | 13.31 | 46.96 | 288.68 | 9.58 | 63.08
Statista - complex | 37.95 | 55.54 | 340.19 | 10.54 | 67.08

Table 2: Dataset analysis of ChartSumm.

In Table 2, we show the average cell counts for the tables, as well as
character and token counts for summaries and titles, respectively. We find
that in terms of average number of tokens, the summaries of simple and complex
Statista charts were about $35\%$ and $59.9\%$ longer than the Knoema charts,
respectively. The data obtained from each source was classified into train,
validation, and test sets, following an 80:10:10 split ratio. We show the
number of samples in our training, validation, and test sets in table 4.
Since, one source of our dataset is Statista that is similar to the Chart-To-
Text Kantharaj et al. (2022b) dataset, we measure the overlaps between the
data samples from Statista in our ChartSumm dataset and the Chart-To-Text
dataset. For similarity measurement, we first tokenize the captions and then
calculate the percentage of matched tokens. We assume two samples are exactly
similar when the similarity is greater than $90\%$. Table 3 shows that only
$5,338$ captions in our dataset overlaps with the samples from Statista in
Chart-To-Text.

| ChartSumm | Chart-To-Text | Overlaps
---|---|---|---
Simple | 33067 | 27868 | 4144
Complex | 8338 | 6943 | 1194
Total | 41405 | 34811 | 5338

Table 3: Overlaps between Chartsumm and Chart-To-Text Statista samples.

Train-k | Train-s | Valid-k | Valid-s | Test-k | Test-s | Total
---|---|---|---|---|---|---
34,503 | 32,985 | 4,338 | 4,101 | 4,338 | 4,098 | 84,363

Table 4: Split distribution of ChartSumm.

##  4 Experiments

In this section, we present the baseline models that we utilize to benchmark
the performance in our proposed dataset, followed by the fine-tuning process,
the evaluation metrics, and the experimental results.

###  4.1 Baselines:

We use T5-Base Raffel et al. (2019) and BART (Lewis et al., 2020) as our
baselines due to their effectiveness in Chart-To-Text tasks Kantharaj et al.
(2022b). T5 is a large pre-trained language model trained on multiple
sequence-to-sequence tasks. BART is a sequence-to-sequence model pre-trained
for the language modeling task using the denoising autoencoder architecture.
We fine-tune three variants of BART: (i) BART-Base, (ii) BART-Large-CNN, and
(iii) BART-Large-XSUM. We implement all models using HuggingFace Wolf et al.
(2020). Below we describe our model fine-tuning process.

###  4.2 Fine-tuning Process

We used chart metadata (title, corresponding data table, labels) to fine-tune
all four of our pre-trained baseline models. We flattened the table by rows
and concatenated it with the caption of the table separated by a separator
token. To mimic the pretraining process of T5, we added the prefix: “Summarize
chart: ” before each example. Figure 3 shows the fine-tuning stages.

All four baseline models were fine-tuned for $3$ epochs with a batch size of
$8$. The initial learning rate during our fine-tuning was $1e-6$. We used
AdamW Kingma and Ba (2014); Loshchilov and Hutter (2018) as our optimizer and
cross-entropy as the loss function. We used Google
Colab555https://colab.research.google.com/ for our experiments.

Figure 3: Fine-tuning process of baseline models

|  |  BLEU (↑) |  BLEURT (↑) |  CIDER (↑) |  CS (↑) |  PPL (↓)
---|---|---|---|---|---|---
Fine-tuned on | Model | Test-S | Test-K | Chart-To-Text | Test-S | Test-K | Chart-To-Text | Test-S | Test-K | Chart-To-Text | Test-S | Test-K | Chart-To-Text | Test-S | Test-K | Chart-To-Text
| T5-Base | 18.06 | 8.89 | 14.98 | -0.2466 | -0.9940 | -0.0969 | 2.735 | 1.143 | 3.239 | 68.86 | 55.2 | 67.2 | 10.8742 | 8.2657 | 11.3778
| BART-Base | 13.3 | 6.79 | 12.99 | -0.0483 | -0.4208 | 0.0080 | 3.028 | 1.304 | 3.283 | 63.03 | 55.43 | 66.1 | 11.0813 | 9.4579 | 9.9658
| BART-Large-CNN | 22.79 | 6.34 | 22.98 | 0.0685 | -0.5161 | 0.0462 | 2.923 | 1.051 | 3.111 | 83.76 | 66.49 | 85.41 | 9.9658 | 6.7668 | 8.248
Chart to Text Kanthara et al. (2022) | BART-Large-XSUM | 22.18 | 6.58 | 21.35 | 0.0496 | -0.4798 | 0.0259 | 2.866 | 1.067 | 2.973 | 83.23 | 65.61 | 85.15 | 9.2262 | 6.996 | 8.1973
| T5-Base | 26.86 | 48.84 | 19.15 | 0.1942 | 0.2400 | 0.0867 | 3.82 | 4.149 | 3.136 | 77 | 77.25 | 71.23 | 8.1932 | 4.9536 | 8.9733
| BART-Base | 11.69 | 21.23 | 11.58 | -0.0265 | -0.0259 | -0.0305 | 3.479 | 3.073 | 3.134 | 65.16 | 70.04 | 66.61 | 10.0825 | 10.5109 | 11.7486
| BART-Large-CNN | 29.91 | 24.37 | 22.87 | 0.1869 | -0.0276 | 0.0910 | 3.401 | 2.614 | 2.992 | 85.33 | 76.59 | 84.52 | 8.7723 | 11.6479 | 9.3914
ChartSumm | BART-Large-XSUM | 31.73 | 26.15 | 24.94 | 0.1963 | 0.0030 | 0.1107 | 3.551 | 2.718 | 3.092 | 85.58 | 77.2 | 84.02 | 9.1612 | 11.262 | 9.1076
| T5-Base | 10.1 | 50.74 | 7.6 | -0.2283 | 0.2624 | -0.2753 | 2.047 | 4.31 | 1.964 | 61.36 | 77.63 | 55.38 | 14.1075 | 5.1099 | 12.8642
| BART-Base | 7.29 | 37.57 | 6.79 | -0.1128 | 0.2255 | -0.1471 | 1.443 | 3.865 | 1.431 | 54.62 | 77.17 | 53.13 | 10.6306 | 9.0209 | 8.7061
| BART-Large-CNN | 10.31 | 29.83 | 10.21 | -0.2537 | 0.0085 | -0.2546 | 1.594 | 3.036 | 1.667 | 69.62 | 81.24 | 72.22 | 24.3287 | 12.2263 | 15.1434
ChartSumm-k | BART-Large-XSUM | 7.12 | 43.27 | 6.12 | -0.1569 | 0.3416 | -0.2128 | 1.672 | 4.422 | 1.638 | 58.72 | 80.06 | 56.45 | 18.7177 | 8.5596 | 14.015
| T5-Base | 26.84 | 14.39 | 21.11 | 0.1970 | -0.1449 | 0.1328 | 3.811 | 1.656 | 3.435 | 76.98 | 62.95 | 73.8 | 8.2032 | 5.6382 | 7.9724
| BART-Base | 21.93 | 10.23 | 17.25 | 0.1582 | -0.1622 | 0.0978 | 3.349 | 1.341 | 3.219 | 74.78 | 62.93 | 71.7 | 9.4447 | 6.2375 | 9.7854
| BART-Large-CNN | 32.27 | 7.6 | 28.71 | 0.1998 | -0.3784 | 0.1628 | 3.563 | 1.101 | 3.235 | 85.33 | 66.97 | 83.6 | 9.4005 | 6.9521 | 9.5459
ChartSumm-s | BART-Large-XSUM | 31.67 | 10.67 | 25.42 | 0.2832 | -0.1178 | 0.2377 | 3.929 | 1.271 | 3.563 | 81.26 | 65.2 | 78.97 | 9.315 | 5.9528 | 9.572

Table 5: Experimental results on different test sets. Here, ‘↑’ means ‘Higher’
and ‘↓’ means ‘Lower’ value is better.

|  Gold: Between 1992 and 2018, Georgia renewable surface water remained stable at around 62.1 billion cubic meters per year. |  T5-Base: In 2018 , renewable surface water for Georgia was 62.1 million cubic meters  per year.  Though Georgia renewable surface water fluctuated substantially in recent years , it tended to increase through 1997 - 2018 period ending at 62.1 million cubic meters per year in 2018 . |  Comments: Model predicts factually incorrect information and captures incorrect trend.
---|---|---|---
|  Gold: A survey from April 2020 revealed that during the coronavirus (COVID-19) lockdown about three out of ten Italian adults subscribed to paid digital services, with slightly higher penetration rates among male respondents. 20 percent of male adults subscribed to video on demand platforms, while just 14 percent of female respondents did the same. |  BART-Base: This statistic shows the number of paid digital subscriptions during the coronavirus outbreak in Italy in 2020, by user gender. |  Comments: Model fails to generate informative summary.
|  Gold: This statistic depicts the gross profit of Nautilus Inc. worldwide from 2014 to 2020 . In 2020 , Nautilus Inc. had a gross profit amounting to just under 230 million U.S. dollars , which is a significant increase compared to 2019 . |  BART-Large-CNN: The statistic shows the gross profit of Nautilus Inc. worldwide from 2014 to 2020 . In 2020 , the company ’s global gross profit amounted to approximately 228.8 million U.S. dollars .  The company is headquartered in Seattle , Washington , United States . |  Comments:  Model hallucinated about its headquarter.

Table 6: Examples of generated summaries containing error.  Red  indicates
factually incorrect output,  orange  indicates failure to capture trend and
violet  indicates hallucination error

###  4.3 Evaluation Metrics:

We use five evaluation metrics in our automated evaluation: (i) BLEU (Post,
2018): it uses n-gram overlaps between reference text and machine-generated
text to determine similarity score, (ii) BLEURT (Sellam et al., 2020): it
evaluates how fluent the candidate is and how well it transfers the
reference’s meaning (we utilize BLEURT base-128 for our evaluation), (iii)
Perplexity: it is a measurement that quantifies how well a probability model
predicts a sample (we utilized pre-trained GPT-2 (Radford et al., 2019) to
measure perplexity), (iv) CIDEr: (Vedantam et al., 2015) it uses n-grams
overlaps and calculates average cosine similarity between the candidate
sentence and the reference sentences, to capture the grammatical qualities
with richer semantics, (v): Content Selection (CS): it measures how closely
the generated text matches the reference documents (Wiseman et al., 2017).

Fine-tuned On | Test-k | Test-s
---|---|---
BLEU-1 | BLEU-2 | BLEU-N | BLEU-1 | BLEU-2 | BLEU-N
Full dataset | 17.07 | 6.41 | 3.22 | 28.45 | 12.28 | 7.69
Knoema | 17.80 | 6.62 | 3.79 | 12.02 | 3.60 | 1.25
Statista | 15.44 | 5.02 | 1.73 | 28.93 | 12.63 | 7.97

Table 7: Experimental results on different test sets of ChartSumm translated
to Bengali.

###  4.4 Experimental Results:

To evaluate the performance, we use three test sets: (i) test-k: denotes
ChartSumm test set compiled from Knoema where the summaries are precise and
well structured, (ii) test-s: denotes ChartSumm test set compiled from
Statista where the summaries are descriptive and have a longer length, and
(iii) Chart-To-Text (Statista): the Statista version of the test set of Chart-
To-Text Kantharaj et al. (2022b). We run experiments using the full training
sets of both ChartSumm and Chart-To-Text, with additional experiments with the
training subsets of Statista and Knoema from ChartSumm.

Our experimental results are shown in Table 5. In terms of BLEURT, CIDER, and
CS metrics, we observe that BART-Large models (Bart-Large-CNN and Bart-Large-
XSUM) outperform other models in all three test sets, while T5 performs the
best in terms of PPL in all test sets. Furthermore, we observe that models
fine-tuned on ChartSumm-K always perform better than other models in test sets
containing data from Knoema only (except the PPL metric). We again observe
similar trends in terms of models fine-tuned on ChartSumm-S. More importantly,
all models fine-tuned on our ChartSumm-S dataset outperform the baselines that
are fine-tuned on Chart-To-Text even in the Chart-To-Text test set, showing
the effectiveness of our proposed dataset. Meanwhile, models fine-tuned on
Chart-To-Text performs very poorly in ChartSumm-K (about 82.48% lower in terms
of best performing T5 model), indicating that the Chart-To-Text dataset is not
generalizable to generate summaries that are shorter and precise. To further
investigate the performance, we do some error analyses in the following
section.

###  4.5 Error Analysis and Challenges:

For error analysis, we randomly sampled $100$ instances with their summaries
generated by different baseline models.

We notice that in many cases, even though the generated summary is fluent and
readable, it contains factually incorrect information and predicted wrong
trend in data (see the first example in Table 6. We also notice that models
sometimes fail to generate informative summaries while also failing to predict
anything about data (see the second example in Table 6). In both Bart-Large
models, we also find some cases where the models generate information that is
fully irrelevant to the chart (i.e., the Hallucination effect (Gong et al.,
2019; Obeid and Hoque, 2020; Wiseman et al., 2021)) (see the third example in
Table 6).

###  4.6 Evaluation of ChartSumm in Other Languages

To assess the potential for expanding the use of ChartSumm to other languages,
we undertook a study where we translated ChartSumm into Bengali and fine-tuned
a pre-trained mT5 Xue et al. (2021) model to evaluate its performance. To our
best knowledge, this is the first study that has explored the task of
summarizing charts in languages other than English.

Translation: To translate the training and validation sets of ChartSumm into
Bengali, we utilized NLLB Costa-jussà et al. (2022), which is a state-of-the-
art neural machine translation model. To ensure proper evaluation, the test
data was translated into Bengali with the assistance of human annotators who
are undergraduate students and have proficiency in both English and Bengali.

Baseline: In our study, we employed mT5 Xue et al. (2021) as a baseline model
due to its efficacy in text summarization tasks in Bengali. We fine-tuned a
variant of mT5 which was pre-trained on multilingual XL-SUM Hasan et al.
(2021). Our baseline model was fine-tuned for 4 epochs using a batch size of
8, with an initial learning rate of 0.000001. The AdamW optimizer was used for
the fine-tuning process and cross-entropy was utilized as the loss function.

Evaluation: In our study, we conducted automatic evaluation of the text
summarization models using the BLEU, which measures the degree of similarity
between the reference and the machine-generated text using n-gram overlaps.
However, we were unable to employ other evaluation metrics such as BLEURT and
CIDEr for the Bengali language, as they require language-specific models that
are not available for Bengali. Therefore, we only used BLEU as an evaluation
metric for our Bengali text summarization experiment. We present the outcomes
of our experiments in Table 7. It is evident from the results that models
fine-tuned on ChartSumm-s and ChartSumm-k performed better on their
corresponding test sets in comparison to the scenario when evaluated on the
combined test set. From this, we can see that the model can perform better in
the respective test sets even with machine-generated translations. This opens
up the possibility to investigate the performance of ChartSumm in other
languages through automatic machine translations. Meanwhile, performance
evaluation via human-annotated training data in other languages is also
something worth investigating in the future.

##  5 Conclusion

In this work, we present a new large scale benchmark dataset for the automatic
chart summarization task to address the low resource problem in such tasks.
Our proposed dataset is almost double in size than the existing largest
dataset available for this task. Thus, the proposed ChartSumm dataset will
serve as a strong benchmark for researchers in this relatively new area of
natural language generation. We utilize three BART models and one T5 model as
baselines and conduct extensive experiments using various evaluation metrics
to identify the challenges in this task. Experimental results showed that
models fine-tuned with our proposed ChartSumm dataset could achieve better
domain generalization than other existing benchmark datasets. We also explored
the possibility of extending ChartSumm to other languages through automatic
machine translation. In the future, we would like to extend ChartSumm to a
multilingual dataset to address the scarcity of well-formatted datasets in
other low-resource languages. We will also study how to incorporate query
relevance Laskar et al. (2022d, 2020a, 2020b), question-answering Masry et al.
(2022); Kantharaj et al. (2022a); Hoque et al. (2022); Laskar et al. (2020c),
and entity recognition Laskar et al. (2022c, b, a) capabilities in this task.

## References

  * Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003.  Latent dirichlet allocation.  _J. Mach. Learn. Res._ , 3(null):993–1022.
  * Carenini et al. (2013) Giuseppe Carenini, Cristina Conati, Enamul Hoque, and Ben Steichen. 2013.  User task adaptation in multimedia presentations.  In _UMAP Workshops_. Citeseer.
  * Chai et al. (2021) Chengliang Chai, Guoliang Li, Ju Fan, and Yuyu Luo. 2021.  Crowdchart: Crowdsourced data extraction from visualization charts.  _IEEE Transactions on Knowledge and Data Engineering_ , 33(11):3537–3549.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Dadhich et al. (2021) Komal Dadhich, Siri Chandana Daggubati, and Jaya Sreevalsan-Nair. 2021.  Barchartanalyzer: Digitizing images of bar charts.  In _BarChartAnalyzer_.
  * Gajbhiye and Lopes (2021) Srushti Gajbhiye and Maria Lopes. 2021.  Template-based nlg for tabular data using bert.  pages 1–5.
  * Gong et al. (2019) Li Gong, Josep M Crego, and Jean Senellart. 2019.  Enhanced transformer model for data-to-text generation.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 148–156.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * Hoque et al. (2022) Enamul Hoque, Parsa Kavehzadeh, and Ahmed Masry. 2022.  Chart question answering: State of the art and future directions.  In _Computer Graphics Forum_ , volume 41, pages 555–572. Wiley Online Library.
  * Hsu et al. (2021) Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021.  SciCap: Generating captions for scientific figures.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3258–3264, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hsu et al. (2020) Yi-Te Hsu, Sarthak Garg, Yi-Hsiu Liao, and Ilya Chatsviorkin. 2020.  Efficient inference for neural machine translation.  In _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing_ , pages 48–53, Online. Association for Computational Linguistics.
  * Kanthara et al. (2022) Shankar Kanthara, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022.  Chart-to-text: A large-scale benchmark for chart summarization.  _arXiv preprint arXiv:2203.06486_.
  * Kantharaj et al. (2022a) Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Ko Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022a.  Opencqa: Open-ended question answering with charts.  _arXiv preprint arXiv:2210.06628_.
  * Kantharaj et al. (2022b) Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022b.  Chart-to-text: A large-scale benchmark for chart summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4005–4023, Dublin, Ireland. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Laskar et al. (2022a) Md Tahmid Rahman Laskar, Cheng Chen, Xue-yong Fu, and Shashi Bhushan Tn. 2022a.  Improving named entity recognition in telephone conversations via effective active learning with human in the loop.  In _Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances)_ , pages 88–93, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Laskar et al. (2022b) Md Tahmid Rahman Laskar, Cheng Chen, Jonathan Johnston, Xue-Yong Fu, Shashi Bhushan TN, and Simon Corston-Oliver. 2022b.  An auto encoder-based dimensionality reduction technique for efficient entity linking in business phone conversations.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 3363–3367.
  * Laskar et al. (2022c) Md Tahmid Rahman Laskar, Cheng Chen, Aliaksandr Martsinovich, Jonathan Johnston, Xue-Yong Fu, Shashi Bhushan Tn, and Simon Corston-Oliver. 2022c.  BLINK with Elasticsearch for efficient entity linking in business conversations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track_ , pages 344–352, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
  * Laskar et al. (2020a) Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Huang. 2020a.  Query focused abstractive summarization via incorporating query relevance and transfer learning with transformer models.  In _Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian AI 2020, Ottawa, ON, Canada, May 13–15, 2020, Proceedings 33_ , pages 342–348. Springer.
  * Laskar et al. (2020b) Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2020b.  WSL-DS: Weakly supervised learning with distant supervision for query focused multi-document abstractive summarization.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5647–5654, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Laskar et al. (2022d) Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022d.  Domain adaptation with pre-trained transformers for query-focused abstractive text summarization.  _Computational Linguistics_ , 48(2):279–320.
  * Laskar et al. (2020c) Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque. 2020c.  Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 5505–5514.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2013) Zhuo Li, Matthew Stagitis, Sandra Carberry, and Kathleen F McCoy. 2013.  Towards retrieving relevant information graphics.  In _Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval_ , pages 789–792.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Luo et al. (2021) Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin. 2021.  Chartocr: Data extraction from charts images via a deep hybrid framework.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_.
  * Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  _arXiv preprint arXiv:2203.10244_.
  * Obeid and Hoque (2020) Jason Obeid and Enamul Hoque. 2020.  Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 138–147, Dublin, Ireland. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Reiter (2007) Ehud Reiter. 2007.  An architecture for data-to-text systems.  In _Proceedings of the Eleventh European Workshop on Natural Language Generation (ENLG 07)_ , pages 97–104, Saarbrücken, Germany. DFKI GmbH.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Sreevalsan-Nair et al. (2021) Jaya Sreevalsan-Nair, Komal Dadhich, and Siri Chandana Daggubati. 2021.  Tensor fields for data extraction from chart images: bar charts and scatter plots.  In _Topological Methods in Data Analysis and Visualization VI_. Springer.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wiseman et al. (2021) Sam Wiseman, Arturs Backurs, and Karl Stratos. 2021.  Data-to-text generation by splicing together nearest neighbors.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4283–4299, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wiseman et al. (2017) Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017.  Challenges in data-to-document generation.  _arXiv preprint arXiv:1707.08052_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zhou et al. (2021) Fangfang Zhou, Yong Zhao, Wenjiang Chen, Yijing Tan, Yaqi Xu, Yi Chen, Chao Liu, and Ying Zhao. 2021.  Reverse-engineering bar charts using neural networks.  _Journal of Visualization_ , 24(2).
  * Zhu et al. (2021a) Jiawen Zhu, Jinye Ran, Roy Ka-wei Lee, Kenny Choo, and Zhi Li. 2021a.  Autochart: A dataset for chart-to-text generation task.  _arXiv preprint arXiv:2108.06897_.
  * Zhu et al. (2021b) Jiawen Zhu, Jinye Ran, Roy Ka-Wei Lee, Zhi Li, and Kenny Choo. 2021b.  AutoChart: A dataset for chart-to-text generation task.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , pages 1636–1644, Held Online. INCOMA Ltd.