11institutetext: University at Buffalo, Buffalo, NY 14228, USA
11email<EMAIL_ADDRESS>
# ChartReformer: Natural Language-Driven Chart Image Editing

Pengyu Yan  0000-0003-1584-2350 Mahesh Bhosale  0009-0003-7157-2129 Jay Lal
0009-0000-0515-1287
Bikhyat Adhikari  2222–3333-4444-5555 David Doermann  0000-0003-1639-4561

###### Abstract

Chart visualizations are essential for data interpretation and communication;
however, most charts are only accessible in image format and lack the
corresponding data tables and supplementary information, making it difficult
to alter their appearance for different application scenarios. To eliminate
the need for original underlying data and information to perform chart
editing, we propose ChartReformer, a natural language-driven chart image
editing solution that directly edits the charts from the input images with the
given instruction prompts. The key in this method is that we allow the model
to comprehend the chart and reason over the prompt to generate the
corresponding underlying data table and visual attributes for new charts,
enabling precise edits. Additionally, to generalize ChartReformer, we define
and standardize various types of chart editing, covering style, layout,
format, and data-centric edits. The experiments show promising results for the
natural language-driven chart image editing. Our dataset and model are
available at: https://github.com/pengyu965/ChartReformer.

###### Keywords:

Chart Editing, Chart Appearance Editing, Chart Data Extraction, Chart
Understanding, Visual Language Model

##  1 Introduction

Charts are designed with specific aesthetics and formats to effectively
visualize tabular data. However, a given visualization may only be ideal for a
specific scenario or purpose. Modifying chart images would allow them to be
adapted for diverse applications by enabling the highlighting of specific data
segments, amplifying the distinctions between data points, converting charts
into different formats, or editing the appearance of the graph style. These
are significant and can enhance accessibility for readers.

In the evolving landscape of data analysis, charts and graphs play an
indispensable role in deciphering complex datasets and facilitating informed
decision-making. The ability to effectively visualize tabular data through
charts is crucial, yet the specificity of a visualization’s design to its
initial context limits adaptability for broader applications. This
necessitates the development of methods to modify chart images, enabling them
to highlight particular data segments, enhance distinctions between data
points, or improve accessibility for diverse readerships. However, traditional
chart-editing methods are fraught with challenges. These processes often
require significant manual intervention, a deep understanding of the plotting
software’s parameters, and access to the original data tables. These
limitations become particularly acute in scenarios where source data are lost
or unavailable, highlighting the need for more flexible and accessible editing
techniques.

Figure 1: Examples of chart editing results from our methods. In total, our
methods define and cover four types of chart editing: style, layout, format
and data-centric edit.

Recent advances in computer vision and natural language processing (NLP) have
opened new avenues for understanding charts. As multi-modal tasks, chart
understanding related research topics such as data extraction, question
answering, and chart summarization are tackled by utilizing visual language
model in [11, 10, 5, 15, 2]. However, for chart editing tasks, [12] still rely
on input visualization code and resource data table, while ChartLlama [5]
fails to cover the full spectrum of possible edits, such as data manipulation
for input charts. To close this gap, we introduce ChartReformer, which edits
chart images from natural language prompts without any prior knowledge of the
underlying data and original plot settings, and also covers a wide range of
edits from appearance, format, layout to data-centric edits. In our methods,
we decompose the input charts and reasoning over the prompts for a new
corresponding data table and visual attributes, allowing for detailed,
comprehensive, and accurate chart editing. This approach predicting and
adjusting the embedded visual attributes and data under original chart images,
enables the creation of customized chart images through a re-plotter without
explicitly predicting the plotting code, producing and delivering robust and
stable chart editing results. Overall, the main contributions of our work can
be summarized as follows:

  * •

Define the image-based chart editing tasks, articulate the challenges, and
provide a detailed taxonomy of edits. Suitable evaluation metrics are designed
for such tasks.

  * •

Generate large datasets for the chart-editing task spanning major edit
categories including style, format, layout and data-centric edits.

  * •

Present ChartReformer, a solution based on visual-language model for the task,
and empirically demonstrate the effectiveness of our system by evaluating on
the dataset.

##  2 Related Work

###  2.1 Natural-language-driven visualization

The intersection of Natural Language Processing (NLP) and data visualization
within the field of human-computer interaction (HCI) has become increasingly
prominent [20]. This surge in interest, especially within the deep learning
community, is driven by advances in natural language understanding [17]. Tools
such as VegaLite [18] and ChartDialogs [19] demonstrate the ability to
generate and adjust visual charts in natural language, the former utilizing
JSON for chart specifications and the latter applying Seq2Seq models for
editing in natural language dialogues. Similarly, VizGpt(vizgpt.ai) employs
GPT models to react with human language instructions for visualization and
styling. These approaches are practical, yet different from ours; they rely on
available underlying tabular data and do not directly alter visualized images.

###  2.2 Chart Comprehension

####  2.2.1 Datasets

Editing charts is a nuanced task that necessitates a grasp of both the chart’s
visual features and the data it represents. Contemporary multi-modal models
such as GPT-4V [23] and LLaVA-1.5 face challenges in analyzing and extracting
the underlying data of the charts [4], while chart manipulation is even more
difficult. To facilitate the model in understanding the charts, several
datasets have been introduced. Some assess understanding via straightforward
question-and-answer formats with human-annotated QA pairs - for example,
ChartQA [14], or utilize templates from crowd-sourced platforms - like PlotQA
[16], or employ synthetic examples created by Large Language Models (LLMs) -
as seen in ChartLlama [5]. Other dataset categories, such as chart-to-text
[7], measure comprehension through summarization. However, to our knowledge,
there is no publicly available datasets for chart editing.

Edit Category |  Aspect |  Attributes
---|---|---
Style |  Colors |  Plot Color
|  Line |  Line Style, Marker
|  Pattern |  Bar Pattern
|  Text |  Font, size
Layout |  Axes |  Grid Lines
|  Legend |  Position, Internal Layout
Format |  Plot |  $Line\leftrightarrow Bar$
Data Centric |  Data Filtering |  Range-based
|  |  Series-based
|  Data Addition |  Add/Update Data-point, Add/Update Data-series
|  |
Table 1: Chart Edit Specification

####  2.2.2 Models

Many existing methods [8, 13, 22, 1] analyze the component of the charts and
extract the underlying data by relating the results of component detection.
This routine relies heavily on the intermediate results and is potentially
vulnerable. Utilizing the large visual language model, direct understanding
and reasoning can be performed on input charts without the need for
intermediate steps. Pix2Struct [9] is a such model on similiar task that
extracts visually-situated language from web page screenshot into structured
text. Then Matcha [11] adapts Pix2Struct for chart reasoning by pre-training
the model on plot deconstruction and numerical reasoning. Deplot [10] fine-
tune the matcha for chart-to-table conversion, while this conversion will loss
the crucial apperance references, vital for chart editing. Both Matcha [11]
and ChartLlama [5] can de-render chart images into tables and plot code.
However, none of these models is explicitly capable of predicting and
adjusting the charts’ visual attributes and corresponding underlying data,
which is vital to editing charts accurately. Meanwhile, predicting code is a
challenging way to deliver the final edit results, since they have a higher
chance of failure in code compilation.

##  3 Standardize Chart Editing Task

###  3.1 Chart Edit Taxonomy

Surveying the typical edits performed on chart images, we can categorize them
into four distinct classes: appearance, layout, format, and data-centric, as
detailed in Table 1. Furthermore, most advance chart edits can be obtained by
chaining these individual edits. The following subsections elaborate on each
of the different edit categories.

####  3.1.1 Style Edits

: Style in chart images encompasses essential low-level features such as plot
colors, styles (including line style, marker style, and bar pattern), and font
characteristics (type, size, etc.). Modifications to a chart’s style strictly
alter these foundational visual attributes of individual elements without
tampering with the data or visualization format. Such appearance-based tweaks
are crucial for various applications. Considering accessibility: A color-
differentiated chart may not be discernible to color-blind individuals, and
transitioning to a different color palette can significantly enhance the chart
comprehension for them.

####  3.1.2 Layout Re-composition

: Our layout considerations encompass two primary aspects: axes grids and
legends. Tweaks layouts in chart editing can ensure that each element is
represented systematically in the modified chart. Furthermore, specific layout
changes can improve data readability, such as the inclusion or exclusion of
grid lines.

####  3.1.3 Format Conversion

: Different chart types highlight specific aspects of the data. Line charts
are particularly effective at showcasing trends over time, allowing viewers to
quickly discern patterns and changes. Bar charts, on the other hand, excel in
comparing quantities among different groups or categories. More specifically,
grouped bar charts show mostly the absolute value comparison, while stacked
bar charts reflect the ratio. Transitioning between these formats can offer
more comprehensive data views from different aspects.

####  3.1.4 Data-Centric Modifications

: Data-centric chart modifications enable precise manipulation of the chart’s
underlying data, facilitating tailored adjustments to the visualization’s
displayed information. These edits, which range from subtle alterations to
significant additions, can dramatically shift the narrative and insights
gleaned from a chart. Critical operations include range filtering to focus on
specific data segments, series filtering to highlight particular data series,
and adding new data series or data points to enrich the visualization. Such
direct interactions with the chart data empower users to create custom views
and explore data in novel and informative ways.

##  4 ChartReformer

Figure 2: A Chart Image and edit-prompt are taken as input by the
ChartReformer model, which predicts visual attributes and data for the
corresponding edited chart. A Replotter software takes in these predicted
parameters and generates the edited chart-image

Chart images are inherently complex. Hence, accurate chart comprehension is a
prerequisite to successful chart editing. The structural decomposition of the
chart image has been seen as a potential approach to the problem. Matcha [11]
and ChartLlama [5] try to predict the Python visualization code corresponding
to the chart. However, predicting accurate plotting code for the wide variety
of charts proves to be a challenging task and easy to fail in success code
compilation. We propose ChartReformer, a method that de-renders charts into
underlying data and visual attributes to address these limitations for chart
editing. Fig. 2 demonstrates how ChartReformer predicts a decomposed chart
image that a replotter software can effectively utilize to construct the
edited charts. The following sections further detail our approach on datasets
and the pipeline.

###  4.1 Dataset

This section provides insight into the data creation process for Chart Edits.
We synthesize paired chart images using data tables from existing chart
datasets to obtain a sizeable chart editing dataset. Our primary emphasis is
on line and bar charts, including both grouped and stacked vertical bar
variations, given their prevalent use in real-world charting scenarios.

####  4.1.1 Data Source

: We utilized data tables from AdobeSynth-19 data previously released as part
of the ICPR ChartInfographics competition [3]. This dataset originally
consists of synthetically generated images using Matplotlib [6], however, the
underlying data is derived from real-world sources such as World Development
Indicators and Gender Statistics (World Bank), among others.

####  4.1.2 Chart Image Generation

: We developed a custom software using Matplotlib to synthesize chart images
with varying visual attributes given as input. The tool supports all the edits
specified in Table 1 while allowing sampling from a comprehensive pool of
visual attributes. A thorough parameter pool is provided in the Appendix Table
LABEL:Table:_Matplotlib_property_pool.

Parameters are randomly selected from this pool, a strategy that introduces
significant diversity in the visual appearance of the generated charts. Since
plotting parameters are explicitly defined, storing them in a modifiable JSON
format becomes straightforward, facilitating further adjustments and reuse.
Appendix Fig. 8 shows an example for such a JSON specifying all parameters.

####  4.1.3 Edit Pair Synthesis

: The data generation pipeline produces, for every sample, edited chart pairs
and corresponding edit text prompts. Each chart in the pair consists of visual
attributes, a data table, and the synthesized image. The edit categories are
described as follows:

  * •

Style and Layout Edits: For each style or layout modification outlined in the
edit specifications, we modify the relevant plotting parameter in the
parameter JSON file.

  * •

Format Edits: This edit category facilitates the conversion between line and
bar charts, and vice versa. We employ the identical data table, generating a
new chart type and substituting the original plotting parameters with those
corresponding to the new chart, the color of each data series before and after
conversion remains the same, while we allow the style for new plot to be
different, e.g., from line chart to bar chart, the line color and bar color
are corresponding to each other while bar pattern can be randomly chosen.

  * •

Data Edits Here, we alter the data table according to the given prompt and
tailor the original plotting parameters accordingly. This ensures that while
the visual attributes of the chart remain consistent, the data in the charts
are updated.

For each type of edit, one base prompt is created along with five variations
to capture the diversity of natural language expressions. Please refer to
Appendix Figure 9 for examples of such prompts along with the corresponding
parameter and data table updates.

####  4.1.4 Dataset Statistics

: Table 2 outlines the statistics of our datasets, categorizing them into four
principal types of edits. Overall, our dataset encompasses approximately
70,000 paired samples.

Table 2: Dataset Statistics: Number of paired samples by Edit Types across Chart Categories Edit Type |  Line |  Stacked Vertical Bar |  Grouped Vertical Bar
---|---|---|---
Style |  17,663 |  8,822 |  6,650
Layout |  2,331 |  1,178 |  1,132
Data-centric |  4,082 |  4,195 |  4,999
Format |  9,996 |  6,125 |  7,764
Total |  34,072 |  20,320 |  20,545

The distribution of detailed edits for style, layout, format, and data
manipulation is shown in Figure 3. The generation of chart-editing dataset is
based on the manipulation of visual attribute parameters. There are overall
two types of visual attribute parameters, chart-type-relevant parameters (line
styles/markers, bar patterns, etc.), and chart-type-irrelevant parameters
(font size, axis labels orientation, etc.). The edits based on the first type
of parameters are oversampled than the second type, since it helps model to
learn more challenging part – identifying and manipulating the plotting graph.
Furthermore, the distributions for data manipulation and layout editing are
presented in Appendix Figures 6 and 7, respectively.

Figure 3: Distribution of Samples for Style Edits across Chart Categories

###  4.2 Implementation

To address the challenge of chart editing, we adapt the visual-language
encoder-decoder-based transformer [9] on our chart editing tasks. We break
down the training into two stages: pretraining for accurate Chart de-rendering
and fine-tuning for chart editing.

####  4.2.1 Chart De-rendering

Accurately de-rendering of chart to visual attributes and underlying data is a
prerequisite. We pre-train ChartReformer on our dataset with unpaired images
to enable accurate visual attributes and underlying data extraction.
Simultaneously predicting the underlying data and visual attributes allows the
model to learn the mapping between them. We initialize the model with the
checkpoint from [10] and train it on our dataset with 100K samples. To avoid
text distortion and blur during the input resizing of chart images, we opt for
a larger input image size of $(800,800)$ with padding to maintain the original
aspect ratio. The maximum output sequence length is 1024, allowing sufficient
prediction of all parameters and data tables.

####  4.2.2 Chart Editing

We fine-tune the pre-trained model on paired images and edit prompts, totally
55k samples, enabling it to interpret prompts and adjust data and visual
properties accordingly. Edited charts are then replotted with updated data and
plotting parameters. To enhance real-world plotting success, we suggest using
JSON repair and default plotting parameters for incomplete predictions, though
our evaluations eschew repairs for unbiased performance assessment.

##  5 Experiments

We use ChartLlama as a baseline for comparison. To the best of our knowledge,
no existing dataset related to chart editing is publicly available111Based on
current information, the dataset developed by ChartLlama [5] has not yet been
formally published. Hence, we perform an evaluation exclusively on our
dataset.

###  5.1 Metrics

####  5.1.1 Image-based Evaluation

To facilitate a model-agnostic comparison, we utilize Structural Similarity
Index Measure (SSIM) [21] to assess the quality of the generated image
relative to the edited image from the ground truth. SSIM offers a nuanced
perspective on the degree to which the edited chart mirrors the expected
outcome, capturing subtle and critical edits speaking to structural similarity
aspective. We also calculate a success rate, which reflects the proportion of
edits where the edited image was successfully generated, accounting for
instances of plotting failures due to inaccurate or incomplete structure
prediction. For comparison method ChartLlama, the success rate measures the
ratio of samples that the predicted code can be successfully compiled.

Table 3: ChartReformer performance across different types of edits. The first
two rows represent Visual Attributes and Data-Table scores for edits, whereas
the last two represent image-level comparison metrics

| Style | Layout | Format | Data-Centric | Total
---|---|---|---|---|---
Metrics | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
VAE | 93.08 | 93.01 | 93.04 | 97.20 | 97.15 | 97.17 | 83.61 | 83.58 | 83.59 | 89.75 | 89.69 | 89.72 | 85.14 | 85.09 | 85.11
RMS | 81.80 | 81.50 | 81.58 | 79.90 | 79.44 | 79.60 | 85.76 | 85.34 | 85.51 | 79.34 | 78.50 | 78.76 | 82.22 | 81.77 | 81.92
SSIM | 87.39 | 86.06 | 74.19 | 83.07 | 83.05
Success Rate | 99.81 | 100 | 96.9 | 99.77 | 98.99
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

####  5.1.2 Evaluating Edit Correctness

To offer a more precise evaluation of the performance of our methods on the
chart editing task, we use the predicted visual attributes and underlying data
table for evaluation. Our replotting process is heurstic, therefore, comparing
those with ground truth reflect the performance well. We use the Relative
Mapping Similarity (RMS) from [10], and Visual Attribute Edit (VAE) score to
evaluate the accuracy of underlyting data table and visual attributes
prediction, respectively.

$\displaystyle S_{changed}$ $\displaystyle=\frac{1}{|X_{c}|}\sum_{e\in
X_{c}}S(e,g)$ (1) $\displaystyle S_{unchanged}$
$\displaystyle=\frac{1}{|X_{u}|}\sum_{e\in X_{u}}S(e,g)$

$S_{f}=\frac{2\cdot S_{changed}\cdot
S_{unchanged}}{S_{changed}+S_{unchanged}}$ (2)

VAE is calculated by grouping attributes into two groups: attributes should be
edited $S_{changed}$, and attributes should remain unchanged $S_{unchanged}$,
as shown in Equation 1. $e$ and $g$ represent the edited attribute value and
the corresponding ground truth value. First, we calculate a similarity matrix
to match the key/value between the ground truth and the prediction. Then the
score for each key is calculated based on the value type: the categorical
value are based on an exact match; and numeric value are scores from 0 to 1
based on threshold of 0.4. Finally, the visual attribute edit score (VAE)
$S_{f}$ is calculated using the harmonic mean among them as Equation 2. This
prevents biased evaluation, as only a minor part of the visual attributes will
change. The precision, recall and F1 score are obtained based on the matching
result.

Table 4: Comparison with ChartLlama across different edits on a subset of the test-set (1K samples) Editing Category | Metric | ChartLlama [5] | Our Methods
---|---|---|---
Style | SSIM | 57.62 | 87.58
Success Rate | 12.03 | 99.6
Layout | SSIM | 65.19 | 85.97
Success Rate | 26.47 | 100
Format | SSIM | 45.32 | 74.44
Success Rate | 5.04 | 100
Data-Centric | SSIM | 70.33 | 82.91
Success Rate | 28.57 | 99.6
Total | SSIM | 59.61 | 82.72
Success Rate | 18.20 | 99.6
|  |  |

###  5.2 Results

Table 3 shows the results of ChartReformer on our test set consisting of 6.8K
samples across different edit types. VAE and RMS measure edit-correctness
corresponding to visual attributes and data, respectively, whereas SSIM
measures image-level similarity with the ground-truth edited image. As can be
seen, performance on visual attributes is higher than data extraction,
indicating the difficulty of predicting each aspect respectively. Further,
data-centric edits also appear to be the hardest as they require a precise
understanding of the existing data and manipulation based on the prompt.
Overall, the model performs reasonably well on chart edits while maintaining
high fidelity with the original chart image, as also seen in Fig. 5.

Compared to recent work ChartLlama [5], since ChartLlama follows different
methodolgy then ours, we could only compare with ChartLlama with SSIM and
success rate. From Table 4, ChartReformer performs better across all edit
categories. This performance gap could be attributed to the lack of training
on a comprehensive edit dataset. Further, the success rate of our method is
better as we do not predict the visualization python code, which is harder to
get correct. We concede that in the current setup, we did not perform prompt
engineering for ChartLlama, which will likely improve the performance of
ChartLlama. par

Figure 4: Qualitative Results for ChartLlama Figure 5: Qualitative Results for
ChartReformer

##  6 Discussion and Limitations

As shown in the Table 3, ChartReformer can successfully extract the detailed
visual attributes for edited charts across all edit types. However, the
overall edit performance is limited by the data-extraction score. Therefore, a
more accurate data-extraction approach would result in more precise data edits
and be a promising avenue for further research. The proposed chart-edit
dataset covers a wide range of edits, yet real-world edit instructions could
be abstract and arbitrarily complex. For example, “Optimize the chart
appearance for projecting in a large conference room” or “Modify the chart’s
color palette making it accessible to the color-blind”. One way to handle such
queries is to use a preprocessing module (for instance, a decoder-only
language model), which could be used to better interpret and simplify them
into a series of simply chained edit prompts ChartReformer can handle.

##  7 Conclusion

In this work, we present and standardize the chart-editing task. ChartReformer
presents a novel approach to chart editing, allowing for modifications
directly from chart images without the need for underlying data tables or
supplementary information. By generating edited charts in a decomposed form
that includes both the data table and visual attributes, ChartReformer enables
precise, natural language-driven edits across style, layout, format, and data-
centric modifications. Our experiments demonstrate promising results,
highlighting ChartReformer’s potential to enhance chart accessibility and
adaptability for diverse applications.

## References

  * [1] Ahmed, S., Yan, P., Doermann, D., Setlur, S., Govindaraju, V.: Spaden: Sparse and dense keypoint estimation for real-world chart understanding. In: International Conference on Document Analysis and Recognition. pp. 77–93. Springer (2023)
  * [2] Cheng, Z.Q., Dai, Q., Li, S., Sun, J., Mitamura, T., Hauptmann, A.G.: Chartreader: A unified framework for chart derendering and comprehension without heuristic rules. arXiv preprint arXiv:2304.02173 (2023)
  * [3] Davila, K., Tensmeyer, C., Shekhar, S., Singh, H., Setlur, S., Govindaraju, V.: Icpr 2020 - competition on harvesting raw tables from infographics. In: Del Bimbo, A., Cucchiara, R., Sclaroff, S., Farinella, G.M., Mei, T., Bertini, M., Escalante, H.J., Vezzani, R. (eds.) Pattern Recognition. ICPR International Workshops and Challenges. pp. 361–380. Springer International Publishing, Cham (2021)
  * [4] Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., Zhou, T.: Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models (2023)
  * [5] Han, Y., Zhang, C., Chen, X., Yang, X., Wang, Z., Yu, G., Fu, B., Zhang, H.: Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483 (2023)
  * [6] Hunter, J.D.: Matplotlib: A 2d graphics environment. Computing in Science & Engineering 9(3), 90–95 (2007). https://doi.org/10.1109/MCSE.2007.55
  * [7] Kantharaj, S., Leong, R.T., Lin, X., Masry, A., Thakkar, M., Hoque, E., Joty, S.: Chart-to-text: A large-scale benchmark for chart summarization. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 4005–4023. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.acl-long.277, https://aclanthology.org/2022.acl-long.277
  * [8] Lal, J., Mitkari, A., Bhosale, M., Doermann, D.: Lineformer: Line chart data extraction using instance segmentation. In: International Conference on Document Analysis and Recognition. pp. 387–400. Springer (2023)
  * [9] Lee, K., Joshi, M., Turc, I., Hu, H., Liu, F., Eisenschlos, J., Khandelwal, U., Shaw, P., Chang, M.W., Toutanova, K.: Pix2struct: screenshot parsing as pretraining for visual language understanding. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23, JMLR.org (2023)
  * [10] Liu, F., Eisenschlos, J.M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., Altun, Y.: Deplot: One-shot visual language reasoning by plot-to-table translation. arXiv preprint arXiv:2212.10505 (2022)
  * [11] Liu, F., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Altun, Y., Collier, N., Eisenschlos, J.M.: Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. arXiv preprint arXiv:2212.09662 (2022)
  * [12] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023)
  * [13] Luo, J., Li, Z., Wang, J., Lin, C.Y.: Chartocr: Data extraction from charts images via a deep hybrid framework. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1917–1925 (2021)
  * [14] Masry, A., Do, X.L., Tan, J.Q., Joty, S., Hoque, E.: ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Findings of the Association for Computational Linguistics: ACL 2022. pp. 2263–2279. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.findings-acl.177, https://aclanthology.org/2022.findings-acl.177
  * [15] Masry, A., Kavehzadeh, P., Do, X.L., Hoque, E., Joty, S.: Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761 (2023)
  * [16] Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: Plotqa: Reasoning over scientific plots. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1516–1525 (2020). https://doi.org/10.1109/WACV45572.2020.9093523
  * [17] Narechania, A., Srinivasan, A., Stasko, J.: NL4DV: A Toolkit for generating Analytic Specifications for Data Visualization from Natural Language queries. IEEE Transactions on Visualization and Computer Graphics (TVCG) (2020). https://doi.org/10.1109/TVCG.2020.3030378
  * [18] Satyanarayan, A., Moritz, D., Wongsuphasawat, K., Heer, J.: Vega-lite: A grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics 23(1), 341–350 (jan 2017). https://doi.org/10.1109/TVCG.2016.2599030, https://doi.org/10.1109/TVCG.2016.2599030
  * [19] Shao, Y., Nakashole, N.: ChartDialogs: Plotting from Natural Language Instructions. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 3559–3574. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.328, https://aclanthology.org/2020.acl-main.328
  * [20] Srinivasan, A., Stasko, J.: Orko: Facilitating multimodal interaction for visual exploration and analysis of networks. IEEE Transactions on Visualization and Computer Graphics 24(1), 511–521 (2018). https://doi.org/10.1109/TVCG.2017.2745219
  * [21] Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861
  * [22] Yan, P., Ahmed, S., Doermann, D.: Context-aware chart element detection. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds.) Document Analysis and Recognition - ICDAR 2023. pp. 218–233. Springer Nature Switzerland, Cham (2023)
  * [23] Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.C., Liu, Z., Wang, L.: The dawn of lmms: Preliminary explorations with gpt-4v(ision) (2023)

##  8 Appendix

###  8.1 Dataset

####  8.1.1 Matplotlib visual attributes pool

: Table LABEL:Table:_Matplotlib_property_pooldescribes the Matplotlib
parameters that we randomly sample from to generate our dataset have more
diverse charts instead of relying on Matplotlib’s default selection of these
properties.

Table 5: Matplotlib Property Pool |  |  |
---|---|---|---
Scope |  Property |  Pool |  Editable
Line |  Color |  [“b”, “g”, “r”, “c”, “m”, “y”, “k”] |  Yes
Line |  Marker |  [“o”, “`^`”, “s”, “*”, “None”] |  Yes
Line |  Line-style |  [“solid”, “dashed”, “dotted”, “dense dotted”, “loose dotted”, “dense dashed”, “loose dashed”] |  Yes
Bar |  Hatch |  [“xx”, “.”, “*”, “/”, “\”, “None”] |  Yes
Bar |  Color |  [“b”, “g”, “r”, “c”, “m”, “y”, “k”] |  Yes
Global |  X-axis Label Font Name |  [“monospace”, “Serif”, “sans-serif”, “Arial Black”] |  Yes
Global |  X-axis Label Font Size |  [“medium”, “large”, “x-large”] |  Yes
Global |  Y-axis Label Font Name |  [“monospace”, “Serif”, “sans-serif”, “Arial Black”] |  Yes
Global |  Y-axis Label Font Size |  [“medium”, “large”, “x-large”] |  Yes
Global |  Legend Location |  [0, 1, 2, 3, 4, 8, 9] |  Yes
Global |  Legend Columns |  [1, 2, 3] |  No
Global |  Title Font Name |  [“monospace”, “Serif”, “sans-serif”, “Arial Black”] |  Yes
Global |  Title Font Size |  [“medium”, “large”, “x-large”] |  Yes
Global |  X-tick Label Size |  [“x-small”, “small”, “medium”, “large”] |  Yes
Global |  X-tick Rotation |  [0, 45] |  No
Global |  Y-tick Label Size |  [“x-small”, “small”, “medium”, “large”] |  Yes
Global |  Grid Visibility |  [True, False] |  Yes
Global |  Grid Axis |  [‘both’, ‘x’, ‘y’] |  No
Global |  Grid Line-style |  [‘solid’, ‘dashed’] |  No
|  |  |

####  8.1.2 Distribution of Data-Centric and Layout edits

The below figure shows the distribution edits of layout and data-centric edit
types.

Figure 6: Distribution of Number of Samples For Data-centric edits Across
Chart Categories

Figure 7: Distribution of Number of Samples For Layout Edits Across Chart
Categories

####  8.1.3 Example of JSON Properties

Figure 8 shows one example of our visual attribute and underlying data JSON
format of a line chart. The data table consists of three values:
‘underlying_data’, ’chart_title’, ’x_axis_title’, ’y_axis_title’. We put the
data table in front of the visual attributes since the model’s text generation
is more stable at front than back, while the data is more sensitive and
valuable than visual attributes speaking to real application.

Figure 8: An example showcasing the JSON configuration for a line chart
alongside the generated chart itself. It is important to note that the JSON
includes the plotting parameters and the underlying data table. This inclusion
ensures a clear and discriminative association between the different data
series and their respective visual attributes.

    {
        ‘‘underlying_data’’: ‘‘Country Name | 2004 | 1997 <0x0A>
        OECD members | 23.66 | 21.08 <0x0A>
        Middle East & North Africa | 35.21 | 29.0
        <0x0A> South Asia | 20.33 | 14.15 <0x0A> ’’,

        ‘‘chart_title’’: ‘‘Imp’’,
        ‘‘x_axis_title’’: ‘‘Country Name’’,
        ‘‘y_axis_title’’: ‘‘Values’’,
        ‘‘global_properties’’: {
            ‘‘chart_type’’: ‘‘line’’,
            ‘‘x_label_params’’: {
                ‘‘fontname’’: ‘‘Serif’’,
                ‘‘fontsize’’: ‘‘x-large’’
            },
            ‘‘y_label_params’’: {
                ‘‘fontname’’: ‘‘Serif’’,
                ‘‘fontsize’’: ‘‘x-large’’
            },
            ‘‘legend_params’’: {
                ‘‘loc’’: 1,
                ‘‘ncol’’: 1
            },
            ‘‘chart_title_params’’: {
                ‘‘fontname’’: ‘‘Serif’’,
                ‘‘fontsize’’: ‘‘medium’’,
                ‘‘rotation’’: 0
            },
            ‘‘x_tick_params’’: {
                ‘‘axis’’: ‘‘x’’,
                ‘‘which’’: ‘‘major’’,
                ‘‘rotation’’: 0,
                ‘‘labelsize’’: ‘‘medium’’,
                ‘‘labelfontfamily’’: ‘‘sans-serif’’
            },
            ‘‘y_tick_params’’: {
                ‘‘axis’’: ‘‘y’’,
                ‘‘which’’: ‘‘major’’,
                ‘‘rotation’’: 0,
                ‘‘labelsize’’: ‘‘medium’’,
                ‘‘labelfontfamily’’: ‘‘sans-serif’’
            },
            ‘‘grid_params’’: {
                ‘‘visible’’: true,
                ‘‘axis’’: ‘‘y’’,
                ‘‘linestyle’’: ‘‘dashed’’
            }
        },
        ‘‘line_properties’’: {
            ‘‘linestyles’’: [
                ‘‘dashdot’’,
                ‘‘solid’’
            ],
            ‘‘markers’’: [
                ‘‘*’’,
                ‘‘s’’
            ],
            ‘‘colors’’: [
                ‘‘k’’,
                ‘‘r’’
            ]
        }
    }

####  8.1.4 JSON Properties Edit Example

Figure 9 shows the parts of the JSON properties, including plotting parameters
and the underlying table before and after the edit according to the prompt
given. Please note that our model can maintain high fidelity w.r.t parts that
should not be edited.

Figure 9: Edit sample with prompt