# Datasets of Visualization for Machine Learning

Can Liu, Ruike Jiang, Shaocong Tan, Jiacheng Yu, Chaofan Yang, Hanning Shao,
Xiaoru Yuan  Can Liu, Ruike Jiang, Shaocong Tan, Jiacheng Yu, Chaofan Yang,
Hanning Shao, and Xiaoru Yuan are with Key Laboratory of Machine Perception
(Ministry of Education), School of Intelligence Science and Technology, Peking
University. E-mail: {can.liu, jiangrk, jiachengyu, chaofanyang, hanning.shao,
<EMAIL_ADDRESS>Xiaoru Yuan is also with National Engineering
Laboratory for Big Data Analysis and Application, Peking University. Xiaoru
Yuan is the corresponding author.Manuscript received November xx, 2023;
revised November xx, 2023.

###### Abstract

Datasets of visualization play a crucial role in automating data-driven
visualization pipelines, serving as the foundation for supervised model
training and algorithm benchmarking. In this paper, we survey the literature
on visualization datasets and provide a comprehensive overview of existing
visualization datasets, including their data types, formats, supported tasks,
and openness. We propose a what-why-how model for visualization datasets,
considering the content of the dataset (what), the supported tasks (why), and
the dataset construction process (how). This model provides a clear
understanding of the diversity and complexity of visualization datasets.
Additionally, we highlight the challenges faced by existing visualization
datasets, including the lack of standardization in data types and formats and
the limited availability of large-scale datasets. To address these challenges,
we suggest future research directions.

###### Index Terms:

Visualization datasets, machine learning

##  1 Introduction

In recent years, there has been a significant trend towards data-driven and
automated processes in visualization [1]. Within these automated processes,
machine learning models are employed, trained on related datasets specifically
to perform various tasks related to visualization [2]. Consequently, datasets
tailored for visualization purposes have become a crucial infrastructure
component in these automated processes. Their importance and utility are
progressively being recognized and emphasized. Compared to datasets in other
domains, such as ImageNet [3] and VQA [4], datasets of visualizations have
higher diversity in terms of data types, data formats, and tasks. The data
types include visualizations, visual components, underlying data, and
additional information. The underlying data in a visualization can take
various forms such as tables, networks, or text, and the results of the
visualization can be displayed in a diverse range of formats. Within the
visualization itself, there are numerous visual elements, including but not
limited to axes, legends, and titles. Beyond the core visual components,
additional information often accompanies the visualization, like a set of
questions and answers related to it, enhancing its informational value. A
broad task space for visualization relies on datasets of various data formats,
for example, constructing visualizations from data, reverse-engineering data
from visualizations, and answering visualization-related questions. To support
these tasks, datasets of visualizations typically consist of multiple data
types and serve as the foundation for supervised model training and
benchmarking. We survey the literature on datasets of visualizations, provide
an overview of existing datasets of visualizations, summarize their data
format, scale, supported tasks, and openness, and aim to emphasize the
challenges and future research directions in datasets of visualizations.

In this survey, we define a dataset of visualization as a collection of data
used for training or testing machine learning models in the visualization
process. Datasets of visualizations can be classified into a multi-layer
scheme, including the content of it, the tasks supported, and how they are
constructed. An example of a dataset of visualization is VizNet [5], which
includes a wide range of visualization design examples, from simple bar charts
to more complex visualizations. Its goal is to provide a comprehensive
overview of the design space. Another example is VizCommender [6], which is a
dataset focused on retrieval. It contains a large collection of visualization
designs, as well as user preferences and search queries, making it suitable
for developing visualization recommendation systems. In terms of data format,
existing datasets of visualizations range from raster images to vector
graphics and even text-based formats. For example, the VizML dataset [7] is a
data-driven visualization construction dataset that focuses on converting data
into visual representations. It contains both data tables and the
corresponding visualization results, allowing the development of models that
can automatically generate visualizations from data.

Figure 1:  The what-why-how model for the dataset of visualizations. What
describes the content of the visualization, why describes the usage of the
datasets, and how describes the construction methods of these datasets.

Despite the consensus among researchers that constructing high-quality
datasets of visualizations is crucial, the existing datasets still face
numerous challenges. One of the major challenges is the lack of
standardization in terms of data types and formats. This lack of
standardization makes it difficult to compare and integrate datasets from
different sources, hindering the development of more advanced models and
algorithms, especially in such an era when large datasets are needed. Another
challenge is the limited availability of large-scale datasets. The current
trend is towards automating data-driven visualization pipelines, which
requires large-scale datasets for supervised model training. To address these
challenges, future research should focus on standardizing data types and
formats, as well as increasing the openness of existing datasets.
Additionally, the intelligent construction of datasets of visualizations is
also an important direction, aimed at building high-quality large-scale
datasets with less manual labor.

We summarized the datasets of visualization presented in these papers using a
what-why-how model, as illustrated in Figure 1.

  * •

In terms of what is contained in the visualization dataset, we have decomposed
the data type based on the formats and relationships present in the dataset.
The visualization dataset may encompass underlying data, visualization
components, presentation of the visualization, and accompanying information.

  * •

As to why the visualization dataset is needed, it is established to support
visualization tasks. We have categorized these tasks into three categories,
namely basic techniques, general tasks, and applications.

  * •

In regards to how the dataset is constructed, we have outlined several
approaches to building a visualization dataset. The construction process has
been summarized as comprising construction, synthesis, processing, annotation,
and labeling.

“Why” represents the machine learning tasks driven by the need for automation
within the visualization process. In such tasks, it is common to employ a
supervised dataset featuring correspondingly paired relationships to
effectively train a model. “What” pertains to the specific nature of these
tasks; for instance, a task like visualization recommendation necessitates a
pair consisting of underlying data and its corresponding visualization. “How”
dimension elucidates the methods employed in the collection of these datasets.
The collection of datasets for visualization is shown in Table I, categorized
by what-why-how model.

TABLE I: Summary of datasets of visualization for machine learning.

| Data Types | Vis Types | Data Construction | Techniques | Tasks | User Tasks |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
Dataset |  VIS |  NL |  VIS Type |  Data |  Mapping |  VIS Element |  Element Position |  Others |  Bar |  Scatter |  Line |  Area |  Pie |  Others |  Crawling |  Extraction |  Crowdsourcing |  Synthesizing |  Combining |  Manual |  Rule-based |  Learning-based |  Object Detection |  Classification |  Regression |  Translation |  OCR |  DR |  Recommendation |  Reverse Engineering |  Feature Extraction |  Generation |  Retrieval  |  Exploration  |  Assessment |  Use for Learning |  Visualization size |  Data size |  Still Growing
SNAP [8] |  |  |  | ● |  |  |  |  |  |  |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  150 |  |
ReVision [9] | ● |  | ● |  |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  |  | ● |  |  |  |  |  | ● |  |  |  |  | ● |  |  | ● |  | ● |  2,500 |  |
MASSVIS [10] | ● |  | ● |  |  |  |  | ● | ● | ● | ● | ● | ● | ● | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 5,693 |  |
DiagramFlyer [11] | ● |  |  |  |  |  |  | ● | ● | ● | ● |  |  |  |  | ● |  |  |  |  |  | ● | ● | ● |  |  | ● |  |  | ● |  |  | ● |  |  | ● |  300,000 |  |
Aesthetics [12] | ● |  |  |  |  |  |  |  |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  330 |  |
WikiTable [13] |  | ● |  | ● |  |  |  |  |  |  |  |  |  |  | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 2,108 D, 22,033 QA |
Network Repository [14] | ● |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  |  |  | ● |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  5,000 |  | ●
Figureseer [15] | ● | ● |  | ● |  | ● |  |  | ● | ● | ● |  |  | ● |  | ● |  |  |  | ● |  |  | ● | ● |  |  | ● |  |  | ● |  |  | ● |  |  | ● |  60,000 |  |
REV [16] | ● |  | ● |  | ● |  |  | ● | ● | ● | ● | ● |  |  | ● |  |  | ● |  |  |  | ● | ● | ● |  |  | ● |  |  | ● |  |  |  | ● |  | ● | 5,125 |  |
ChartSense [17] | ● |  | ● |  |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  |  | ● | ● |  |  | ● |  |  |  |  |  |  | ● |  | ● |  |  | ● | 6,997 |  |
DataClips [18] | ● |  | ● | ● |  |  |  | ● | ● | ● |  | ● | ● | ● | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  70 |  |
BubbleView [19] |  |  |  |  |  |  |  | ● | ● |  | ● |  |  | ● |  |  |  |  | ● | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 393 |  |
FigureQA [20] | ● | ● |  |  |  |  |  | ● | ● | ● | ● |  |  |  |  |  |  | ● |  |  | ● |  |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● | 100,000 | 1,000,000 QA |
Sightline [21] | ● |  |  |  |  |  |  |  | ● | ● | ● | ● | ● | ● |  |  | ● |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 1,300 |  |
Beagle [22] | ● |  | ● |  |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  |  |  |  | ● |  |  |  | ● |  |  |  |  | ● |  |  | ● |  | ● |  41,000 |  |
ScatterNet [23] | ● |  |  |  |  | ● |  |  |  | ● |  |  |  |  |  |  |  | ● |  | ● |  |  |  | ● |  |  |  |  |  |  | ● |  | ● |  |  | ● | 50,677 |  |
DeepEye [24] | ● |  |  |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  | ● |  | ● |  |  |  | ● |  |  |  |  | ● |  |  | ● |  |  |  | ● |  33,400 |  |
DVQA [25] | ● | ● |  |  |  |  |  | ● | ● |  |  |  |  |  |  |  |  | ● |  |  | ● |  |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● | 300,000 | 3,400,000 QA |
VizByWiki [26] | ● | ● |  |  |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  |  |  | ● | ● |  | ● |  |  |  |  |  |  | ● |  | ● |  |  | ● |  3,000,000 |  | ●
GMVR [27] |  |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  | ● | 200000 |  |
Perception [28] | ● |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  |  |  | ● |  |  |  | ● | ● | 100,000 |  |
Text-to-Viz [29] | ● | ● |  |  |  |  |  |  |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  |  | ● |  |  |  |  |  |  |  | ● |  |  |  | ● | 200 |  |
D3 search [30] | ● |  | ● | ● |  |  |  | ● | ● | ● | ● | ● | ● | ● | ● |  |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 7,860 |  |
VizNet [5] | ● |  |  | ● |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  | ● |  | ● |  |  |  | ● |  |  |  |  |  | ● |  |  |  | ● | ● |  |  31,000,000 |
Timeline [31] | ● |  |  |  |  |  |  |  |  |  |  |  |  | ● | ● |  |  | ● |  | ● |  |  | ● | ● |  |  | ● |  | ● |  |  | ● |  |  |  | ● | 4,689 |  |
VizML [7] | ● |  |  | ● | ● |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  |  |  | ● |  |  | ● |  |  |  |  | ● |  |  | ● |  |  |  | ● |  1,000,000 |  |
Data2Vis [32] | ● |  |  | ● | ● |  |  |  | ● | ● | ● | ● | ● | ● |  |  |  | ● |  |  | ● |  |  |  |  | ● |  |  | ● |  |  | ● |  |  |  | ● |  4,300 |  |
Sherlock [33] |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  | ● |  | ● | ● |  |  |  |  |  |  |  |  |  |  |  |  |  | ● |  | 686, 675 columns |
InSituNet [34] |  |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  | ● | 125,000 |  |
DNN-VolVis [35] |  |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  | ● | 10000 |  |
AutoCaption [36] | ● | ● |  | ● |  |  |  |  | ● | ● | ● |  |  |  |  |  |  | ● |  |  | ● |  |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● | 3,000 |  |
Auto Annotation [37] | ● |  |  |  |  | ● |  |  | ● | ● |  |  | ● |  | ● |  |  |  |  | ● |  |  | ● | ● |  |  | ● |  |  | ● | ● | ● |  |  |  | ● |  400 |  |
MultiViewLayout [38] | ● |  |  |  |  |  |  | ● |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 360 |  |
VisImages [39] | ● |  |  |  |  |  |  |  | ● | ● | ● | ● | ● | ● |  | ● |  |  |  | ● |  |  | ● | ● |  |  | ● |  |  | ● |  |  | ● |  |  | ● | 12,267 |  |
InfoVIF [40] | ● |  |  | ● |  | ● | ● | ● |  |  |  |  |  | ● | ● |  |  |  |  | ● |  | ● | ● | ● |  |  | ● |  |  | ● |  |  |  | ● |  | ● |  13,000 |  |
Retrie-then-adapt [41] | ● |  |  |  |  | ● | ● |  |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  | ● | ● |  |  | ● |  | ● | ● |  | ● |  |  |  | ● | 1,000 |  |
InteractiveArticles [42] | ● |  |  |  |  |  |  | ● |  |  |  |  |  | ● |  |  | ● |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  60 |  |
Leaf-QA [43] | ● | ● |  |  |  |  |  | ● | ● | ● | ● |  | ● | ● |  |  |  | ● |  |  | ● | ● |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● | 240,000 | 2,000,000 QA |
Leaf-QA++ [44] | ● | ● |  |  |  |  |  | ● | ● | ● | ● |  | ● | ● |  |  |  | ● |  | ● | ● |  |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● | 244,000 | 2,500,000 QA |
PlotQA [45] | ● | ● |  |  |  |  |  | ● | ● | ● | ● |  |  |  |  |  |  | ● |  | ● | ● |  |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● | 224,000 | 28,000,000 QA |
Vis-QA [46] | ● | ● |  |  |  |  |  | ● | ● |  | ● |  |  |  | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 52 | 629 QA |
FigCAP [47] | ● | ● |  |  |  |  |  |  | ● |  | ● |  | ● |  |  |  |  | ● |  |  | ● |  |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● | 124,217 |  |
VizCommender [6] | ● | ● |  |  | ● |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  |  |  | ● |  |  | ● |  |  |  |  |  |  | ● |  | ● |  |  | ● | 18,820 |  |
ChartSeer [48] | ● |  |  | ● | ● |  |  |  | ● | ● | ● | ● |  |  |  |  |  | ● | ● |  |  |  |  |  |  |  |  | ● |  |  | ● |  |  | ● |  | ● | 9,925 |  |
Calliope [49] | ● |  | ● |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  | ● |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 230 | 4,186 segments |
DVAO [50] |  |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  | ● | 477 |  |
CrisisVis [51] | ● |  | ● |  |  |  |  |  | ● | ● |  |  |  |  |  |  | ● |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 663 |  |
ViralVis [52] |  |  |  |  |  |  |  |  | ● | ● | ● | ● |  | ● | ● |  |  |  |  | ● |  |  |  |  |  | ● |  |  |  |  | ● |  |  | ● |  | ● |  41,000 |  |
VIS30K [53] | ● |  |  |  |  |  |  | ● |  |  |  |  |  | ● |  | ● |  |  |  | ● |  |  | ● | ● |  |  | ● |  |  | ● |  |  | ● |  |  | ● | 29,689 |  |
ExcelChart400K [54] | ● |  | ● | ● |  |  |  |  | ● |  | ● |  | ● |  | ● |  |  |  |  |  |  | ● |  |  |  |  | ● |  |  | ● |  |  | ● |  |  | ● | 386,966 |  |
Visually29K [55] | ● |  |  |  |  | ● | ● |  |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  | ● | ● |  |  | ● |  |  | ● |  |  | ● | ● |  | ● |  29,000 |  |
AutoClips [56] | ● |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  230 |  |
Motion [57] |  |  |  |  |  |  |  |  |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 82 |  |
Kineticharts [58] |  |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 259 |  |
Data-GIF [59] | ● |  |  |  |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 108 |  |
NLV Corpus [60] | ● | ● |  |  |  |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  | ● |  |  |  | ● |  |  |  |  |  |  | ● | ● |  |  |  | ● |  | 893 Sentences |
NL2VIS [61] | ● | ● |  |  |  |  |  | ● |  |  |  |  |  | ● |  |  |  |  | ● | ● | ● |  |  |  |  | ● |  |  | ● |  |  | ● |  |  |  | ● | 25,750 | 25,750 |
SCICAP [62] | ● | ● | ● |  |  |  |  |  |  |  | ● |  |  |  | ● | ● |  |  |  |  |  | ● |  |  |  | ● | ● |  |  |  | ● | ● |  |  |  | ● | 358,972 | 358,972 |
Table2Charts [63] | ● | ● |  | ● | ● |  |  |  |  |  |  |  |  | ● |  |  | ● |  |  |  | ● |  |  | ● |  |  |  |  | ● |  |  | ● |  |  |  | ● | 266,000 | 165,000 D |
STNet [64] |  |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  | ● | 650 | 650 |
ChartQA [65] | ● | ● |  |  |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  |  | ● |  | ● |  |  |  | ● |  |  |  |  | ● | ● |  |  |  | ● |  | 9,600 Q, 23, 100 A |
MultiVision [66] | ● |  |  |  |  |  |  |  | ● | ● | ● |  |  | ● |  |  |  |  | ● |  |  |  |  | ● |  |  |  |  | ● |  |  | ● |  |  |  | ● |  | 3,920,941 QA |
GoTreeScape [67] | ● |  |  | ● | ● |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  | ● |  |  | ● |  | ● | 62,340 | 62,340 |
Pictorial [68] |  |  |  |  |  |  |  |  | ● |  | ● | ● | ● | ● | ● |  |  |  | ● | ● |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 1,371 | 1,371 |
CoordNet [69] |  |  |  | ● |  |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  | ● |  |  |  |  |  | ● | ● |  |  |  | ● | 32,000 | 32,000 |
AutoTitle [70] |  | ● |  | ● |  |  |  |  |  |  |  |  |  |  |  |  |  | ● | ● | ● |  | ● |  |  |  | ● |  |  | ● |  |  | ● |  |  |  | ● |  | 6,000 fact-title |
OldVisOnline [71] | ● |  | ● |  |  |  |  |  | ● | ● | ● | ● | ● | ● | ● |  |  |  |  | ● |  | ● |  | ● |  |  |  |  |  |  |  |  |  | ● |  | ● | 13,511 |  | ●
MoneyVis [72] |  |  |  | ● |  |  |  | ● |  |  |  |  |  |  |  |  | ● |  |  |  | ● |  |  | ● |  |  |  |  |  |  | ● |  |  |  |  |  |  | 1 |

  * *

Abbreviations for Data Size: Data (D), Visualization (V), Question (Q), Answer
(A).

  * *

The online version can be accessed through: https://visdataset.github.io/.

##  2 Related Surveys

Wang et al. [2] discussed machine learning models applied to visualization.
Addressing artificial intelligence, Wu et al. [1] provided a review of AI
techniques in VIS research. Davila et al. [73] conducted a comprehensive
survey of Chart Mining, compiling works that support the recovery of
underlying data from charts. Liu et al. [74] made a detailed review of open
collections of visualization resources, including papers that collect
metadata, visualization survey papers that provide image browsing interfaces,
visualization books, and websites containing examples of visualization
programming libraries, etc. Chen and Liu [75] proposed a survey focused on the
creation of visualization corpora for automated chart analysis. Differing from
their approaches, our work primarily targets datasets as the research subject,
analyzing the combination of data forms to support various machine learning
tasks in visualization.

##  3 Methodology

###  3.1 Definition and Scope

A dataset of visualization is a collection of data used for training or
testing machine learning models in the visualization process. The following
criteria must be met for a dataset to be considered in this paper:

  * •

The dataset should not consist solely of underlying data used for
visualization purposes, such as tables (e.g., R datasets [76]), networks, or
volume data. For example, if a dataset only contains underlying data for
scientific visualization, it is not within our scope. However, if a dataset
includes derived batches of rendering results and supports deep learning work,
it is considered within our purview.

  * •

The dataset should primarily comprise data used in machine learning-related
work within the visualization domain or data employed in the training or
evaluation processes of machine learning models for visualization.

  * •

The dataset should mainly encompass large-scale, annotated visualization data.
This ensures that the dataset can facilitate the development and evaluation of
machine learning models.

We collected works from IEEE VIS, EuroVis, and PacificVis that utilized
machine learning models and employed training data in their methods. We
traversed the datasets used in these works, as well as the works they cited
and the works that cited them.

###  3.2 Coding

Figure 1 provides the overall conceptual framework, categorized according to
the ”what-why-how” three-level structure.

From the ”what” perspective, we discuss the components of a visualization
dataset, including whether it contains underlying data, detailed components,
its form of expression, and additional information. We annotated each dataset
with information on these dimensions. Additionally, we also focus on the scale
of the visualization dataset, including the number of visualizations and the
quantity of other forms. Regarding the tasks supported or potentially
supported by the dataset (WHY), we annotate on three levels: basic techniques,
visualization tasks, and user tasks. As for how the dataset is constructed
(HOW), we categorize the data into parts that can be directly scraped or
synthesized, referred to as raw data, and parts that require additional
annotation, termed data augmentation.

In the annotation process, all markings are done by at least two people. If
there are inconsistencies, another person independently annotates before a
discussion takes place. These details are all listed in Table I.

On our supplementary website, visdataset.github.io, we provide an interactive
system that supports filtering and selection based on several aspects,
including tasks, data formats, and construction methods. Details information
also include links to papers and the corresponding datasets.

Figure 2:  Relationship of different datasets in “What”. The underlying data
are objects for visual mapping and rendering to build visualization
components. These components are combined and laid out to compose a
visualization presentation, which can be enhanced with NL, such as
descriptions and annotations. NL can be an interface to create, query,
comment, and give feedback on a visualization. In addition, different forms of
visualization and their components can be used for tasks such as user
perception experiments and machine learning training.

##  4 What: Content of Dataset

Datasets of visualization are collections of data formats related to
visualization, including underlying data, visualization results (components
and presentation), and additional information, as depicted in Figure 2. The
underlying data refers to the content that needs to be visualized, such as
tabular and network data. The visualization results can be described through
their internal components and presentation format. Visualization components
include the content of the visualization, encompassing visual elements that
encode data and functional components like axes and legends. Visualization
presentation refers to the type of visualization. Additional information is
also a key aspect of visualization. While it is not a constitutive element of
visualization, it enhances the expressiveness of the visualization, such as
question and answer related to the visualization.

###  4.1 Underlying Data

Figure 3:  Example of underlying data and its visualization presentation [5].

The foundation for designing and creating visualizations is the underlying
data, as visualization serves as the medium for data presentation. As a
result, the underlying data is a crucial aspect of visualization datasets. The
most prevalent data formats are structured data, such as tabular, network, and
volume data. For instance, tabular data is the most widely used format and
enables the application of deep learning techniques in visualization
construction. VizNet [5] is a large-scale corpus of 31 million underlying data
obtained from open data repositories and online visualization galleries; the
underlying data have an average of 17 records across 3 attributes. Of the
entire corpus, 51% of dimensions recorded categorical data, 44% recorded
quantitative data, and the remaining dimensions recorded temporal data. VizNet
serves as a common reference for training automated visualization tools and
evaluating the effectiveness of visualization design, including comparing
visualization techniques and developing benchmark models and algorithms for
automated visual analysis. Figure 3 illustrates a visualization and its
corresponding underlying data. Text-to-Viz [29] provides a dataset of
proportion-related statements and corresponding infographics consisting of 200
distinct infographic sheets and 983 infographic units. It categorizes the
infographic units into several groups, including statistical-based,
proportion-based, quantity-based, change-based, rank-based, timeline-based,
process-based, and location-based infographics. The information from the
infographics is extracted using CRF-based approaches. SNAP [77] focuses on
network and graph datasets, with a collection of over 50 large network
datasets, ranging from tens of thousands to tens of millions of nodes and
edges, including social networks, web graphs, road networks, internet
networks, citation networks, collaboration networks, and communication
networks. ChartSense [17] provides the underlying data for chart images
through semi-automatic, interactive data extraction algorithms that are
optimized for each chart type. In scientific visualization, volume data [78,
79] and flow data [80, 81] have also garnered interest among researchers.

###  4.2 Visualization Components

Visualization components are constituents of a visualization presentation with
certain semantics and functions, including visual marks and other auxiliary
visual elements. For image-based visualizations, datasets are obtained from
visualization presentations through manual annotation or by using computer
vision techniques such as image segmentation and semantic segmentation. For
specifications-based visualization, the content can be directly parsed
according to the rules. There have been some approaches [16, 37] that labeled
visualization components (e.g., legends and axes) with trained deep-learning
models. REV [16] merged the images from three chart corpora: Vega Charts
(VEG), Quartz news website (QTZ) 111Quartz news website: https://qz.com/, and
Academic Paper Figures (ACA) for a total of 5,125 images. The reverse-
engineering visualizations recovered visual encoding from chart images using
inferred text elements. This work detects textual elements in charts,
classifies their role (e.g., chart title, X-axis label, Y-axis title, etc.),
and recovers the textual content using the optical character recognition (OCR)
technique. With the identified text elements and graphical mark type, the
encoding specifications of an input chart image are automatically inferred.
Visually29K [55] collects infographics with stylistically and semantically
diverse visuals from the Visually design website and textual elements and
labels stand-alone visual elements in infographics using a trained object
detector, including the icon positions and semantics. As shown in Figure 5,
each infographic is annotated with 1-9 tags out of a set of 391 tag
categories. Automatic-Annotation [16, 37] identified and extracted visual
elements in the target visualization, along with their visual properties, with
a Mask R-CNN model. This work also extracts texts in the chart using optical
character recognition (OCR) and parses the textual description using Natural
Language Processing (NLP) engines. Chen et al. [31] provided a synthesized
timeline dataset and a collected real-world timeline dataset from the
Internet. The information in the bitmap timelines is deconstructed using a
multi-task deep neural network, including the global information, i.e., the
representation, scale, layout, and orientation, and the local information,
i.e., the location, category, and pixels of each visual element on the
timeline. Lu et al. [40] annotated the positions and types of important
elements in infographics to provide information on their structure.

Figure 4:  The 10-category chart image corpus of Revision [9]. Figure 5:  The
output of the fully-automatic annotation system in visually29K [55].

###  4.3 Visualization Presentation

A visualization presentation can come in either raster or vector format and
can be either static or animated, with the option of being a single-view or
multi-view representation. These visualizations have various purposes,
including exploring visualization techniques, conducting user experiments on
visual perception, and training machine learning models with the underlying
data. Approaches, such as ReVision [9], ChartSense [17], and MASSVIS [10],
have collected static, single-view raster visualizations. ReVision [9] employs
computer vision and machine learning to identify chart types, such as pie
charts, bar charts, and scatterplots, in a bitmap image and creates an
interactive gallery of redesigned charts using principles of perception-based
design (as shown in Figure 4). ChartSense [17] uses a deep learning-based
classifier to determine the chart type, which can then be further analyzed or
improved for better perception. MASSVIS [10] collects a dataset of 2,070
single-panel visualizations from sources like news media, government reports,
scientific journals, and infographics, categorizing them by type (e.g., bar
charts and line graphs) and annotating them with attributes, such as data-ink
scale and visual density ratings. The visualizations are also assessed for
memory scores through Amazon’s Mechanical Turk to identify the key factors
that influence the retention and forgetting of visualizations. Beagle [22]
collects visualizations in vector graphics format. In addition to static,
single-view visualizations, multi-view [38] and animated visualizations [59,
56] have also been collected and studied.

###  4.4 Additional Information

The information that extends beyond the construction of a visualization
includes user queries, feedback, and comments. This information may be
incorporated into the visualization in the form of descriptions and
annotations or may be separate from the visual presentation as the natural
language interface and human perceptions. FigureQA [20] is a visual reasoning
dataset that includes questions and answers related to charts, with over one
million question-answer pairs based on over 100,000 images. The images are
synthetic and depict scientific-style figures belonging to five classes,
including line plots, dot-line plots, vertical and horizontal bar graphs, and
pie charts. Figure 6 shows an example of FigureQA. The dataset features
multiple plot elements and information synthesized spatially throughout the
figures, for instance, bounding-box annotations for all plot elements, which
can be used to train models to recognize patterns in visual representations of
data. Some datasets also capture users’ perceptions and understanding of
visualizations. Srinivasan et al. [60] collected a corpus that maps natural
language utterances to visualizations, with utterances labeled by over one
hundred participants in a series of ten visualizations for a given dataset.
This corpus can be used to evaluate existing natural language interfaces (NLI)
for data visualization and to develop new systems and models to generate
visualizations from natural language utterances. Moreover, human perception is
a crucial aspect of visualization, such as users’ recognition [10] and ratings
of aesthetics [12].

Figure 6:  Sample line plot figure with question-answer pairs in FigureQA
[20].

##  5 Why: Usage of Dataset

The integration of machine learning into visualization has enhanced its
generation, retrieval, exploration, and assessment. To implement machine
learning, visualization datasets are required as training data, which vary in
dataset components and data pairs for different tasks. We analyzed the tasks
and methods presented in current works, and organized the utilization of
datasets into a three-layer structure, as illustrated in Figure 7,
encompassing basic techniques, general tasks, and applications. This section
will examine the features of each level and the connections between the
different levels.

Figure 7:  We organized the usage of visualization datasets into a three-layer
structure, including basic machine learning techniques, general tasks, and
user tasks (from bottom to top). Each row represents a layer, and each
rectangular node represents papers belonging to that type. Additionally, each
strip between nodes represents papers belonging to both types. The width of
the node and strip is proportional to the number of papers.

###  5.1 Basic Techniques

In the field of machine learning, there are many common basic techniques such
as classification, regression, object detection, machine translation, OCR
(Optical Character Recognition), and dimension reduction. For each technique,
various machine learning algorithms have been proposed and trained on
different datasets to obtain a model that meets the requirements. In
visualization, these machine learning algorithms can be used as a means of
analyzing and understanding visualization and the results obtained can be used
for subsequent visualization recommendation, reverse engineering, and feature
extraction tasks.

Classification identifies which group or category a sample belongs to based on
some features or characteristics of the data. In the field of machine
learning, algorithms such as logistic regression, support vector machine,
random forest, decision tree, deep learning, etc. can be used to solve
classification tasks. In visualization, classification tasks include
classifying visualization chart types, such as ReVision [9], Vizbywiki [26],
ChartSense [17], and FigureSeer [15]. It also includes classifying symbols and
text in visualizations, with FigureSeer [15] and ReVision [9] using
convolutional neural networks to classify mark types, and FigureSeer [15]
training a classifier to determine the text role as a legend label.
Additionally, it includes classifying the data in visualizations, with Text-
to-Viz [29] training a text analyzer to extract segments from natural language
statements and generate infographics. Sherlock uses a deep neural network to
classify semantic types on data columns from the VizNet [5] corpus, which can
better aid in data mining and visualization.

Regression is a supervised task that predicts the value of a target variable
based on the input of independent variables. Traditional regression methods
include linear regression, ridge regression, random forests, and deep learning
methods. In visualization, regression tasks include evaluating and scoring
visualization. For example, the memorability and aesthetics of visualized
images are evaluated [82], and the analysis and evaluation of graph
visualization are performed [83, 84]. In recent years, the evaluation of graph
visualization has been a topic of interest, and machine learning techniques
have been employed to tackle the problem. Various readability metrics have
been proposed to model the layout of graph visualization. However, computing
these metrics can be time-consuming, especially for large and dense graphs. To
address this issue, Haleem et al. [83] proposed a deep learning-based approach
for directly evaluating the readability of graph layouts from images. They
designed a convolutional neural network architecture, trained it on a
benchmark dataset of graph layout images, and generated readability metrics
like node occlusion, edge crossing, and group overlapping. There is a
connection between the readability metrics and the user’s preference for a
graph layout. Cai et al. [84] leveraged the readability metrics to train a
deep learning model using transfer learning methods, which can predict
people’s preferences for a graph layout. For scientific visualization, there
are two categories: predicting the final rendering results and predicting the
rendering parameters needed to improve the results. Rendering parameters and
corresponding results are widely used as data pairs for scientific
visualization datasets for learning methods. Those two are used as input and
predicted targets for various tasks, including rendering synthesis [34, 35],
occlusion prediction [50], super-resolution [64, 69]. Completing regression
tasks can effectively reduce rendering costs and time consumption and improve
rendering quality by using learning methods appropriately.

Dimensionality reduction algorithms transform data from high-dimensional
spaces to low-dimensional spaces, preserving meaningful attributes of the
original data. Algorithms such as principal component analysis, t-SNE, and VAE
[85] are commonly used for dimensionality reduction. In visualization,
representing visualization images as high-dimensional vectors facilitates the
reconstruction of visualizations and exploration of the design space.
Visualization Assessment [82] uses VAE for dimensionality reduction of
visualization images for the reconstruction of new visualizations. ChartSeer
[48] and GoTreeScape [67] use variational autoencoders to generate high-
dimensional vector representations of graph visualization.

Translation is a process of using machine learning algorithms to translate
text from one language to another. In visualization, the translation task can
be abstracted as a process of transforming one object into another. For
example, Dibia et al.[32] view the visualization creation process as a direct
generation of specification files for visualization from data. Luo et al.[61]
translate the user-given natural language into structured visualization
queries, and then generate the desired visualization that meets the
requirements.

Object Detection algorithms are a category of computer vision algorithms that
aim to identify objects in images or videos and annotate them with bounding
boxes. Deep learning-based target detection algorithms such as Fast R-CNN [86]
and YOLO [87] have shown remarkable performance, providing great assistance
for computer understanding of images. These algorithms are extremely important
in the field of computer vision, as they can be applied to a wide range of
applications such as autonomous driving. In visualization, the use of target
detection algorithms can detect and extract the positions of visual elements
and function components (e.g., legends, axes, and titles) in visualization
images.

OCR (Optical Character Recognition) is a technology that enables the
recognition of text from images or scanned documents and its conversion into
editable electronic text. There are OCR algorithms based on rules, template
matching, and deep learning. In visualizations, to obtain the original
information contained in the visual elements, approaches [31, 37] often
extract text content such as titles, captions, and texts in the axes. Once the
text is extracted and processed, the information can be used to further
reconstruct the visualizations.

###  5.2 General Tasks

Building upon basic machine learning techniques, general tasks of
visualization can be constructed. The pipeline of visualization highlights the
importance of visualization recommendations, including the selection of
appropriate data attributes and visual mapping. Conversely, reverse
engineering aims to restore this information from existing visualizations.
Furthermore, another general task is feature extraction, which extracts
features from objects such as data and visualizations.

Visualization recommendation refers to the recommendation of suitable
visualizations based on data and user intent, including the selection of data
attributes and visual mappings. According to the basic techniques used in the
recommendation, existing works can be divided into three categories. The first
is the classification-based recommendation. For example, Hu et al. [7]
predicted the chart type of visualizations based on the features of data
tables. The second is the regression-based recommendation, such as DeepEye
[24], which enumerated the combination of data attributes and visual mappings,
scored them using regression methods, and recommended the best. The third is
the translation-based recommendation, such as translating data directly into a
visualization specification [32] or translating natural language into
structured visualization queries [61].

Reverse engineering refers to the process of extracting data and encoding
information from visualizations. A common pipeline consists of three steps
[88, 16]: deconstructing visualization images, recovering the visual mappings,
and extracting underlying data. The deconstruction of visualization components
involves detecting and categorizing various components in the visualization
image, such as graphics, axes, and legends. After deconstruction, the type of
chart needs to be predicted through classification, and visual mapping needs
to be inferred from components such as axes and legends through heuristic
rules. Finally, based on the visual mapping, the underlying data can be
extracted by regression tasks or position information of graphics.

Figure 8: Feature extraction: (a) Extracting data features explicitly in
AutoCaption [36], which generates descriptive feature expressions from visual
elements; (b) Extracting hidden space vectors of infographics implicitly,
through reconstructing the images by Autoencoders [82].

Feature extraction refers to the process of extracting features from objects
(e.g., visualizations, data, visual components). Based on the form of the
feature, it can be divided into explicit feature extraction and implicit
feature extraction. Explicit feature extraction represents the features
explicitly as categories, numerical values, and texts, corresponding to the
basic supervised techniques such as classification, regression, and
translation. For example, Liu et al. [36] built a dataset that contains data,
visual elements, and descriptive feature expressions to extract data features
(Figure 8 (a)). Implicit feature extraction entails the autonomous
identification of suitable data representations by a deep model. For example,
an autoencoder architecture functions by encoding data into a condensed vector
form and subsequently reconstructing it. Through rigorous training, the
vectors within this latent space are refined to effectively encapsulate the
essential characteristics of the original data form. For instance, Fu et al.
[82] used a dimensionality reduction method based on VAE to learn vectors from
a large number of infographics images (Figure 8 (b)). When data labels are
available, supervised machine learning can be used to generate implicit space
vectors. For example, Lee et al. [52] used a supervised trained chart
classification model to extract vectors from COVID-19 visualizations. In the
absence of data labels, dimensionality reduction methods can be used.

###  5.3 User Tasks

An information visualization process comprises data transformation,
visualization encoding, and visualization rendering. Once visualization
results are obtained, authors create descriptions, titles, or implement
question-answering methods to aid user understanding. Deep learning approaches
can facilitate parts of the process from data to visualization via supervised
fitting. Furthermore, with a set of visualizations, one can retrieve specific
visualizations, explore the set, or conduct evaluations based on it.

Generation refers to the creation of new information carriers from existing
ones. In the visualization area, generation encompasses the generation of
charts, volume rendering results, responses to questions, and the production
of annotations and captions. The generation of charts involves recommending
appropriate visual mappings based on a combination of data attributes. This is
typically performed through automatic generation from data, either by
iterating through different combinations of data attributes and visual
mappings and evaluating their effectiveness [24], or by directly recommending
a visualization specification based on the data [32, 7, 66]. A large number of
charts can be generated from a given dataset through these methods, but the
user often needs to manually inspect them to determine which ones best match
their intention. To integrate the user’s intention more effectively into the
generation process, some research has proposed generating charts from natural
language, which involves extracting tasks, data attributes, and values from
natural language statements and recommending visual representations
accordingly. For example, Text-to-Viz [29] extracts data entities related to
proportion from natural language statements and recommends infographics, while
Luo et al. [61] convert natural language into structured visualization queries
and recommend charts. Generation of volume rendering results refers to the
process of synthesizing volume rendering results through parameters such as
viewing points or enhancing the original results. Starting from the
parameters, the volume rendering result can be directly generated. The input
parameters can be the viewing point and the transfer function [27], as well as
the simulation parameters and visualization parameters [34]. Enhancements to
the original volume rendering results include super-resolution [64, 69] and
predicting ambient occlusion [50]. Han et al. [64] create high-resolution
images from low-resolution volume rendering images. Engel et al. [50] use a
deep learning model to predict ambient occlusion based on the opacity volume
derived from the intensity volume and the transfer function.

Question-answering is the process of obtaining information from data and
visualizations by asking questions. In the field of visualization, some
research generates natural language answers based on visualization and natural
language [20, 25, 65]. However, when users pose analytical questions, it can
sometimes be challenging to provide answers through simple natural language,
such as when inquiring about the correlation between two attributes. To tackle
this, certain approaches generate visualization results based on underlying
data and natural language questions [89, 61]. Unlike generating visualizations
from underlying data, some methods [46, 37] focus on question-answering (QA)
based on existing visualizations. For example, Kim et al.[46] suggested
describing the reasoning process in natural language alongside the answer. Lai
et al.[37] determine the content to be annotated using natural language, and
extract text and graphic entities for annotation through feature extraction
and reverse engineering before creating annotations by linking them, which
also represents a form of QA based on existing charts.

Figure 9: Retrieval: (a) searching by matching, which retrieves infographics
by hashtags[55]; (b) searching by the similarity of hidden space vectors in
ScatterNet [23], which is used to retrieve similar scatterplots.

Retrieval refers to the process of searching for a subset that meets certain
criteria from a collection of information carriers. This can involve searching
for visualizations that match certain conditions in a collection of
visualizations. The input for the query can include visualization images [23],
data attributes and visual mappings [26, 11, 15], or themes [6, 55]. Once the
query input is specified, a direct approach is to match it [11, 55]. For
example, Madan et al. [55] parse text and icon information from infographics
through reverse engineering and support keyword-based retrieval (Figure 9(a)).
The disadvantage of this matching approach is that the users need to specify
the query input accurately. Otherwise, they may miss the desired results. To
address this issue, some works propose representing the existing query input
as hidden space vectors and calculating similarities, such as ScatterNet [23],
which learns hidden space vectors based on the similarity of scatterplots and
uses them for retrieval (Figure 9(b)).

Figure 10: Exploration: Projection analysis with recommendations using the
hidden space vectors of charts in ChartSeer [48].

Exploration refers to the examination of data sets to uncover valuable
insights. One category is the analysis of visualization usage, such as Beagle
[22], which crawled a significant number of charts from the web, obtained the
chart types through explicit feature extraction, and analyzed the prevalent
chart types found on the internet. Madan et al. [55] parsed categories,
positions, and related text information of icons from infographics, and
synthesized themes and icons. Another category is the exploration of the
design space of visualizations, where users need to invest a considerable
amount of time in examining the vast space due to the abundance of options for
data attributes, data transformation methods, and design dimensions. To
support users in discovering relevant visualizations from the pool, some works
extract the hidden space vectors of charts through feature extraction methods,
which are then utilized for subsequent projection analysis and
recommendations. For example, ChartSeer [48] constructed a data set of a large
number of Vega-Lite [90] configuration files by traversing data attributes and
design dimensions to learn the hidden space vectors and assist users in
identifying suitable charts (Figure 10). In addition to basic charts, there
are explorations for tree visualizations [67] and volume rendering [91].

Figure 11: Assessment: Training a model using both a large-scale infographic
dataset and a small dataset of infographics with aesthetics labels [82].

Evaluation of visualization have been conducted from various efforts,
including the effectiveness of visual encoding [5], readability [83], user
preference [84], and aesthetics [82]. A straightforward approach is to use the
metrics values as inputs and outputs for models trained through explicit
feature extraction. For instance, Hu et al. [5] leveraged the visualization
effectiveness and encoding effectiveness to learning effectiveness. However,
this approach may not perform well when the dataset is small. To address this
issue, some works propose to obtain latent space vector representations of
visualizations through feature extraction, which can be used as inputs for the
above methods. For example, Fu et al. [82] trained a model using both a large-
scale infographic dataset and a small dataset of infographics with aesthetics
labels (Figure 11). Another solution is to first train a model using data
generated by algorithms and then fine-tune it using human-labeled data. Cai et
al. [84] used both readability metrics and user preference data to train a
deep learning model for predicting people’s preferences for graph layouts.

###  5.4 Findings

We analyzed the types of existing papers across different layers (Figure 7),
and figured out some key findings. The most commonly observed application is
generation, followed by retrieval, exploration, and evaluation. For general
tasks, feature extraction is the most prevalent, followed by visualization
recommendation and reverse engineering. Among the fundamental machine learning
methods, translation appears most frequently, followed by classification,
object detection, OCR, regression, and dimensionality reduction. In terms of
the relationships between different layers, several significant patterns
emerge. First, the process of generation is primarily accomplished through
feature extraction. This involves utilizing translation methods to extract
relevant features from the data and subsequently generating visualizations.
Second, when it comes to visualization recommendation and generation,
classification techniques play a crucial role. These techniques are employed
to classify and categorize visualization data, enabling the system to
recommend suitable visualizations. Third, for exploration and retrieval
purposes, feature extraction plays a key role, and it is predominantly
achieved through classification techniques. Fourth, in the context of reverse
engineering for exploration and retrieval, multiple techniques are used. This
involves the use of classification techniques, along with object detection and
OCR techniques.

##  6 How: Construction of Dataset

Constructing visualization datasets includes raw data construction and data
augmentation, as shown in Figure 12. Raw data means data information that can
be directly obtained from documents, usually with only one type of data
information, including underlying data like data tables or graph data, and
visualizations in the formats of raster images, and vector images. Processing
of raw data calculating derived information to be included in the datasets.
The annotation of raw data can include detailed information related to
visualizations and human perception.

Figure 12:  Dataset construction consists of raw data construction and data
augmentation obtaining detailed information from visualizations.

###  6.1 Raw Data Construction

The raw data mainly comes from five sources, namely, crawling, extraction,
crowdsourcing, synthesizing, and combining. Some approaches [51] collect the
data by crowdsourcing, utilizing the power of the general public. Synthesizing
[20, 25] dataset can construct a large scale of the dataset. The limitation of
the synthesized dataset is that it lacks diversity in underlying data, visual
encoding, and style. Therefore, other works directly collect desired data
types (e.g. visualizations) from the real world like online websites, for
example, Beagle [22] and ReVision [9]. Some visualizations are stored in
documents, e.g., PDF files. Extraction methods are applied to collect
visualizations in these media, to create visualization datasets, for example,
VisImages [39].

Automatic web crawling can be conducted either with or without a specified
list of websites. The list may include multiple centralized websites, such as
news media websites that provide data news, visualization blogs, government
agencies, or international organizations that publish data reports. For
instance, MASSVIS [10] crawls visualizations from various websites across
different fields automatically. D3 search [30] crawls D3-based visualization
collection sites, while VizML [7] crawls data and visual encoding
specifications from the Plotly website [92]. InfoVIF [40] collects
infographics from two infographic design websites. Another centralized
collection approach utilizes search engine sites, such as video search on
video sites like YouTube and Vimeo, and image search functions on sites like
Google, Flickr, Twitter, and Reddit. This approach is widely used in the field
of natural images; for example, ImageNet [3] searches for images and tags
using keywords from WordNet [93]. In the visualization domain, the dataset for
training automatic annotation models [37] is collected by using visualization
types as keywords for crawling with search engines. Shi et al. [57] also use
video search engine results as one of their data sources when collecting data
videos. The advantage of web crawling is that it can involve visualizations
used and disseminated in reality, and novel visualizations can be collected to
construct the visualization datasets. However, it also has disadvantages, such
as the difficulty in identifying websites, the similarity of visualizations
from the same website, and the potential for bias in the analysis results due
to the way of specifying data sources. Additionally, appropriate crawling and
web structure analysis strategies need to be developed for different websites,
and the selection of keywords and filtering of visualization images must be
considered for search engine types. This method is limited in the amount of
visualization data collected and may also suffer from bias due to the
limitations of search engines and recommendation algorithms. The automatic
approach requires a special design to filter relevant images for
visualization, and although some existing work can be automated, the dataset
collection process is often performed with manual review.

Automatic extraction from static documents is a valuable technique for
acquiring visualization-related components, including visualization diagrams
and descriptions. One example of static documents are PDF files. Parsing PDF
files enables the analysis of academic papers, utilizing sophisticated
toolkits capable of accurately determining the position of text and images.
Several research initiatives, including DiagramFlyer [11], FigureSeer [15],
SCICAP [62], and VizioMetrix [94], have employed automatic extraction and
visualization methods to obtain visualization and extract underlying data.
Additionally, multi-view visualization datasets, such as those collected by
Chen et al [38] and VisImages [39], are extracted from professional
visualization papers and are considered to be of high quality due to their
design and evaluation by experts. Other files like Excel spreadsheets may also
contain data and visualizations and can be used to construct visualization
datasets [54]. However, the challenge of extracting visualization datasets
from static documents includes the diverse formats and increased error in the
extraction process.

Crowd-Sourcing collection has the potential to improve the quality of data
collection from decentralized sources. In the case of the CrisisVis dataset
[51], some of the visualizations were obtained through word-of-mouth and
manually extracted and labeled by the authors. SightLine [21] uses a browser
plugin to extract visualizations and upload them to the server based on set
heuristic rules as the user browses the visualization page. However, this
approach faces challenges in finding users and may raise privacy concerns.
Sites such as Plotly [92] allow users to upload raw data and set up encoding
mappings or select templates to generate visualizations. Therefore, these
sites collect a large amount of real raw data and user-defined mappings. The
disadvantage is that these methods require a lot of time and other expenses
and are suitable as a long-term collection tool, while it is difficult to
guarantee the number of collections in the short term. Moreover, these methods
require a platform with a large influence to attract users.

Synthesizing of Data and Visualization. Data and visualization can be
synthesized based on rules to reduce the cost of collecting large volume
datasets. For example, FigureQA [20] generates datasets for training QA tasks
for visualization. Synthesizing method is also often used in scientific
visualization dataset construction. Appropriate rendering techniques are
required to obtain scientific visualizations. The data required by these
learning models mainly includes paired information: rendering results (images)
and corresponding parameters. The most important part of constructing a
scientific visualization dataset lies in the rendering parameters. When facing
different tasks, the corresponding parameters needed by the learning models
are also different. However, in most cases, researchers will sample the
corresponding parameter space based on rule-based methods to construct a
visualization dataset. For example, Berger et al. [27] and Hong et al. [35]
used the viewpoint and transfer function as parameters, and constructed
datasets for parameter to rendering results pairs. He et al. [34] sampled the
ensemble data simulation parameter space, added parameters on the basis of
viewpoint and transfer function, and constructed training datasets for
ensemble scientific data. Engel et al.[50] explored different opacity mappings
and corresponding ambient occlusion in direct volume rendering results, and
established a dataset for deep learning-based techniques for volumetric
ambient occlusion. In some cases, those rendering parameters are hidden behind
datasets. Han et al.[69] used a uniform coordinate representation to
incorporate parameters for different learning tasks into the same framework,
and designed a corresponding universal training network. In addition, Han et
al. [64] constructed a dataset for super-resolution, which only includes pairs
of input low-resolution rendering results and targeted high-resolution
results. However, they still traversed the rendering parameter space when
constructing this training dataset. The corresponding generation parameters
were not directly input to the adversarial network since super-resolution
tasks do not require them. Overall, the methods used in constructing training
datasets in the field of scientific visualization are all based on sampling in
the corresponding parameter space and obtaining visualization results using
rendering methods based on those parameters. Thus, researchers can construct
data pairs of parameters and render results as training data for deep learning
models. It is worth noting that work dedicated to using machine learning for
tasks such as interpolation prediction and feature detection is not included
as they focus on the underlying data instead of visualizations.

Combination of Existing Datasets. Some work has been devoted to collecting
data from the Internet Web, such as WebTables [95], while there are also
platforms that allow users to expose their data and visualizations like Plotly
[92] and Tableau [96]. Datasets from different sources can be combined
together by converting into a unified format. VizNet [5] contains publicly
available datasets from WebTables and Manyeyes and also provides cleaning of
the data and calculation of important statistical values.

###  6.2 Data Augmentation

Data augmentation consists of the process of augmenting existing components
with additional information components from like visualization type,
visualization component, etc. Based on whether the augmentation is achieved by
humans or machines, the augmentation method can be divided into three
categories, including manual method (e.g.,, icon detection [55], bounding box
[39], and natural language utterances [60]), rule-based method, and learning-
based method get new information based on the knowledge distilled from
training data. Datasets can be extended by combining multiple augmentation
methods to leverage their benefits.

Manual Method. Datasets can be augmented by involving humans to label
information including visualization types and component positions and types,
collecting perception data by asking participants to conduct tasks on
visualizations, or asking people to describe visualizations with language.
Visualization-type information is obtained by human labeling in datasets such
as Revision [9], MASSVIS [10], and that of Choi et al. [88] Visual components
in standard visualization charts are manually annotated with positions and
component types [37, 16], and views in multi-view visualization systems are
also annotated [38]. Infographics with more diverse styles are annotated by
crowdsourcing to get information like icons [31, 55], texts [31], and
structural information [40]. The result of icons and texts annotated in a
timeline is illustrated in Figure 13 (1). The part of the visualization
dataset that pertains to perceptual cognition is typically obtained through
user experiments. Accuracy [97, 10] of completing tasks are commonly
collected. Human subject assessment of visualizations is also an important
target to collect like aesthetics ratings [12], the similarity of
visualization pairs [23], ranking of visualization quality [98]. Relative
ranking among visualizations is often collected instead of absolute ratings
[23]. While natural language is an important way to interact with
visualizations and communicate information and insights inside visualizations,
they are usually collected in a crowdsourcing way to get more diverse and
natural expressions. Descriptions and answers on visualizations are analyzed
to understand patterns in them [46, 60]. The manual method is also used to
derive templates [36] or get paraphrased expressions of existing sentences
[99] to augment the collected dataset. The manual method enables the creation
of more accurate and natural datasets but costs more than automatic methods.

Figure 13:  (a) Icon and text annotation in the Timeline dataset [31] (b)
Datasets of icons in infographics proposed by Madan et al. [55]

Rule-based Method. The raw data collected during the construction of the
dataset may need further processing to be suitable for visualization tasks.
Such processing includes the rule-based method for data cleaning and data
derivation. For instance, MultiVision [66] filtered out data tables with more
than ten columns and Moreover, charts encoded over four columns from the Excel
corpus in Table2Charts [100]. Meta information of visualizations and
underlying data can be derived by rule-based methods. For example, VizNet [5]
is derived from meta-information with algebraic calculations. Hulsebos et al.
[33] generate semantic types of attributes by matching templates. Liu et al.
[36] synthesized caption templates to expand the visualization to caption
dataset. Information regarding the data and visual mappings in visualizations
is typically embedded within the visualization charts, however, such
information is not always provided simultaneously. D3 search [30] acquires
information about underlying data and mappings by parsing the SVG-based
visualizations generated by D3. The D3 search leverages the capability of D3
to bind data to HTML document objects, as well as heuristics of parsing
visualizations, to enable data and template extraction [101]. The rule-based
method can be applied to increase the amount of training data. For example,
some icons in Visually29K [55] are generated icons by detecting blank areas
and adding icons randomly and the location and size information is obtained
during the generation process, as illustrated in Figure 13 (2). Natural
language data like descriptions of visualization content or question-answer
pairs based on charts are totally or partially generated with rule-based
methods like generating based on templates in AutoCaption [36], FigureQA [20],
DVQA [25], PlotQA [45], LEAF-QA [43], and STL-CQA [44]. These templates are
manually created by the dataset creators or summarized from manually collected
sentences. FigureQA [20] is further converted into visualization and caption
pairs from figure caption dataset [47]. The rule-based method can extend
datasets with a relatively small cost but it suffers from the problem of
lacking diversity of real-world visualizations.

Learning-based Method. Dataset augmentation can be achieved through learning-
based methods. For instance, Beagle [22] trains a support vector machine to
categorize visualization types based on the number and distribution of various
elements. AutoTitle [70] use large language model to generate the fact to
title dataset. Tools such as DiagramFlyer [11], FigureSeer [15], and the work
of Choudhury et al. [102] extract common data from visual graphs with deep
learning models like Convolutional Neural Network. Among these works,
FigureSeer [15] identifies neatly aligned text to determine the coordinates of
the axes and classifies whether the text is part of the legend by using a
trained random forest method. The reverse-engineered dataset [16] employs
Convolutional Neural Networks to classify whether each pixel is text or not,
and then further categorize the labels and titles of the axes and the text in
the legend based on the arrangement pattern of the text. The inclusion of
natural language elements, such as captions, questions, and answers, has
become increasingly important in visualization datasets. Learning methods can
also serve as supplements to existing data methods (manual-based or ruled-
based), such as enhancing the diversity of natural language through large
language models. For example, Luo et al. [61] use translation techniques in
the field of natural language processing to improve the diversity and accuracy
of the templates generated. Fu et al. [99] aims to reduce the cost of
crowdsourcing annotation while collecting high-quality natural language
questioning data on a larger scale. They divide the annotation process into
two parts: expert users asking questions based on the task and data, and
ordinary users paraphrasing the expert users’ questions to increase the
diversity of the natural language data. Furthermore, Zhang et al. [103]
proposed a solution to alleviate the tedious process of manual labeling by
introducing a system that utilizes active learning to generate annotations and
minimize the requirement for human labor. The learning method helps generate
datasets mimicking a real-world style, however, it is important to ensure the
quality of the final results as the quality of data largely influences the
quality of model [104]. Liu et al [70] use LLM to rewrite the natural language
of title content to construct a fact-to-title pair dataset. To ensure the
quality of the training dataset, they check the quality manually.

##  7 Research Challenge and Discussion

In recent years, machine learning methods have played a crucial role in the
visualization process. Datasets serve as the foundation for these works, and
many studies have recognized their importance and made initial attempts.
However, the formats used in data visualization are diverse, ranging from
specifications (e.g., Vega-Lite [90]), intermediate representations (e.g., D3
[105]), vector graphics (e.g., SVG), to raster graphics (e.g., JPG).

###  7.1 Visualization Datasets Standardization

The conversion between these visualization formats is often challenging and
may involve risks such as loss of information and inability to convert with
precision. A standardized visualization data structure should strive to be
independent of specific tools and file formats and should be universally
accessible to both machines and humans. This pursuit aims to establish a
unified data structure for expressing visual information that can become a new
standard for visualization files. Such a data structure can inherently encode
the maximum amount of visual data and mapping information.

Another kind of standardization for visualization datasets is the use of a
multi-modal transformer. Multi-modal universal methods for different
visualization learning tasks, such as classification, visualization
representation, and visualization-natural language representation, should be
developed by establishing a shared pre-trained neural network foundation. In
natural language and images domains, multi-modal universal models (for
example, CLIP [106] and ViT [107]) seek to represent different forms of
information through a unified latent space, thus supporting translation
between these forms. By leveraging the large-scale, weakly-supervised
visualization-natural language correspondences in network media, models
similar to those used in large-scale image-natural language learning can be
trained using large-scale, weakly-supervised datasets to learn universal
values [108].

Figure 14:  The x-axis encodes the release year, the y-axis presents the
number of data items, and the color represents the data construction method.
The scale of datasets used for visualization is growing over time, and as
these datasets begin to expand, existing visualization datasets are often
limited in terms of scale. Consequently, large-scale datasets are typically
constructed through methods such as web crawling or synthesizing.

###  7.2 Visualization Datasets Size

The size, construction method, and release year of the datasets are shown in
Figure 14. The figure demonstrates an evident trend in recent years: a
significant increase in the number of larger and more extensive visualization
datasets. In terms of construction methods, there has been a gradual shift
from crawling methods to the inclusion of extraction, crowdsourcing,
synthesizing, and combining techniques. Compared with datasets in
image222Image: https://paperswithcode.com/datasets?mod=images or natural
language area333Text: https://paperswithcode.com/datasets?mod=texts,
visualization datasets are often limited in scale because of heavy cost in
manual annotation, and large-scale data is typically obtained through crawling
or rule-based synthesis without human labeling. However, many machine learning
tasks, particularly end-to-end tasks, require a large amount of labeled
training data. Data labeling is a topic of interest in academic circles, and
studies such as Zhang et al. [103] summarize the primary processes involved in
labeling. To label large-scale data, a significant amount of manual effort is
typically required. Therefore, it is worthwhile to investigate ways to expand
the scale of data while reducing manpower investment and ensuring quality. One
possible approach is to establish an intelligent data annotation process that
involves human input to ensure the quality and quantity of visualization
annotations. This process requires the machine to continuously learn from the
annotation results and provide feedback to the annotation loop, as MI3 did
[109]. As users become more familiar with the annotation process, the machine
can provide more initial annotations, enabling users to focus on the data sets
that the machine does not understand and avoiding an infinite annotation
process.

###  7.3 Visualization Datasets Openness

Easy access to visualization datasets can be facilitated through three primary
methods. The most commonly used approach is to provide users with a download
link, where the entire dataset is compressed and made available for local
usage after downloading. This method is best suited for datasets with a
relatively low volume. However, for large datasets, this method can be time-
consuming, which presents a significant constraint to the widespread
utilization and dissemination of these datasets. An alternative approach is to
provide users with data construction techniques. For synthesized datasets, the
authors should provide a detailed description of the construction methodology,
including the models, coefficients, initial parameters, simulation equations,
and other relevant details. By providing this information, other researchers
can independently reproduce the synthetic datasets locally, eliminating the
need for network transmission of the datasets and improving their usage.
However, this method is not appropriate for non-synthetic datasets,
particularly those that require collection and annotation processes, as these
processes contribute significant value to the release of the datasets.

Cloud services have emerged as a prevalent method for data management and
access. These services involve hosting datasets on servers that respond to
client requests by transmitting the data. Many of these servers are located on
supercomputers, which offer additional benefits such as online data processing
capabilities. This eliminates the need for clients to download datasets, as
they can simply log into their accounts to access them. Some datasets, such as
Gaia[81], are actively seeking to establish a platform community that not only
facilitates remote data access but also provides additional resources, such as
user-friendly data processing methods and shared discussion spaces. While
building such a platform is a significant undertaking, it has the potential to
create a more supportive research and communication environment for the
community.

Despite the significant progress in the field of visualization and machine
learning, there is currently a lack of a unified and accessible platform for
sharing visualization datasets. In the natural language and image domains,
standard datasets are available for most tasks, including both training and
test sets. Relevant papers for a specific task are also listed for comparison
and evaluation. In contrast, AI-for-visualization approaches often spend a
significant amount of time on data construction, and it is difficult to
conduct cross-comparative evaluations due to data openness issues. Therefore,
we call for a more open data platform that provides data standards, supports
data uploads, and provides standard training and benchmark data for
visualization tasks. Such a platform would help to streamline the data
construction process and promote better cross-comparative evaluations of
visualization approaches.

## Acknowledgments

This work is supported by NSFC No. 62272012.

## References

  * [1] A. Wu, Y. Wang, X. Shu, D. Moritz, W. Cui, H. Zhang, D. Zhang, and H. Qu, “AI4VIS: Survey on artificial intelligence approaches for data visualization,” _IEEE Trans. Vis. Comput. Graph._ , vol. 28, no. 12, pp. 5049–5070, 2022.
  * [2] Q. Wang, Z. Chen, Y. Wang, and H. Qu, “A survey on ml4vis: Applying machine learning advances to data visualization,” vol. 28, no. 12, p. 5134–5153, dec 2022. [Online]. Available: https://doi.org/10.1109/TVCG.2021.3106142
  * [3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2009, pp. 248–255.
  * [4] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2425–2433.
  * [5] K. Hu, S. Gaikwad, M. Hulsebos, M. A. Bakker, E. Zgraggen, C. Hidalgo, T. Kraska, G. Li, A. Satyanarayan, and Ç. Demiralp, “Viznet: Towards a large-scale visualization learning and benchmarking repository,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2019, pp. 1–12.
  * [6] M. Oppermann, R. Kincaid, and T. Munzner, “Vizcommender: Computing text-based similarity in visualization repositories for content-based recommendations,” _IEEE Trans. Vis. Comput. Graph._ , 2020.
  * [7] K. Hu, M. A. Bakker, S. Li, T. Kraska, and C. Hidalgo, “VizML: A machine learning approach to visualization recommendation,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2019, pp. 1–12.
  * [8] J. Leskovec and A. Krevl, “SNAP Datasets: Stanford large network dataset collection,” http://snap.stanford.edu/data, Jun. 2014.
  * [9] M. Savva, N. Kong, A. Chhajta, L. Fei-Fei, M. Agrawala, and J. Heer, “ReVISion: Automated classification, analysis and redesign of chart images,” in _Proceedings of the ACM Symposium on User Interface Software and Technology_ , 2011, pp. 393–402.
  * [10] M. A. Borkin, A. A. Vo, Z. Bylinskii, P. Isola, S. Sunkavalli, A. Oliva, and H. Pfister, “What makes a visualization memorable?” _IEEE Trans. Vis. Comput. Graph._ , vol. 19, no. 12, pp. 2306–2315, 2013.
  * [11] Z. Chen, M. Cafarella, and E. Adar, “DiagramFlyer: A search engine for data-driven diagrams,” in _Proceedings of the International Conference on World Wide Web_ , 2015, pp. 183–186.
  * [12] L. Harrison, K. Reinecke, and R. Chang, “Infographic aesthetics: Designing for the first impression,” in _Proceedings of the ACM Conference on Human Factors in Computing Systems_ , 2015, pp. 1187–1190.
  * [13] P. Pasupat and P. Liang, “Compositional semantic parsing on semi-structured tables,” in _Proceedings of the Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_. Association for Computational Linguistics, Jul. 2015, pp. 1470–1480.
  * [14] R. A. Rossi and N. K. Ahmed, “An interactive data repository with visual analytics,” _SIGKDD Explor. Newsl._ , vol. 17, no. 2, p. 37–41, feb 2016\. [Online]. Available: https://doi.org/10.1145/2897350.2897355
  * [15] N. Siegel, Z. Horvitz, R. Levin, S. Divvala, and A. Farhadi, “FigureSeer: Parsing result-figures in research papers,” in _Proceedings of the European Conference on Computer Vision_. Springer, 2016, pp. 664–680.
  * [16] J. Poco and J. Heer, “Reverse-engineering visualizations: Recovering visual encodings from chart images,” in _Computer Graphics Forum_ , vol. 36, no. 3. Wiley Online Library, 2017, pp. 353–363.
  * [17] D. Jung, W. Kim, H. Song, J.-i. Hwang, B. Lee, B. Kim, and J. Seo, “Chartsense: Interactive data extraction from chart images,” in _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ , 2017, pp. 6706–6717.
  * [18] F. Amini, N. H. Riche, B. Lee, A. Monroy-Hernandez, and P. Irani, “Authoring data-driven videos with dataclips,” _IEEE Trans. Vis. Comput. Graph._ , vol. 23, no. 1, pp. 501–510, 2017.
  * [19] N. W. Kim, Z. Bylinskii, M. A. Borkin, K. Z. Gajos, A. Oliva, F. Durand, and H. Pfister, “Bubbleview: An interface for crowdsourcing image importance maps and tracking visual attention,” _ACM Transactions on Computer-Human Interaction (TOCHI)_ , vol. 24, no. 5, pp. 1–40, 2017.
  * [20] S. E. Kahou, V. Michalski, A. Atkinson, Á. Kádár, A. Trischler, and Y. Bengio, “FigureQA: An annotated figure dataset for visual reasoning,” _arXiv preprint arXiv:1710.07300_ , 2017.
  * [21] J. Sechler, L. Harrison, and E. M. Peck, “Sightline: Building on the web’s visualization ecosystem,” in _Proceedings of the ACM SIGCHI Conference Extended Abstracts on Human Factors in Computing Systems_ , 2017, pp. 2049–2055.
  * [22] L. Battle, P. Duan, Z. Miranda, D. Mukusheva, R. Chang, and M. Stonebraker, “Beagle: Automated extraction and interpretation of visualizations from the web,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2018, pp. 1–8.
  * [23] Y. Ma, A. K. Tung, W. Wang, X. Gao, Z. Pan, and W. Chen, “Scatternet: A deep subjective similarity model for visual analysis of scatterplots,” _IEEE Trans. Vis. Comput. Graph._ , vol. 26, no. 3, pp. 1562–1576, 2018.
  * [24] Y. Luo, X. Qin, N. Tang, and G. Li, “DeepEye: Towards automatic data visualization,” in _Proceedings of the IEEE International Conference on Data Engineering_. IEEE, 2018, pp. 101–112.
  * [25] K. Kafle, B. Price, S. Cohen, and C. Kanan, “DVQA: Understanding Data Visualizations via Question Answering,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE, jun 2018, pp. 5648–5656. [Online]. Available: https://ieeexplore.ieee.org/document/8578690/
  * [26] A. Y. Lin, J. Ford, E. Adar, and B. Hecht, “Vizbywiki: Mining data visualizations from the web to enrich news articles,” in _Proceedings of the 2018 World Wide Web Conference_ , 2018, pp. 873–882.
  * [27] M. Berger, J. Li, and J. A. Levine, “A generative model for volume rendering,” _IEEE Trans. Vis. Comput. Graph._ , vol. 25, no. 4, pp. 1636–1650, 2018.
  * [28] D. Haehn, J. Tompkin, and H. Pfister, “Evaluating ‘graphical perception’with cnns,” _IEEE Trans. Vis. Comput. Graph._ , vol. 25, no. 1, pp. 641–650, 2018.
  * [29] W. Cui, X. Zhang, Y. Wang, H. Huang, B. Chen, L. Fang, H. Zhang, J.-G. Lou, and D. Zhang, “Text-to-Viz: Automatic generation of infographics from proportion-related natural language statements,” _IEEE Trans. Vis. Comput. Graph._ , vol. 26, no. 1, pp. 906–916, 2019.
  * [30] E. Hoque and M. Agrawala, “Searching the visual style and structure of d3 visualizations,” _IEEE Trans. Vis. Comput. Graph._ , vol. 26, no. 1, pp. 1236–1245, 2019.
  * [31] Z. Chen, Y. Wang, Q. Wang, Y. Wang, and H. Qu, “Towards automated infographic design: Deep learning-based auto-extraction of extensible timeline,” _IEEE Trans. Vis. Comput. Graph._ , vol. 26, no. 1, pp. 917–926, 2019.
  * [32] V. Dibia and Ç. Demiralp, “Data2Vis: Automatic generation of data visualizations using sequence-to-sequence recurrent neural networks,” _IEEE Computer Graphics and Applications_ , vol. 39, no. 5, pp. 33–46, 2019\.
  * [33] M. Hulsebos, K. Hu, M. Bakker, E. Zgraggen, A. Satyanarayan, T. Kraska, Ç. Demiralp, and C. Hidalgo, “Sherlock: A deep learning approach to semantic data type detection,” in _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019, pp. 1500–1508.
  * [34] W. He, J. Wang, H. Guo, K.-C. Wang, H.-W. Shen, M. Raj, Y. S. Nashed, and T. Peterka, “Insitunet: Deep image synthesis for parameter space exploration of ensemble simulations,” _IEEE Trans. Vis. Comput. Graph._ , vol. 26, no. 1, pp. 23–33, 2019.
  * [35] F. Hong, C. Liu, and X. Yuan, “DNN-VolVis: Interactive volume visualization supported by deep neural network,” in _Proceedings of the IEEE Pacific Visualization Symposium_. IEEE, 2019, pp. 282–291.
  * [36] C. Liu, L. Xie, Y. Han, D. Wei, and X. Yuan, “Autocaption: An approach to generate natural language description from visualization automatically,” in _Proceedings of the 2020 IEEE Pacific Visualization Symposium_. IEEE, 2020, pp. 191–195.
  * [37] C. Lai, Z. Lin, R. Jiang, Y. Han, C. Liu, and X. Yuan, “Automatic annotation synchronizing with textual description for visualization,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–13.
  * [38] X. Chen, W. Zeng, Y. Lin, H. M. Al-Maneea, J. Roberts, and R. Chang, “Composition and configuration patterns in multiple-view visualizations,” _IEEE Trans. Vis. Comput. Graph._ , 2021.
  * [39] D. Deng, Y. Wu, X. Shu, M. Xu, J. Wu, S. Fu, and Y. Wu, “VisImages: A large-scale, high-quality image corpus in visualization publications,” _arXiv e-prints_ , pp. arXiv–2007, 2020.
  * [40] M. Lu, C. Wang, J. Lanir, N. Zhao, H. Pfister, D. Cohen-Or, and H. Huang, “Exploring visual information flows in infographics,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–12.
  * [41] C. Qian, S. Sun, W. Cui, J.-G. Lou, H. Zhang, and D. Zhang, “Retrieve-then-adapt: Example-based automatic generation for proportion-related infographics,” _IEEE Trans. Vis. Comput. Graph._ , vol. 27, no. 2, pp. 443–452, 2020.
  * [42] F. Hohman, M. Conlen, J. Heer, and D. H. P. Chau, “Communicating with interactive articles,” _Distill_ , vol. 5, no. 9, p. e28, 2020.
  * [43] R. Chaudhry, S. Shekhar, U. Gupta, P. Maneriker, P. Bansal, and A. Joshi, “Leaf-qa: Locate, encode & attend for figure question answering,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 3512–3521.
  * [44] H. Singh and S. Shekhar, “Stl-cqa: Structure-based transformers with localization and encoding for chart question answering,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020, pp. 3275–3284.
  * [45] N. Methani, P. Ganguly, M. M. Khapra, and P. Kumar, “Plotqa: Reasoning over scientific plots,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 1527–1536.
  * [46] D. H. Kim, E. Hoque, and M. Agrawala, “Answering Questions about Charts and Generating Visual Explanations,” _Conference on Human Factors in Computing Systems - Proceedings_ , pp. 1–13, 2020.
  * [47] C. Chen, R. Zhang, E. Koh, S. Kim, S. Cohen, and R. Rossi, “Figure captioning with relation maps for reasoning,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , March 2020.
  * [48] J. Zhao, M. Fan, and M. Feng, “ChartSeer: Interactive steering exploratory visual analysis with machine intelligence,” _IEEE Trans. Vis. Comput. Graph._ , vol. 2626, no. c, pp. 1–1, 2020.
  * [49] D. Shi, X. Xu, F. Sun, Y. Shi, and N. Cao, “Calliope: Automatic visual data story generation from a spreadsheet,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 27, no. 2, pp. 453–463, 2020.
  * [50] D. Engel and T. Ropinski, “Deep volumetric ambient occlusion,” _IEEE Trans. Vis. Comput. Graph._ , vol. 27, no. 2, pp. 1268–1278, 2020.
  * [51] Y. Zhang, Y. Sun, L. Padilla, S. Barua, E. Bertini, and A. G. Parker, “Mapping the landscape of covid-19 crisis visualizations,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–23.
  * [52] C. Lee, T. Yang, G. D. Inchoco, G. M. Jones, and A. Satyanarayan, “Viral visualizations: How coronavirus skeptics use orthodox data practices to promote unorthodox science online,” in _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–18.
  * [53] J. Chen, M. Ling, R. Li, P. Isenberg, T. Isenberg, M. Sedlmair, T. Moller, R. S. Laramee, H.-W. Shen, K. Wunsche _et al._ , “Vis30k: A collection of figures and tables from ieee visualization conference publications,” _IEEE Trans. Vis. Comput. Graph._ , 2021.
  * [54] J. Luo, Z. Li, J. Wang, and C.-Y. Lin, “Chartocr: Data extraction from charts images via a deep hybrid framework,” in _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2021, pp. 1917–1925.
  * [55] S. Madan, Z. Bylinskii, C. Nobre, M. Tancik, A. Recasens, K. Zhong, S. Alsheikh, A. Oliva, F. Durand, and H. Pfister, “Parsing and Summarizing Infographics with Synthetically Trained Icon Detection,” in _2021 IEEE 14th Pacific Visualization Symposium_. IEEE, apr 2021, pp. 31–40. [Online]. Available: https://ieeexplore.ieee.org/document/9438764/
  * [56] D. Shi, F. Sun, X. Xu, X. Lan, D. Gotz, and N. Cao, “AutoClips: An automatic approach to video generation from data facts,” in _Computer Graphics Forum_ , vol. 40, no. 3. Wiley Online Library, 2021, pp. 495–505.
  * [57] Y. Shi, X. Lan, J. Li, Z. Li, and N. Cao, “Communicating with motion: A design space for animated visual narratives in data videos,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–13.
  * [58] X. Lan, Y. Shi, Y. Wu, X. Jiao, and N. Cao, “Kineticharts: Augmenting affective expressiveness of charts in data stories with animation design,” _IEEE Trans. Vis. Comput. Graph._ , 2022.
  * [59] X. Shu, A. Wu, J. Tang, B. Bach, Y. Wu, and H. Qu, “What makes a data-gif understandable?” _IEEE Trans. Vis. Comput. Graph._ , vol. 27, no. 2, pp. 1492–1502, 2021.
  * [60] A. Srinivasan, N. Nyapathy, B. Lee, S. M. Drucker, and J. Stasko, “Collecting and characterizing natural language utterances for specifying data visualizations,” in _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–10.
  * [61] Y. Luo, N. Tang, G. Li, C. Chai, W. Li, and X. Qin, “Synthesizing natural language to visualization (nl2vis) benchmarks from nl2sql benchmarks,” in _Proceedings of the 2021 International Conference on Management of Data_ , 2021, pp. 1235–1247.
  * [62] T.-Y. Hsu, C. L. Giles, and T.-H. Huang, “SciCap: Generating captions for scientific figures,” in _Findings of the Association for Computational Linguistics: EMNLP 2021_. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 3258–3264. [Online]. Available: https://aclanthology.org/2021.findings-emnlp.277
  * [63] M. Zhou, Q. Li, X. He, Y. Li, Y. Liu, W. Ji, S. Han, Y. Chen, D. Jiang, and D. Zhang, “Table2charts: recommending charts by learning shared table representations,” in _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021, pp. 2389–2399.
  * [64] J. Han, H. Zheng, D. Z. Chen, and C. Wang, “STNet: An end-to-end generative framework for synthesizing spatiotemporal super-resolution volumes,” _IEEE Trans. Vis. Comput. Graph._ , vol. 28, no. 1, pp. 270–280, 2021.
  * [65] A. Masry, D. Long, J. Q. Tan, S. Joty, and E. Hoque, “ChartQA: A benchmark for question answering about charts with visual and logical reasoning,” in _Findings of the Association for Computational Linguistics: ACL 2022_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 2263–2279. [Online]. Available: https://aclanthology.org/2022.findings-acl.177
  * [66] A. Wu, Y. Wang, M. Zhou, X. He, H. Zhang, H. Qu, and D. Zhang, “MultiVision: Designing Analytical Dashboards with Deep Learning Based Recommendation,” _IEEE Trans. Vis. Comput. Graph._ , vol. 28, no. 1, pp. 162–172, 2022.
  * [67] G. Li and X. Yuan, “Gotreescape: Navigate and explore the tree visualization design space,” _IEEE Trans. Vis. Comput. Graph._ , pp. 1–17, 2022.
  * [68] Y. Shi, P. Liu, S. Chen, M. Sun, and N. Cao, “Supporting expressive and faithful pictorial visualization design with visual style transfer,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 29, no. 1, pp. 236–246, 2023.
  * [69] J. Han and C. Wang, “CoordNet: Data generation and visualization generation for time-varying volumes via a coordinate-based neural network,” _IEEE Trans. Vis. Comput. Graph._ , 2022.
  * [70] C. Liu, Y. Guo, and X. Yuan, “Autotitle: An interactive title generator for visualizations,” _IEEE Transactions on Visualization and Computer Graphics_ , pp. 1–12, 2023.
  * [71] Y. Zhang, R. Jiang, L. Xie, Y. Zhao, C. Liu, T. Ding, S. Chen, and X. Yuan, “Oldvisonline: Curating a dataset of historical visualizations,” _IEEE Transactions on Visualization and Computer Graphics_ , pp. 1–11, 2023.
  * [72] E. E. Firat, D. Vytla, N. V. Singh, Z. Jiang, and R. S. Laramee, “Moneyvis: Open bank transaction data for visualization and beyond,” 2023.
  * [73] K. Davila, S. Setlur, D. Doermann, B. U. Kota, and V. Govindaraju, “Chart mining: A survey of methods for automated chart analysis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 11, pp. 3799–3819, 2021.
  * [74] X. Liu, M. S. Alharbi, J. Chen, A. Diehl, D. Rees, E. E. Firat, Q. Wang, and R. S. Laramee, “Visualization resources: A survey,” _Information visualization_ , vol. 22, no. 1, pp. 3–30, 2023.
  * [75] C. Chen and Z. Liu, “The state of the art in creating visualization corpora for automated chart analysis,” _Computer Graphics Forum_ , vol. 42, no. 3, pp. 449–470, 2023.
  * [76] “R datasets,” https://vincentarelbundock.github.io/Rdatasets/datasets.html.
  * [77] J. Leskovec and R. Sosič, “Snap: A general-purpose network analysis and graph-mining library,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 8, no. 1, p. 1, 2016.
  * [78] V. Avsarkisov, S. Hoyas, M. Oberlack, and J. García-Galache, “Turbulent plane Couette flow at moderately high Reynolds number,” _Journal of Fluid Mechanics_ , vol. 751, p. R1, Jul. 2014. [Online]. Available: https://www.cambridge.org/core/product/identifier/S0022112014003231/type/journal_article
  * [79] P. Bartashevich, W. Knors, and S. Mostaghim, “Vector Field Benchmark for Collective Search in Unknown Dynamic Environments,” in _Swarm Intelligence_ , M. Dorigo, M. Birattari, C. Blum, A. L. Christensen, A. Reina, and V. Trianni, Eds. Cham: Springer International Publishing, 2018, vol. 11172, pp. 411–419, series Title: Lecture Notes in Computer Science. [Online]. Available: http://link.springer.com/10.1007/978-3-030-00533-7_36
  * [80] B. Kim, V. C. Azevedo, M. Gross, and B. Solenthaler, “Transport-Based Neural Style Transfer for Smoke Simulations,” _ACM Transactions on Graphics_ , vol. 38, no. 6, pp. 1–11, Dec. 2019, arXiv:1905.07442 [cs]. [Online]. Available: http://arxiv.org/abs/1905.07442
  * [81] A. Moitinho, A. Krone-Martins, H. Savietto, M. Barros, C. Barata, A. J. Falcão, T. Fernandes, J. Alves, A. F. Silva, M. Gomes, J. Bakker, A. G. A. Brown, J. González-Núñez, G. Gracia-Abril, R. Gutiérrez-Sánchez, J. Hernández, S. Jordan, X. Luri, B. Merin, F. Mignard, A. Mora, V. Navarro, W. O’Mullane, T. Sagristà Sellés, J. Salgado, J. C. Segovia, E. Utrilla, F. Arenou, J. H. J. de Bruijne, F. Jansen, M. McCaughrean, K. S. O’Flaherty, M. B. Taylor, and A. Vallenari, “Gaia Data Release 1: The archive visualisation service,” _Astronomy & Astrophysics_, vol. 605, p. A52, Sep. 2017. [Online]. Available: http://www.aanda.org/10.1051/0004-6361/201731059
  * [82] X. Fu, Y. Wang, H. Dong, W. Cui, and H. Zhang, “Visualization assessment: A machine learning approach,” in _Proceedings of the IEEE Visualization Conference_ , 2019, pp. 126–130.
  * [83] H. Haleem, Y. Wang, A. Puri, S. Wadhwa, and H. Qu, “Evaluating the readability of force directed graph layouts: A deep learning approach,” _IEEE Computer Graphics and Applications_ , vol. 39, pp. 40–53, 2018.
  * [84] S. Cai, S.-H. Hong, J. Shen, and T. Liu, “A machine learning approach for predicting human preference for graph layouts,” in _2021 IEEE 14th Pacific Visualization Symposium_ , 2021, pp. 6–10.
  * [85] L. P. Cinelli, M. A. Marins, E. A. B. Da Silva, and S. L. Netto, _Variational methods for machine learning with applications to deep networks_. Springer, 2021.
  * [86] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in _Advances in Neural Information Processing Systems_ , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 28. Curran Associates, Inc., 2015.
  * [87] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _ArXiv_ , vol. abs/1804.02767, 2018.
  * [88] J. Choi, S. Jung, D. G. Park, J. Choo, and N. Elmqvist, “Visualizing for the non-visual: Enabling the visually impaired to use visualization,” in _Computer Graphics Forum_ , vol. 38, no. 3. Wiley Online Library, 2019, pp. 249–260.
  * [89] C. Liu, Y. Han, R. Jiang, and X. Yuan, “Advisor: Automatic visualization answer for natural-language question on tabular data,” in _Proceedings of the 2021 IEEE 14th Pacific Visualization Symposium_. IEEE, 2021, pp. 11–20.
  * [90] A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer, “Vega-Lite: A grammar of interactive graphics,” _IEEE Trans. Vis. Comput. Graph._ , vol. 23, no. 1, pp. 341–350, 2016.
  * [91] J. Shen, H. Li, J. Xu, A. Biswas, and H.-W. Shen, “IDLat: An importance-driven latent generation method for scientific data,” _IEEE Trans. Vis. Comput. Graph._ , vol. 29, no. 1, pp. 679–689, 2022.
  * [92] “Plotly,” https://chart-studio.plotly.com/feed/.
  * [93] G. A. Miller, “Wordnet: a lexical database for english,” _Communications of the ACM_ , vol. 38, no. 11, pp. 39–41, 1995.
  * [94] P.-s. Lee, J. D. West, and B. Howe, “Viziometrix: A platform for analyzing the visual information in big scholarly data,” in _Proceedings of the International Conference Companion on World Wide Web_ , 2016, pp. 413–418.
  * [95] M. J. Cafarella, A. Halevy, D. Z. Wang, E. Wu, and Y. Zhang, “Webtables: exploring the power of tables on the web,” _Proceedings of the VLDB Endowment_ , vol. 1, no. 1, pp. 538–549, 2008.
  * [96] “Tableau public,” https://public.tableau.com/app/discover/trending.
  * [97] D. Moritz, C. Wang, G. L. Nelson, H. Lin, A. M. Smith, B. Howe, and J. Heer, “Formalizing visualization design knowledge as constraints: Actionable and extensible models in draco,” _IEEE Trans. Vis. Comput. Graph._ , vol. 25, no. 1, pp. 438–448, 2018.
  * [98] A. Wu, L. Xie, B. Lee, Y. Wang, W. Cui, and H. Qu, “Learning to automate chart layout configurations using crowdsourced paired comparison,” in _Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–13.
  * [99] S. Fu, K. Xiong, X. Ge, S. Tang, W. Chen, and Y. Wu, “Quda: Natural language queries for visual data analytics,” _arXiv preprint arXiv:2005.03257_ , 2020\.
  * [100] M. Zhou, T. Wang, P. Ji, S. Han, and D. Zhang, “Table2Analysis: Modeling and recommendation of common analysis patterns for multi-dimensional data,” in _AAAI 2020 - 34th AAAI Conference on Artificial Intelligence_ , apr 2020, pp. 320–328. [Online]. Available: https://www.aiide.org/ojs/index.php/AAAI/article/view/5366
  * [101] J. Harper and M. Agrawala, “Deconstructing and restyling d3 visualizations,” in _Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology_ , 2014, pp. 253–262.
  * [102] S. Ray Choudhury and C. L. Giles, “An architecture for information extraction from figures in digital libraries,” in _Proceedings of the 24th International Conference on World Wide Web_ , 2015, pp. 667–672.
  * [103] Y. Zhang, Y. Wang, H. Zhang, B. Zhu, S. Chen, and D. Zhang, “Onelabeler: A flexible system for building data labeling tools,” in _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , 2022, pp. 1–22.
  * [104] B. Vidgen and L. Derczynski, “Directions in abusive language training data, a systematic review: Garbage in, garbage out,” _Plos one_ , vol. 15, no. 12, p. e0243300, 2020.
  * [105] M. Bostock, V. Ogievetsky, and J. Heer, “D3 data-driven documents,” _IEEE transactions on visualization and computer graphics_ , vol. 17, no. 12, pp. 2301–2309, 2011.
  * [106] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _Proceedings of International Conference on Machine Learning_ , 2021, pp. 8748–8763.
  * [107] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [108] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten, “Exploring the limits of weakly supervised pretraining,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 181–196.
  * [109] Y. Zhang, B. Coecke, and M. Chen, “Mi3: Machine-initiated intelligent interaction for interactive classification and data reconstruction,” _ACM Transactions on Interactive Intelligent Systems_ , vol. 11, no. 3-4, pp. 1–34, 2021.