# Lost in Visualization: Investigating the Knowledge Gap between Visualization
Research and Practice

Nam Wook Kim<EMAIL_ADDRESS>Boston College140 Commonwealth AveChestnut
HillMassachusettsUSA02467 ,  Grace Myers<EMAIL_ADDRESS>Boston College140
Commonwealth AveChestnut HillMassachusettsUSA02467 ,  Jinhan Choi
<EMAIL_ADDRESS>Boston College140 Commonwealth AvenueChestnut
HillMAUSA02467 ,  Yoonsuh Cho<EMAIL_ADDRESS>Boston College140
Commonwealth AvenueChestnut HillMAUSA02467 ,  Changhoon Oh
<EMAIL_ADDRESS>Yonsei University50 Yonsei-roSeoulSouth Korea03722
and  Yea-Seul Kim<EMAIL_ADDRESS>University of Wisconsin-Madison500
Lincoln DrMadisonWIUSA53706

(2018)

###### Abstract.

Empirical research on perception and cognition has laid the foundation for
visualization design, often yielding useful design guidelines for
practitioners. However, it remains uncertain how well practitioners stay
informed about the latest findings in visualization research. In this paper,
we employed a mixed-method approach to explore the knowledge gap between
visualization research and real-world design guidelines. We initially
collected existing design guidelines from various sources and empirical
studies from major publishing venues, analyzing their alignment and uncovering
missing links and contradictory knowledge. Subsequently, we conducted surveys
and interviews with practitioners and researchers to gain further insights
into their experiences and attitudes towards design guidelines and empirical
studies, and their views on the knowledge gap between research and practice.
Our findings highlight the similarities and differences in their perspectives
and propose strategies to bridge the divide in visualization design knowledge.

Empirical research, design guideline, knowledge gap, survey, interview,
content analysis

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, 2018,
Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human-
centered computing Empirical studies in visualization††ccs: Human-centered
computing User studies

Figure 1. Overview of research. We conducted a comparative analysis between
guidelines and empirical studies to assess the existing knowledge gap.
Subsequently, we carried out a study to gauge the perceptions and experiences
of visualization stakeholders regarding this gap.

Two boxes- first box titled Phase 1 and second box titled Phase 2, with arrow
pointing between Phase 1 and 2. Phase 1, Content Analysis, answers the first
research question: “What gaps currently exist in guidelines and empirical
studies?” 390 guidelines were collected, and open coding led to a
comprehensive guideline landscape. Alignment mapping was done between these
guidelines and 235 empirical studies. Phase 2, Surveys and Interviews, answers
the second research question: “How do practitioners perceive and feel about
these knowledge gaps?” Practitioners and researchers were administered a
survey and followed up with interviews.

##  1\. Introduction

Data visualization has become an integral part of our lives, as it is widely
used to communicate complex data related to various aspects of our lives. On
the web, people encounter millions of data visualizations, whether they are
related to climate change, election results, or pandemic health risks.
Nowadays, people have a plethora of tools to create data visualizations, such
as Tableau and PowerBI. Data visualization producers include not only
scientists and analysts, but also journalists, managers, and even casual
users. Despite their efforts to address various intricate design factors, such
as perceptual effectiveness, aesthetics, and engagement, we still frequently
come across bad or weird visualizations that result in heated discussions on
social media (Lo et al., 2022).

Empirical research has provided foundational knowledge for data visualization.
Graphical perception studies, beginning with the seminal work by Cleveland and
McGill (Cleveland and McGill, 1984), have provided useful insights into which
visual channels are more perceptually effective. Visual cognition studies have
contributed to a better understanding of the higher-level processing of
visualizations, including comprehension (Bateman et al., 2010), engagement
(Borgo et al., 2012), memorability (Borkin et al., 2015), and biases (Dimara
et al., 2018). Guidelines for visualization design, such as including a zero
baseline for a bar chart and using a bar chart instead of a pie chart for
value comparison, are based on past empirical studies and are now frequently
shared among practitioners. Empirical research continues to produce new design
knowledge and provide additional usage context for existing guidelines.
Nevertheless, the methods by which visualization practitioners stay abreast of
the latest advancements in design knowledge are not well-defined, just as the
processes researchers employ to draw inspiration from practice remain unclear.

In this work, we use a two-phase mixed approach to investigate the knowledge
gap between visualization research and practice, aiming to understand their
interplay better and enhance their collective impacts on the visualization
field. We first collected data visualization design guidelines by conducting a
keyword-based web search, and then conducted a qualitative content analysis to
characterize their attributes. Our findings indicate a concentration of
guidelines on basic charts and task types, while identifying three distinctive
categories of the guidelines: problem-solving, enhancement, and alternative
suggestion. To understand the agreement and discrepancy between guidelines and
empirical research, we analyzed how they are aligned to each other; we used
empirical studies that produce design knowledge via the evaluation of user
performance on diverse visualization designs (Lam et al., 2011). The alignment
analysis confirmed that most of the empirical research is not being
transferred to practice and that conflicting knowledge exists between
guidelines and empirical study results.

To delve into the real-world experiences and viewpoints of researchers and
practitioners pertaining to these knowledge discrepancies, we engaged them in
surveys and interviews. Practitioners were mainly recruited from the Data
Visualization Society (dat, 2023c), while researchers were recruited through
community mailing lists. A total of 33 practitioners and 36 researchers filled
out our survey questionnaire on guidelines, empirical studies, and knowledge
transfer between the two communities. The survey results provided similarities
and differences in how they accessed guidelines, perceived their usefulness,
and identified problems they faced with them, while also indicating a cohesive
opinion on the lack of knowledge transfer between the two communities
(Parsons, 2022). In follow-up interviews with ten selected participants from
each community, we gained deeper insights into their experiences and struggles
with guidelines, as well as their attitudes and perceived causes of the
knowledge gap and ideas for addressing it. We distilled our findings as
considerations for tackling the gulf of design knowledge in the discussion
section.

##  2\. Background

The field of data visualization has always been closely tied to practical
needs and applications. Widely used software tools and programming toolkits,
such as Tableau (tab, 2023; Stolte et al., 2002), D3 (Bostock et al., 2011),
and Vega-Lite (Satyanarayan et al., 2017), began as research projects, as did
visualization techniques such as treemaps (Johnson and Shneiderman, 1998) and
parallel coordinates (Inselberg and Dimsdale, 1990). Researchers have also
devoted much effort to understanding the challenges and struggles faced by
practitioners in deriving new research insights (Joyner et al., 2022; Bigelow
et al., 2014) and building new visualization tools for practitioners (Kim et
al., 2016; Liu et al., 2018). Furthermore, the growing sub-fields of visual
analytics (Cook and Thomas, 2005) and design studies (Sedlmair et al., 2012)
directly address problems faced by domain experts across disciplines.

Similarly, the visualization research community has developed theories and
frameworks over the past few decades that provide established vocabularies for
reading and creating visualizations, such as visual encoding channels, data
types, and analytical tasks (Brehmer and Munzner, 2013; Munzner, 2014). To
evaluate the usefulness of a visualization design, researchers conduct
experiments that compare design alternatives. The results of these experiments
provide guidance for selecting effective visual encoding channels (Cleveland
and McGill, 1984; Heer and Bostock, 2010) and determining their effectiveness
for different tasks (Kim and Heer, 2018; Saket et al., 2018). Some of these
results are integrated into existing tools that automate visual encoding
design (tab, 2023; voy, 2023; Hu et al., 2019). Although these automated tools
may be sufficient for exploratory analysis, designers must still consider a
range of design factors to ensure the successful and engaging communication of
data, such as selecting appropriate color schemes and determining axis ranges.

Researchers have developed pedagogical approaches to teaching data
visualizations, such as the Five Design-Sheet Methodology (Roberts et al.,
2016) and VisItCards (He and Adar, 2017). Other studies have examined data
visualization literacy, which is defined as the ability to read information
encoded in data visualizations and create them to answer questions about the
data (Lee et al., 2016; Börner et al., 2019; Boy et al., 2014). Chevalier et
al. found that although data visualizations are prevalent in K-4 education,
students are rarely taught how to read and construct them (Alper et al.,
2017). Several past studies have investigated ways to assess data
visualization literacy (Boy et al., 2014; Lee et al., 2016; Börner et al.,
2019), but all have focused on teaching and evaluating the basics of data
visualizations, such as higher-level design processes and lower-level
visualization vocabularies.

Several recent studies have attempted to address more practical design needs.
Diehl et al. launched VisGuides, a forum that provides practitioners with a
central place to ask questions about the validity and quality of their
visualizations and discuss guidelines around visualization design (Diehl et
al., 2018). Wang et al. proposed a cheatsheet structure to introduce a
visualization technique, including its construction process, similar
techniques, and potential issues (Wang et al., 2020). Likewise, other
researchers have discussed guideline models for connecting with empirical
studies (Scott-Brown, 2018) and describing their relationships across
different stages in the design process (Meyer et al., 2015).

Similar efforts have also been made by practitioners as well. Resources such
as From Data to Viz and DataVizProject provide user interfaces to select
charts based on different data types and tasks (dat, 2023a, d). Companies
publish design systems that often include recommendations on visualization
design (ado, 2023; goo, 2023), although they mostly focus on styling
guidelines to match their brands.

Currently, visualization design guidelines are dispersed across different
resources, often through personal blogs and published books. It is relatively
unknown how practitioners learn critical design knowledge from these
guidelines, if at all. Moreover, researchers’ efforts to convey research
outcomes through blogs (mul, [n. d.]), podcasts (dat, [n. d.]), or social
media are typically ad-hoc. While two recent studies have begun investigating
the longstanding gap between research and practice, they mostly focus on
understanding holistic design processes rather than acquiring fundamental data
visualization design knowledge (Parsons, 2022; Zhang et al., 2023).

This paper aims to comprehend the potential knowledge gap between
visualization design practice and research. We utilize mixed methods, ranging
from content analysis to surveys and interviews, involving the two stakeholder
groups.

##  3\. Guideline and Empirical Study Alignment

As an initial step toward understanding the knowledge gap, we investigated
what kinds of visualization design guidelines are available and how they are
aligned with the latest research in empirical research in visualization.

###  3.1. Data Collection

####  3.1.1. Collecting guidelines

To ensure comprehensiveness in our collection, we extensively consulted a
variety of resources. We first used web search by trying different keywords
incrementally, such as data visualization, guidelines, and similar synonyms.
We also looked at known resources such as blogs (e.g., DataWrapper (dat,
2023f)) and websites (e.g., from Data to Viz (dat, 2023a), Data Journalism
(dat, 2023e)). Finally, we used the guideline collection curated by the Data
Visualization Society (dat, 2023c), providing additional blogs and websites as
well as books (e.g., How Charts Lie by Cairo (Cairo, 2019)). We went through
the collection and gathered guidelines or statements relevant to data
visualization design, following the definition from Chen et al. (Chen et al.,
2017) as our primary inclusion criterion; i.e., a guideline describes a
process or a set of actions that may lead to a desired outcome or,
alternatively, actions to be avoided to prevent an undesired outcome.

To serve our analysis purpose, we established several exclusion criteria.
First of all, we excluded guidelines that are not specifically relevant to
data visualization design, such as those that provide narrow styling advice,
such as increasing font size, or broad recommendations, such as making
visualizations interactive. Similarly, we excluded simple chart suggestions,
such as using a bar chart for a single series data, that are too evident to be
considered guidelines. We also consolidated overlapping guidelines. After the
filtering and aggregation process, we obtained the final collection of 390
guidelines.

####  3.1.2. Collecting empirical studies

We used a combination of automated and manual searches to collect empirical
papers. First, we initially used a keyword search (e.g., visualization,
evaluation, study) on Scopus to collect papers from ACM CHI and EuroVis over
the period from 2011 to 2020. We collected the same data for IEEE VIS from
VisPub (Isenberg et al., 2017). Expanding on this initial collection, we
supplemented it with recent papers published in ACM CHI, IEEE VIS, and Eurovis
from 2021 to 2022 to ensure the dataset reflects the latest research. We also
manually included relevant past papers that were not captured by our initial
search from additional sources, including VisPerception (vis, [n. d.]) and a
recent survey of perception-based visualization studies (Quadri and Rosen,
2022); this process aided in gathering influential works published prior to
2011, such as the notable graphical perception experiment conducted by
Cleveland & McGill (Cleveland and McGill, 1984). Finally, two researchers
cleaned up the data collection by removing any papers that did not present
original empirical research and by only including studies that evaluate user
performance on different visualization designs based on the characterization
in a prior work (Lam et al., 2011) i.e., we excluded algorithm performance and
system usability studies. The final collection of empirical studies consists
of 235 research papers.

###  3.2. Analyzing Process

##### Design guideline content analysis:

We performed open coding content analysis (Potter and Levine‐Donnerstein,
1999) of the guidelines. A lead researcher initially went through a portion of
the data collection to establish initial codes and code values. Two additional
researchers then independently coded the data using the initial codebook,
expanding it as needed by adding new codes in consultation with the lead
researcher when questions or coding conflicts arose. Finally, the lead
researcher thoroughly reviewed the final codes.

##### Empirical study content analysis:

We only had a basic analysis of the purpose of each study to facilitate the
following alignment process. As there are recent existing studies
investigating the empirical study landscape (Kosara, 2016; Zeng and Battle,
2023; Quadri and Rosen, 2022), we did not codify the empirical studies.
However, it would be worthwhile to investigate finer-grained code-level
alignment in future work beyond our entity-level alignment.

##### Aligning two corpus:

The alignment process followed a similar procedure to the content analysis.
The lead researcher established exemplary mappings, and the remaining mapping
tasks were distributed to five researchers (three are co-authors in this
paper.) The lead researcher then reviewed the final mapping. For instance, for
each guideline entity, we went through the empirical study collection to find
relevant papers by reading their abstracts as well as the body text as needed.
A guideline may have multiple relevant papers and vice versa.

Figure 2. A Sankey diagram showing our final collection of 390 guidelines,
revealing the proportional relationship among six attributes of visualization
design guidelines—context (purpose, problem, approach. chart type, chart
element, data type.

Sankey Diagram oriented horizontally with seven distinct color-coded
categories. Start node is “purpose” and “Task Type” is the end node. Thickest
nodes for each category, representing the most occurrence, include: N/A for
Problem, Use Alternative Chart for Approach, Any for Chart Type, N/A for Chart
Element, Any for Data Type, and N/A for Task Type.

###  3.3. Guideline Landscape

We identified primary high-level codes such as purpose, problem, approach, as
well as additional low-level codes, including chart type, chart elements, data
type, and task type. Figure 2 shows an overview of our coding results.

##### Purpose

Each guideline serves a specific purpose, representing a goal it aims to
achieve. Our analysis identified various key objectives for these guidelines,
including readability (35%), storytelling (15%), accuracy (14%), clarity (9%),
understandability (8%), credibility (8%), insightfulness (5%), accessibility
(4%), engagement (1%), and aesthetics (1%). Readability refers to how easy it
is to read text or visual elements; e.g., when working with large or dense
data sets, avoid dot plots. Understandability concerns more about
comprehension; e.g., when creating a scatterplot, show a trend line. Clarity
refers to how information is clearly expressed; e.g., When detail isn’t
needed, summarize. Accuracy pertains to a more precise perception of data;
e.g., for a more accurate comparison, consider using a bar chart rather than a
radial bar chart. On the other hand, credibility relates to how well the chart
can be trusted; e.g., To prevent a false impression of the data, do not
cherry-pick data.. Other examples for accessibility, aesthetics, and
insightfulness include: for colorblind people, color only the most important
values; to establish unity and cohesion, align all elements of the graph to
create clean vertical and horizontal lines; when drawing a scatter plot, add
marginal distributions to detect the distribution hidden in the overplotted
parts of the graphic.

##### Problem

We observed 12 common problems discussed in our guideline collection. In the
order of frequency, they include cognitive burden (15%), clutter (9%), hidden
patterns (9%), unsupported task (8%), misleading patterns (8%), unsupported
data (7%), distortion (3%), color-blindness (3%), non-zero baseline (3%),
oversimplification (2%), and meaningless colors (1%). Many of these problems
pertain to familiar issues. For instance, there is the problem of avoiding
inverted axes, which can lead to distortion. Another problem involves avoiding
choropleth maps when trying to identify subtle differences between regions, as
this may result in an unsupported task. Additionally, when dealing with hidden
or overlapping data points, it’s advisable to use transparency to reveal
hidden patterns. Some problems are more generic, while others are more
specific. For instance, cognitive burden is a group of generic issues
concerning inconsistency, illegibility, and ambiguity that might cause
eyestrain and attention fatigue; e.g., to avoid back and forth eye movement,
eliminate legends and try to position data labels near the data.. On the other
hand, meaningless colors focuses on a narrow issue, such as when creating a
bar chart, avoid giving each bar a categorical color. Not all guidelines
involved problems (33%), as some of them only suggest improvements (Figure 3).

##### Approach

We noticed the utilization of 29 distinct approaches aimed at tackling design
issues or improving the situation, highlighting considerable diversity in the
methods employed. The list of top 10 approaches includes: use alternative
chart (24%); select proper color scheme (14%); add different encoding (8%);
refrain from embellishment (6%); annotate details (6%); highlight key items
(4%); use right scale (4%); adjust layout (4%); arrange items in order (4%);
and avoid overplotting (3%). The majority of these concerns centered on
selecting effective visual encodings, accomplished by choosing alternative
chart types; for example, to bring two line charts together, use a connected
scatterplot instead of a dual-axis chart. Another approach involved utilizing
diverse mark types and encoding channels; for instance, when red and green
must be used together, incorporate symbols as a secondary method of
communication. Color was also a prominent issue, with 98 out of 390 guidelines
pertaining to color. The category select proper color scheme encompassed
guidelines addressing rainbow color palettes, color blindness, highlighting,
low-contrast issues, and inconsistent color semantics, such as using darker
colors for lower values.

##### Types of Charts, Data, Tasks

A large portion of the guidelines (40%) were not specific to particular chart
types; however, when specificity was provided, it centered largely on standard
charts like bar charts (11%), line charts (9%), pie charts (6%), scatter plots
(4%), and choropleth maps (2%).

Many of the other guidelines were related to variations of these standard
chart types, such as stacked bar charts (4%), area charts (4%), and bubble
charts (1%). Oftentimes, guidelines were focused on specific chart elements
(42%). Mark was the most frequently discussed topic (15%), followed by axis
(8%), annotation (7%), and axis labels (3%). As an example, the guidelines for
mark encompass suggestions such as keeping the distance between the bars at
half the width of the bars to enhance data comprehension and if a bar chart
contains numerous bars causing clutter, consider using a dot plot as circles
consume less ink than bars. An illustrative guideline under axis recommends
that when creating time-series plots with unevenly collected data, ensure that
the tick marks on the time axis reflect the irregular intervals to prevent
distortion of the information.

We endeavored to code data types using the well-established framework in the
research community, which includes categorical, numerical, temporal, and
geospatial data types. However, we observed that design guidelines rarely
specify such data types (Figure 2), often resulting in unreliable codes. Such
data types could often be inferred from the chart types, however. We used Amar
et al.(Amar et al., 2005)’s taxonomy of low-level analytic tasks to code task
types. A significant portion of the 150 guidelines (38%), where task types
were available, centered around the basic tasks of compare (18%) and retrieve
value (7%); compare is a combination of retrieve value and compute derived
value. Other task types observed include correlate (5%), characterize
distribution (3%), and sort (2%).

###  3.4. Guideline Templates

Figure 3. The structure template of guidelines and examples across the three
different characteristics: enhancement, problem-solving, and alternative
suggestion. The template consists of a combination of attributes and
components of a sentence. Each component of the sentence is color-tagged
according to each attribute it includes.

Template of guidelines with each necessary attribute color coded for
distinction. “When/if Contexts” highlighted purple, “Approach*)” green,
“Alternative” red, “because of Problem” yellow, and “For Purpose blue. Below
the template are examples categorized by characteristics and color coded
corresponding to the template attributes.

While inspecting guideline sources during the coding process, we observed
recurring patterns in the guideline structures. We categorize them into three:
problem-solving, enhancement, alternative suggestion. We synthesized these
patterns to derive a holistic guideline template (Figure 3): What/if
{Context}s, {Approach* + (Alternative)}, because of {Problem} + for {Purpose}.

Problem-solving guidelines include the “problem” attribute as a necessity, as
they suggest to solve the problematic situation; e.g., distortion, misleading
patterns, and clutter. Most guidelines belong to this category (67%), with
common purposes including readability, accuracy, and clarity; for example, to
accurately depict trends, ensure that the aspect ratio for trend charts is set
to 45 degrees..

Alternative suggestion guidelines are similar to problem-solving guidelines,
but they deserve separate consideration given their prevalence in practice.
Most of the guidelines that have user alternative chart (24%) and add
different encoding (8%) approaches belong to this category; for instance, when
you need more than six categorical colors, try alternative visual encoding
such as position.

Enhancement guidelines do not necessarily have problematic issues to be
resolved. The primary objectives of these guidelines include storytelling,
engagement, and aesthetics, contributing to enhanced visualization quality and
improved user experience. Example guidelines include: when formulating titles,
use concise and active phrases that encapsulate the intended argument instead
of descriptive titles; and use graphics like icons only when they support
interpreting the graph.

The existing guideline description lacked a cohesive structure; however, these
templates provide a well-defined framework for effectively conveying
actionable guidance to designers. We used these templates to generate concise
one-sentence descriptions for each guideline, further streamlining our open
coding process.

###  3.5. Guideline and Empirical Study Alignment

Figure 4. Overview of guidelines mapped and not mapped to empirical studies.

Four different bar graphs: Alignment Overview, Guideline Purpose, Guideline
Approach, and Guideline Problem. In the Alignment Overview graph, four bars
are depicted: 281 guidelines not mapped, 109 mapped. Within this context, 49%
are supportive, 32% are mixed, and 21% are opposed. The Guideline Purpose,
Approach, and Problem graphs has ‘Mapped’ and ‘Not Mapped’ as the x axis, with
various attributes on the y-axis, sorted in decreasing frequency. In the
Purpose graph, ‘Credibility’ is highlighted pink. ‘Highlight key items, use
faceting/small multiples, avoid intentional omission, and include zero
baseline’ highlighted pink for Approach Graph. ‘Misleading patterns’ and ‘Non-
zero baseline’ highlighted pink for Problem Graph.

Among the 390 collected guidelines, 109 (28%) were mapped to empirical
studies. Among the 235 empirical studies, only 94 were linked to the
guidelines. In other words, each guideline was often mapped to more than one
empirical study ($avg$ / guideline = 2.00, $std$=1.62), while an empirical
study was also mapped to more than one guideline ($avg$ / empirical study =
2.15, $std$=1.72). Many of the highly mapped guidelines and empirical studies
were about well-known graphical perception issues such as regarding pie
charts, embellishments, color maps, and scales & axes.

Among the total 219 guideline and empirical study mappings, the majority (49%)
were found to be supported by corresponding empirical studies. However, we
also observed mixed support (32%) or even conflicting results (21%) from
empirical studies. For example, a guideline suggests avoiding the use of 3D
charts because they can skew perception. However, a study found that 3D
heatmaps were superior to 2D heatmaps for reading and comparing single data
items, but 2D heatmaps were better for overview tasks (Kraus et al., 2020).
Similarly, some guidelines recommend minimalist chart designs without
embellishments to avoid distracting from the data, but studies suggest that
visual embellishments such as icons can improve long-term recall and
interpretation accuracy (Haroz et al., 2015). Similarly, other guidelines
recommend avoiding rainbow colormaps, while several empirical studies
contradict this guideline by demonstrating the value of multi-hue maps on
continuous data for class discriminability (Liu and Heer, 2018).

We also looked at whether any different trends exist in the mapped and
unmapped guidelines. The distribution of unmapped guidelines mostly remained
similar to the overall distribution mentioned in the guideline landscape
analysis (Sec. 3.4). However, there were several meaningful changes in the
topic distribution of the mapped guidelines (Figure 4), indicating that
empirical studies tend to prioritize verifiable guidelines over qualitative
ones. For instance, we observed a notable drop in the frequency of the clarity
dimension within the guideline purpose category. Likewise, the frequency of
the annotate details and adjust layout dimensions were much lower in the
mapped guidelines, while some other approaches were not mapped at all,
including adjust label/title/annotation position, improve axis label format,
and choose the right orientation. Similarly, misleading patterns and non-zero
baseline were much more frequent in the mapped guidelines. For chart types and
chart element types, bar, line, and pie charts consistently remained at the
top, while scatter plots and axis elements (compared to annotation elements)
more prominent in the mapped guidelines. We did not observe noticeable trends
in task types.

| Mapped empirical study topics |  | Unmapped empirical study topics
---|---|---|---
1 | point, bar, depict, rank, simple | 1 | color, bias, background, work, interaction
2 | different, popout, range, conduct, effect | 2 | uncertainty, link, flow, different, text
3 | ratio, trend, measure, similarity, aspect | 3 | different, encode, mark, value, position
4 | effect, caption, convey, form, color | 4 | accuracy, time, compare, different, overlay
5 | find, correlation, model, encode, embellishments | 5 | title, encode, value, pair, compare
6 | line, accuracy, people, alternative, qualitative | 6 | time, different, evaluate, plot, series
7 | color, different, encode, perception, map | 7 | graph, cluster, present, base, model
8 | forecast, structure, size, trust, scatterplots | 8 | present, interactive, effect, people, type
9 | wrap, compare, model, scale, value | 9 | color, edge, base, perception, present
10 | color, size, concepts, mark, time | 10 | size, network, base, large, node

We also analyzed the characteristics of the unmapped empirical studies, as
they could potentially indicate knowledge that has not yet been effectively
transferred into practice. We conducted topic modeling using Latent Dirichlet
Allocation (LDA) with the number of topics set to 10 to analyze paper
abstracts (subsection 3.5). Although the topics were largely similar to each
other (focusing on perception and cognitive issues), the unmapped empirical
studies revealed distinct topic keywords. These keywords appear to encompass
more recent and underexplored subjects, including network, graph layout,
cluster, interaction, uncertainty, and bias. This aligns with the observation
that the unmapped empirical studies were skewed towards more recent
publications.

####  3.5.1. Takeaways

The analysis of guidelines and their alignment with empirical studies has
yielded valuable insights into the design knowledge currently available and
its empirical backing. The majority of guidelines were focused on basic chart
types and tasks, indicating a noticeable lack of support for other commonly
used advanced visualizations, such as parallel coordinates and network
diagrams. The existence of mixed and conflicting knowledge was also revealed
by empirical findings that refuted existing guidelines. Finally, the large
number of unmapped empirical studies suggest a need for further knowledge
transfer to practice and more inspirations from practices for further research
investigation.

##  4\. Practitioners and Researchers’ Attitudes and Experiences

The content analysis allowed us to glimpse the current knowledge gap in
guidelines and empirical research. To further contextualize the observations,
we conducted follow-up surveys and interviews that provided insights into the
perceptions and lived experiences of practitioners and researchers regarding
the potential knowledge gap.

###  4.1. Survey Recruitment & Procedure

Our recruitment methodology varied slightly between practitioners and
researchers. In the case of practitioners, we initiated recruitment efforts by
sending direct messages to members on the Data Visualization Society’s Slack
workspace. Specifically, we targeted individuals who had recently demonstrated
activity within the workspace, such as posting messages or comments in public
channels. Additionally, we endeavored to expand our reach through postings on
the Storytelling with Data forum (sto, 2023) and direct messages to
visualization practitioners on Twitter, although these efforts were not as
successful as the initial approach.

For the recruitment of researchers, we pursued an alternative approach,
reaching out to key stakeholders via email. Specifically, we contacted
individuals associated with the<EMAIL_ADDRESS>and<EMAIL_ADDRESS>tuebingen.de mailing lists, as well as researchers with expertise in empirical
studies in visualization. Our targeted population included esteemed faculty
members, post-doctoral researchers, and Ph.D. students in candidacy, all of
whom were deemed suitable candidates for our research endeavor.

We distributed both surveys online and participants took approximately 6-10
minutes to complete them. The surveys included similar questions, grouped by
background, awareness, and opinion about visualization design guidelines, and
perception and expectation about visualization empirical research (see example
questions in Figure 5 and Figure 6). The full survey questionnaires are in
Supplementary Material. To compensate participants, we offered a $25 raffle
reward for every 25 participants who completed the survey.

###  4.2. Survey Participants

Overall, 33 practitioners and 36 researchers filled out the surveys.
Practitioners indicated their jobs as the following: eight analysts (24%), six
designers (18%), six scientists (18%), three managers (9%), three developers
(9%), two teachers (6%), one journalist (3%), and four others (12%). The
others category includes one project lead, one consultant, and two engineers.
Sixteen of them (48%) had 3-5 years of experience, eight had 11-15 years
(24%), six had 6-10 years (6%), two had 0-2 years (6%), and one had more than
16 years of experience (3%). In terms of research areas, 20 participants
indicated applications (24%), followed by 19 indicating empirical research
(23%), 16 indicating representation & interaction (19%), 10 indicating
theoretical work (12%), eight indicating analytics & decisions (10%), seven
indicating systems & rendering (8%), and three indicating data transformation
(4%).

###  4.3. Survey Results

Figure 5. Survey responses regarding design guidelines

Four survey questions regarding design guidelines are mapped onto a horizontal
dual bar graph with Practitioners represented in blue, Researchers represented
in orange, and various survey answer options as the y axis. For the first
three questions, the x-axis represents the percentage of responses, while the
fourth question measures intensity, color coded from not at all (red) to
extremely (blue).

Figure 6. Survey responses regarding empirical studies

Two survey responses regarding empirical studies are mapped onto a horizontal
dual bar graph with Practitioners represented in blue, Researchers represented
in orange, and various survey answer options as the y axis. For the first
question, the x-axis represents the percentage of responses, while the second
question measures intensity, color coded from not at all (red) to extremely
(blue).

Figure 5 and 6 provide an overview of survey responses. Generally,
practitioners and researchers showed similar perceptions and opinions toward
design guidelines and empirical studies, but differences were observed in
their backgrounds and experiences. For example, practitioners reported that
they learned about guidelines mainly from books (25%) and websites (23%), such
as online articles and blogs. In contrast, researchers relied more on research
papers (20%) and colleagues (20%) to learn about guidelines, with a relatively
higher percentage compared to practitioners (13% for each category). Regarding
empirical studies, practitioners indicated that their primary access channels
were online resources, such as websites, articles, and blogs, accounting for
23%. In contrast, researchers indicated conferences and research paper
archives the most, representing 30% of their preferred access channels.

Both stakeholders agreed that guidelines could be beneficial for creating
better visualizations (Practitioners: 27%, Researchers: 28%), improving
confidence in design decisions (P: 20%, R: 23%), and facilitating better
communication with stakeholders (P: 20%, R: 22%)). However, researchers
weighed the usefulness of guidelines on increasing visualization design
knowledge less (15%) than practitioners (24%). In the Other comment section,
several practitioners mentioned additional benefits of guidelines: serving as
a “standard of practice”; helping avoid “misleading, ambiguous”, and
“deceitful” visualization designs; increasing “user understanding” and
“trust”, producing “consistent user experience”; “facilitating design decision
making”. Researchers’ comments were similar, while they additionally mentioned
that guidelines provide a “scaffold” or “starting point” for learners.

When asked about the problems they face with existing guidelines, both
stakeholders identified inconsistent and conflicting guidelines as the most
frequent problem (P: 26%, R: 23%). Practitioners weighted more on the lack of
design resources (P: 19%, R: 12%), while researchers emphasized more on the
trustfulness of guidelines (P: 15%, R: 19%). In the Other comment section,
practitioners provided additional comments:“[a guideline] limits the ability
to take into account semantics and intent”; “[it is] difficult to explain to
stakeholders why they should follow those guidelines”; “[it’s] best to use own
judgment. When it comes to bad design, it’s usually bad intentions getting in
the way”. The comments of the researchers echoed similar sentiments: “better
designs depend critically on context, and guidelines are context-free”;
“sometimes [they’re] overgeneralized, leading to unwarranted confidence in
design decisions”; “[it is] not always easy to find authoritative sources for
guidelines, which makes it challenging to convince stakeholders/clients”.

Both practitioners and researchers indicated that they are generally familiar
with design guidelines (Figure 5 rightmost). As expected, practitioners found
design guidelines more relevant to their practice. Although both groups
generally understood the concept of misleading visualization designs,
practitioners placed a greater emphasis on the usefulness of guidelines for
addressing such issues (Figure 5 rightmost), compared to researchers who did
not prioritize guidelines to the same extent as practitioners. When we asked
researchers how they communicate their research to the public, if any, most
said: conference presentations (41%), followed by social media (26%) and
online resources such as blogs (20%).

When we asked about the availability of empirical studies, both practitioners
and researchers agreed that empirical studies are generally not very
accessible to practitioners. Despite differences in familiarity with empirical
studies, there is a shared recognition of the significant role empirical
studies play in producing design guidelines. Similarly, they agreed that the
current transfer of knowledge from empirical research findings into practical
visualization guidelines might not be effective.

When we generally asked for additional comments on guidelines, several
practitioners commented that it would be useful to “update examples” of
guidelines, while it is hard to find which guidelines are “relevant” and
“trustworthy”. On the other hand, researchers suggested “including information
about their limitations,” avoiding the tendency to draw “a false dichotomy
between good and bad,” and taking into account “practical differences” when
making recommendations between one approach and another.

Practitioners did not provide extensive comments on empirical studies,
although one participant emphasized the importance of authors “translating
empirical work into practical applications” for practitioners who may not have
the time or inclination to read academic papers. On the other hand,
researchers indicated the tendency to “prioritize efficiency” excessively,
limited “meaningful outreach with practitioners”, and the “overlooking of
practical contexts” in empirical studies.

###  4.4. Interview Procedure

Two researchers conducted interviews remotely. The interview questions were
prepared in advance and tailored to each interviewee’s survey response to
understand their experiences and perspectives in depth. For instance, if
participants selected a particular category of challenges, we probed further
by asking them to elaborate on their perceptions and ideas. The questions were
structured to follow the same format as the survey questionnaire and were kept
consistent across all practitioners and researchers involved in the
interviews. One researcher facilitated the questioning process, while the
other researcher took notes and transcribed the interviews.

###  4.5. Interview Participants

Practitioners Interviewees | Researcher Interviewees
---|---
ID | Job Title | Experience | Guideline Familiarity | ID | Job Title | Experience | Guideline Familiarity
P1 | Manager | 6-10 yrs | Extremely familiar | R1 | Postdoc researcher | 11-15 yrs | Very familiar
P2 | Product Lead | 3-5 yrs | Very familiar | R2 | Graduate assistant | 0-2 yrs | Moderately familiar
P3 | Scientist | 6-10 yrs | Very familiar | R3 | Graduate assistant | 6-10 yrs | Extremely familiar
P4 | Consultant | 11-15 yrs | Extremely familiar | R4 | Postdoc researcher | 6-10 yrs | Very familiar
P5 | Scientist | 3-5 yrs | Very familiar | R5 | Research scientist | 3-5 yrs | Extremely familiar
P6 | Developer | 6-10 yrs | Extremely familiar | R6 | Graduate assistant | 3-5 yrs | Moderately familiar
P7 | Manager | 11-15 yrs | Extremely familiar | R7 | Faculty | 11-15 yrs | Very familiar
P8 | Analyst | 3-5 yrs | Very familiar | R8 | Faculty | 16 or more yrs | Slightly familiar
P9 | Designer | 3-5 yrs | Extremely familiar | R9 | Faculty | 16 or more yrs | Very familiar
P10 | Analyst | 11-15 yrs | Extremely familiar | R10 | Research scientist | 16 or more yrs | Very familiar
Table 1. Demographic characteristics of practitioner and researcher
interviewees

After following up with survey participants to inquire about their
availability for interviews, 21 out of 33 practitioners agreed to participate.
Thus, we selected ten participants based on differing opinions on how well
knowledge of empirical research transferred to visualization practice (one
thought it transferred extremely well, two thought it transferred very well,
two thought it transferred moderately well, three thought it transferred
slightly well, and two thought it did not transfer well at all). Upon choosing
the participants, we also considered participants who are more acquainted with
design guidelines (six were extremely familiar, and four were very familiar),
as they can provide more substantial perspectives on the knowledge gap
associated with the guidelines.

Out of the 36 researchers, 16 expressed interest in participating in the
interviews. We carefully chose ten interviewees who have familiarity with
conducting empirical experiments (three participants were extremely familiar,
five were very familiar, and two were slightly familiar) as their insights can
help uncover nuances in the challenges and opportunities associated with the
current knowledge transfer effort. The selected participants exhibited varying
levels of familiarity with visualization design guidelines (two were extremely
familiar, five were very familiar, two were moderately familiar, and one was
slightly familiar), which allowed us to capture a broad spectrum of viewpoints
on how design guidelines are perceived and integrated into empirical research
practices.

###  4.6. Interview Analysis

All interviews were conducted within a 30 to 45-minute timeframe, and each was
transcribed using an automatic transcription service. The transcripts were
then manually checked for errors. During the initial coding process, a
research assistant pulled related quotes from the transcripts and coded them
in a spreadsheet. The lead researcher reviewed and refined the codes. They
repeated this process for approximately one-third of the interviews until
arriving at a consistent set of codes. The research assistant then applied the
final codes to the remaining interviews, which were comprehensively reviewed
by the lead researcher. Overall, we categorized the quotes into eleven themes:
five focused on experience and perception, challenges related to guidelines,
two themes concerning perception and accessibility of empirical studies, four
themes addressing the perception of the knowledge gap, strategies to mitigate
this gap, and expectations within the visualization communities.

###  4.7. Interview Results: Guidelines

####  4.7.1. Personal Definitions of Visualization Guidelines

Practitioners viewed visualization design guidelines as a set of best
practices, principles, or rules that guide the creation of effective and
consistent visualizations. They noted that guidelines help organizations
improve the way they communicate data and to be able to make better decisions,
covering things like axis labeling and composition strategies (P1, P2, P6,
P9). However, they also highlighted that there aren’t any absolute rules in
data visualization and that it is important to “tailor what you do and tailor
what you make to your audience and to your data” (P6). Researchers generally
agreed that guidelines are a set of basic rules of good or bad practice.

Although practitioners frequently viewed guidelines as more prescriptive
(e.g., maintaining consistency, brand identity), researchers viewed them as
more flexible heuristics that can adapt to different situations and audiences.
For instance, R2 shared “it’s just the designer’s intuition on what looks
right and what doesn’t.”, while R3 commented, “each organization or designer
[…] usually come[s] up with their sets of soft guidelines versus the hard
ones.” They also emphasized that guidelines are not meant to be
“authoritative” (R6) and current guidelines are often “written more
prescriptive than they probably should be” (R10).

####  4.7.2. Lived Experiences with Visualization Guidelines

Practitioners had mixed views on the applicability and frequency of using
visualization design guidelines in their practice. They mentioned that they do
not strictly adhere to the guidelines but rather underscored the significance
of taking into account user needs and feedback (P1, P4, P5, P7). They also
pointed out that experience, intuition, and learning from others played
crucial roles in shaping their decision-making process (P5, P6, P7, P9, P10).
For instance, as their experience grows, they rely less on guidelines (P6) and
instead trust their instincts and inspirations (P5, P9). Similarly, P10 noted
that observing real-world applications in projects helps them gauge the
utility of a guideline. Despite this somewhat critical perspective on
guidelines, other practitioners emphasized that guidelines provide fundamental
knowledge and foster critical design thinking (P5, P9).

The responses from researchers were generally consistent with those of
practitioners, reinforcing the notion that there is no universal solution for
visualization design guidelines. They also underscored the significance of
considering the project’s context, objectives, audience, and collaborators.
For instance, R3 noted that guidelines are “flexible tools that can
accommodate the needs and viewpoints of various stakeholders involved in
visualization design.” On the other hand, R10 emphasized that guidelines are
valuable to navigate the fluidity and vagueness inherent in data visualization
design, particularly for beginners, even though “they are not so clear cut and
should be taken with a grain of salt.”

####  4.7.3. How They Encounter Visualization Guidelines

When inquiring about their sources for accessing data visualization
guidelines, practitioners mentioned a blend of formal and informal channels,
including books, websites, podcasts, and professional networks. For instance,
they referred to resources offered by notable companies and organizations like
Google, Apple, IBM, and HDX (P1, P2). On the contrary, P4 discussed a
collection of books authored by Tufte, Few, and Knaflic that predominantly
focus on charts, contrasting them with the book by Tamara Munzner, which
offers more theoretical models and knowledge. Others cited specific resources,
such as the Data Stories podcast (P3), Medium blogs (P7), Twitter posts (P6),
and additional resources (like Nightingale Magazine) available through the
Data Visualization Society (P2, P6, P7, P10). In general, they do not actively
seek out data visualization design guidelines. Instead, they tend to encounter
guidelines organically, such as when researching aspects of their projects
(P9). However, they do deliberately select particular books to read,
especially those focused on topics like dashboard design (P10).

In general, researchers encountered visualizations in a similar fashion (e.g.,
books, blogs, social media, etc.), albeit with a more frequent mention of
academic sources. They specifically noted encountering guidelines through
papers presented at conferences like ACM CHI and IEEE VIS (R1, R6, P7 R8).
Similarly, R2 described “absorbing” guidelines “passively through reading
papers,” while others referred to course materials (R1, R4, R10) or academic
workshops (R9). Interestingly, R7 highlighted that effective visualizations
crafted by creative professionals or journalists often act as valuable design
guidelines.

####  4.7.4. Perceived Usefulness of Visualization Guidelines

Practitioners had generally positive opinions regarding the usefulness of
visualization design guidelines. They expressed that they found them helpful
because they provided a framework and direction for visualizing data,
simplified their work, and made their communication more effective (P3, P5,
P7, P9, P10). For instance, P9 said, guidelines “help me communicate what I
want to do to other people […] also help people to understand what I’m
saying.”, and further elaborated that “it’s like basic knowledge you have to
have […] to apply for your job.” However, P1 indicated less enthusiasm toward
guidelines due to their lack of applicability and thoroughness.

Researchers shared positive perceptions akin to practitioners. R1 pointed out
that guidelines can help “convince coworkers” and themselves toward better
designs. Others indicated that they use guidelines “mainly to teach” students
(R6) and to help with the “exploration of the design space” of what works and
what does not (R7). In addition, R2 noted that “they are useful in the sense
that if they are integrated into some kind of system, so like Tableau, for
instance, automatically select[ing] correct color scales.” Nevertheless,
practitioners also highlighted that guidelines are less effective without
contextual examples (R7) and often lack sufficient actionability unless
accompanied by clear alternative suggestions, even if the conceptual
foundations behind the guidelines hold value (R9).

####  4.7.5. Struggles and Challenges with Visualization Guidelines

Practitioners observed that the existing guidelines have limitations and do
not adequately cover emerging concerns like interactivity and narrative
elements (P1, P4). They also emphasized that the present guidelines seem
overly constrictive, lacking an embrace of subjectivity and the contextual
nuances of diverse projects (P4, P6, P7, P9). For instance, P6 said, “with
climate change […] small differences have a really big effect […] always
including like zero degrees in the y-axis just doesn’t make any sense.”
Similarly, a number of practitioners highlighted encountering trade-offs with
business requirements (P3, P8, P9). For example, as noted by P3, “if [..] they
already plotted their data […] a certain way […], they might not want to
change it.” Likewise, they indicated that effectively disseminating the
guidelines to accommodate a variety of stakeholders (designers, analysts,
managers, etc.) presents a distinct challenge (P2, P3, P4, P10). What
exacerbates the situation is the lack of consensus in the field regarding what
constitutes good design, resulting in conflicting and scattered guidelines
(P2, P3, P4, P7, P10). Hence, they also emphasized the necessity for improved
examples of how to apply the guidelines and when to deviate based on specific
situations, rather than solely relying on negative instances that advise
against certain practices (P5, P7, P8, P9, P10).

Researchers were mainly concerned about challenges in making guidelines more
accessible and useful for practitioners. One common theme was the difficulty
of aligning the guidelines to specific user requirements and design goals (R1,
R2, R3, R8, R10), as R2 noted, “I couldn’t really type in like I have this and
what should I use?” They also underscored the overwhelming number of scattered
guidelines, along with the challenge of staying abreast of new research (R2,
R8, R9). For instance, R10 shared “ there are so many tasks, so many types of
data and so many variations to think…”, while R9 pointed out the
“impossibility of actually making sense of all the studies out there.” As with
practitioners, many researchers expressed frustration with the conservative
nature of existing guidelines and their limited versatility, suggesting the
need to acknowledge contextual nuances and offer practical flexibility (R4,
R7, R8, R9, R10). In a similar vein, they highlighted the necessity for
clearer communication of provenance and limitations (R7), addressing
conflicting arguments (R6), and establishing a shared vocabulary across
disciplines to facilitate the interpretation of guidelines (R5). Moreover, the
overarching lack of consensus on design standards and the variation in
outcomes from empirical studies seem to further complicate the establishment
of universally applicable guidelines (R9, R10).

###  4.8. Interview Results: Empirical Studies

####  4.8.1. Perceived Usefulness of Empirical Studies

All practitioners generally agree that empirical studies provide evidence and
validation for design choices. Several practitioners stressed the value of
helping to better understand the reasoning behind graphical selections (P2,
P4, P5, P6, P7, P10). For instance, P10 commented, “it’s really helpful for me
to see the research that says, if you’re in this situation you can do this or
like in this situation […] this is how the participants interpreted this
graph.” P6 appreciated anecdotal evidence that had been rigorously validated,
while P5 highlighted that it saves them time by sparing them from the need for
trial and error. Others said empirical research helps to “convince those
naysayers and the extreme skeptics among us” (P6), by “having proof” (P5). On
the other hand, P1 emphasized the importance of qualitative research too,
saying “doing research like how people feel when they look at visualization is
important as well.”

Although researchers held similar opinions, some expressed skepticism about
the practicality of using empirical results in design practice. For instance,
R1 noted, “design guidelines were from refining the results of empirical
studies, so I think there are some differences between the two.” Likewise,
others highlighted the need for qualitative research to gain a holistic
understanding of user preferences and behaviors (R2, R10), as R2 said “if you
just use […] empirical quantitative stuff it might not always capture that
enjoyment and stuff aspect of visualization.” Several interviewees stressed
the importance of replicating studies and verifying the statistical
significance of claims. For instance, R6 said, “I’m not sure if I ran in one
place and I ran with a different group of people, we’ll get the same results”,
while R9 commented, “there are different levels of evidence behind
guidelines.” In addition, R7 emphasized empirical research should be a
concerted effort involving multiple parties, including “researchers,
designers, and tool developers.”

####  4.8.2. Accessibility of Empirical Study Findings

Participants acknowledged challenges in accessing empirical research, citing
factors such as paywalls (P2, P3, P6), complex language (P3, P4), and time
constraints (P6). For instance, P4 noted “the language of academia is not
fully accessible to practitioners.” P6 similarly added, “The amount of time to
find, read and understand is a big barrier for entry.’ They utilize a variety
of sources to access empirical studies, academic websites, journals, blogs,
and even networking platforms like Twitter (P2, P4, P5, P8, P10). Several
practitioners expressed a preference for summarized findings and design
guidelines (P6, P7, P10). For instance, P7 noted, “I would want to know what
the results are and how it impacts my work.”, while P6 said, “I think for most
people … guidelines are good enough.”

Researchers expressed difficulty in accessing empirical studies for
practitioners, mainly due to complex jargon and limited availability of easy-
to-understand resources (R1, R5, R6, R8, R10). They also acknowledged that
paywalls can hinder practitioners’ access to valuable research (R6, R10), as
well as the lack of necessary background knowledge (R8). Some participants
suggested that providing access to well-organized and summarized research
could improve accessibility (R1, R10). On the flip side, some said
practitioners might not need to check studies themselves (R2, R5), while
others pointed out a need to understand the reasons behind guidelines (R6, R8,
R9).

###  4.9. Interview Results: Knowledge Gap

####  4.9.1. Perception and Attitude toward Knowledge Gap

Practitioners expressed differing opinions on the existence of a gap between
research findings and practical knowledge. Some noted a significant gap, where
information from conferences takes time to reach public books (P3) or
mentioned challenges in translating research into practical applications (P4).
Others felt there was no gap due to the availability of information if one
knows where to look (P8). On the other hand, some others were unsure about the
extent of the gap since not all practitioners might engage with research
directly, possibly contributing to the perceived gap (P9, P10).

Researchers also indicated varying perceptions of the knowledge gap. Some of
them emphasized the challenges in accessing and understanding research studies
for practitioners (R1, R4, R5, R6); for instance, R5 attributed the issue to
the interdisciplinary nature of the field, while R6 pointed out the
contradicting knowledge spread across the field. Others highlighted that
individuals who publish guidelines in blog posts or company-driven guidelines
might lack the necessary background to effectively convey knowledge. For
instance, as mentioned by participant R7, “it pays to have a blog […] because
it increases your exposure on the web.” On the other hand, R2 believed that
the gap is not significant, saying “Use like a scatter plot if you have two
variables. Bar charts work great for 90 percent of things. I don’t know how
many more design guidelines we need.”

####  4.9.2. Expectations for Visualization Communities

The practitioners expressed a need for better communication and collaboration
between researchers and practitioners. For instance, P4 highlighted the
necessity of terminology adaptation between practitioners and researchers.
Meanwhile, P1 emphasized the importance of practitioners being involved in
research to make the outcome more actionable. They also emphasized the
importance of making research more accessible and applicable to practitioners,
including real-world applications of the studies (P7), broader access to
facilitate the implementation of findings (P8), and creating guides and
summaries (P5, P10). Some expressed a degree of cynicism about the possibility
of successful collaboration (P6), while others remained hopeful for more open
data and knowledge sharing (P9).

The researchers equally emphasized the importance of effective communication
and collaboration between the two groups (R3, R5, R6). R6 specifically
mentioned that researchers should take extra steps to reach practitioners,
such as creating casual articles, videos, and podcasts to make their work more
accessible to practitioners. On the other hand, they said practitioners also
have a responsibility to be critical of their sources and seek credible
information, as R6 noted “Like you shouldn’t believe every medical advice
that’s out there on the internet.” R10 emphasized that “there probably should
be more cross transfer between more practitioner-oriented conferences and more
research-oriented conferences.” Similarly, R7 conveyed a wish for the ongoing
publication of guidelines, especially by practitioners, even in the face of
potential adverse consequences, envisioning a growing space where both
scientific and practitioner-oriented research coexist.

####  4.9.3. Perceived Causes for the Gap

Not a lot of practitioners articulate specific causes for the knowledge gap,
except P4 who discussed the cultural and educational differences between
practitioners and researchers, which can create tension and challenges in
communication. On the other hand, researchers pointed out the challenges in
applying research findings universally (R1, R9); e.g., R1 noted, “the research
was done in specific data, specific tasks, and in a particular environment,
[so] it isn’t easy to apply it universally.” Some mentioned the limited
interaction between practitioners and researchers poses a barrier to knowledge
dissemination (R2, R3), while others highlighted the absence of research that
caters to practitioners’ specific needs and preferences (R4, R5, R9). They
also noted that academic researchers might not usually bear the responsibility
of disseminating their knowledge beyond their specific disciplines (R5, R10).
Several researchers also voiced criticism regarding the current reviewing
system, which places more emphasis on new studies rather than replication and
guideline development (R10). Additionally, they highlighted the absence of
funding to support such endeavors (R9).

####  4.9.4. Ideas and Methods for Alleviating the Gap

The main theme from practitioners was again centered around making resources
more accessible and applicable. They suggested simplifying technical language
to partnering with practitioners for applied studies (P4, P6, P7, P10), as P6
said, “It would be good if the academics would venture a little more out into
practice. And it would be good if practitioners would open their minds a
little bit”. They also emphasized the importance of showcasing real-world
examples (P1, P4, P6, P7), as well as simplified summaries of research
findings (P10). For instance, P1 suggested augmenting existing programming
libraries with design guidelines. Practitioners also advocated for publicly
available content, such as workshops, newsletters, and courses, to enhance
knowledge transfer (P2, P3, P5). In addition, they recommended increasing the
visibility and accessibility of research through online platforms and open-
access knowledge channels (P9).

Researchers’ ideas were not very different but provided different angles for
tackling the gap. Participants emphasized the importance of mutual
understanding through workshops, collaborations, and sharing perspectives (R1,
R6, R7, R8, R9). They expressed a need for practitioners’ viewpoints in the
design process and guidelines development (R1, R4, R6). Suggestions included
making research findings more accessible through mediums like articles,
videos, and blog posts (R1, R2, R6, R7). Furthermore, the significance of
actionable design guidelines grounded in empirical evidence was highlighted
(R3, R9). For instance, R9 elaborated, “And so having the pros and cons,
having examples of where this went wrong, having examples of where actually it
made sense to use a pie chart or what not yeah. So to give the practitioner a
bit of a feel for what these arguments are about and what the layer of the
land is there in the current research.” Researchers also commented on
incentives to engage in outreach and communication efforts (R7, R10). Finally,
understanding the current knowledge transfer process and its potential harms &
incentives was highlighted as a crucial first step (R7).

##  5\. Discussion

In this section, we would like to brainstorm a comprehensive idea that can
synthesize the lessons learned to improve the current knowledge transfer
between visualization research and practice.

First of all, there appears to be a need to establish consistent vocabularies
for describing data visualizations, including chart names and task types. This
issue was observed during our examination of guidelines and was reiterated in
interviews with practitioners. While researchers may require a more granular
vocabulary, such as the task taxonomy proposed by Brehmer and Munzner (Brehmer
and Munzner, 2013), it must be tailored to the level of understanding of
practitioners. Currently, it is common for practitioners to fail to
differentiate between tasks, chart types, and data types when categorizing
different visualizations (dat, 2023a).

A critical issue we observed is conflicting design knowledge available in
practice, as well as authoritative guidelines that do not acknowledge
contextual factors. Interviewees often recognized this issue and suggested the
need for improving current design guidelines with empirical evidence details.
We can imagine building a platform where people can access design guidelines
backed up with related empirical studies, as we did in our alignment analysis.
Empirical evidence should be presented in language that is understandable to
practitioners, and it should explain the context of the research so that they
understand its limitations and applicability. This will improve the
trustworthiness of each guideline and increase its adoption by practitioners.

We also observed how practitioners appreciate intuition and experience in
design practice. We need to consider how to incorporate such experiential
knowledge into a knowledge platform. VisGuides (Diehl et al., 2018) is an
example of a platform where practitioners can share their examples for
discussion and feedback. We can also imagine connecting existing examples to
guidelines to create a triadic model of guidelines, empirical evidence, and
relevant examples. The examples can demonstrate how people applied or deviated
from the guidelines and present their success and failure cases. Furthermore,
these examples might inspire researchers to develop future practice-oriented
studies, thereby building a feedback and feedforward loop.

Participants also shared the difficulty of finding relevant guidelines.
Therefore, annotating the guidelines with detailed metadata, such as tasks,
chart types, or other contextual factors, would be necessary to facilitate
efficient navigation of the landscape of visualization design knowledge.
Current efforts by practitioners (dat, 2023a, b, d) provide hints on how this
idea can be realized. For example, a decision tree can be used to find a chart
based on data types or task goals. However, as mentioned earlier,
practitioners’ taxonomy is currently not consistent and streamlined compared
to the research side. To alleviate this issue, we could utilize templates to
create more structured and easily consumable guidelines, as demonstrated in
Figure 3.

Although having a comprehensive and accessible platform for every stakeholder
sounds like a great idea, a significant challenge remains: who will keep this
platform running? We believe that it will require community efforts from both
practitioners and researchers. Even if there are efforts for both stakeholders
to meet and collaborate, we have not yet seen a community that brings both
together. For example, IEEE VIS is mainly for researchers, while the Data
Visualization Society is mainly for practitioners. Instead of creating a new
community site, it might be helpful to leverage such existing communities to
reach out to both stakeholders.

The most significant roadblock to building a successful platform would be
keeping the design knowledge up to date. Who will author new design guidelines
or maintain existing ones? Who will translate the results of empirical
studies? How can we provide incentives for this extra work? Currently,
practitioners may write guidelines to gain traction for their commercial tools
(e.g., DataWrapper (dat, 2023f)) or attract potential clients (e.g., Data Viz
Project (dat, 2023d)). Others may do it for recognition, as researchers often
write blog articles about their research. However, building a successful
online community is not an easy task (Kraut and Resnick, 2012).

In fact, the lack of incentives was one of the issues raised in the
interviews. How can we provide both intrinsic and extrinsic motivation to
encourage contributing to the maintenance and development of the knowledge
base? To reduce human effort, one can seek intelligent assistance in
translating empirical results into guidelines. Recent advancements in large
language models demonstrate that this is a possibility. However, we might
still need humans to verify the outcomes. A peer-review model to monitor and
verify the input to the knowledge base is one idea to prevent malicious
participation. To motivate reviews, Ko’s recent proposal of a market-based
incentive model provides an inspiring idea for this issue (Ko, 2023) in which
tokens are provided for contributions.

##  6\. Conclusion

Our study investigated the knowledge gap between visualization research and
practice. We found a concentration of knowledge on basic chart types and task
types, a lack of transfer of empirical research to practice, and a cohesive
opinion on the knowledge gap among practitioners and researchers. We distilled
the lessons by proposing an idea for an online community platform encompassing
both practitioners and researchers. We plan to continue this work to build
such a platform.

## References

  * (1)
  * dat ([n. d.]) [n. d.].  Data Stories.  https://datastori.es/.  Accessed: 2023-03-21.
  * mul ([n. d.]) [n. d.].  Multiple Views: Visualization Research Explained.  https://medium.com/multiple-views-visualization-research-explained.  Accessed: 2023-03-21.
  * vis ([n. d.]) [n. d.].  VisPerception.  http://visperception.com/.  Accessed:2020-12-31.
  * dat (2023a) Accessed: February 25, 2023a.  Data To Viz.  https://www.data-to-viz.com/.
  * dat (2023b) Accessed: February 25, 2023b.  The Data Visualisation Catalogue.  https://datavizcatalogue.com/.
  * dat (2023d) Accessed: February 25, 2023d.  Data Viz Project.  https://datavizproject.com/.
  * dat (2023e) Accessed: February 25, 2023e.  DataJournalism.com.  https://datajournalism.com/.
  * dat (2023f) Accessed: February 25, 2023f.  Datawrapper.  https://www.datawrapper.de/.
  * ado (2023) Accessed: March 22, 2023.  Adobe design system.  https://spectrum.adobe.com/.
  * goo (2023) Accessed: March 22, 2023.  Google design system.  https://m2.material.io/design/.
  * dat (2023c) Accessed: March 28, 2023c.  Data Visualization Society.  https://www.datavisualizationsociety.com.
  * sto (2023) Accessed: March 28, 2023.  Storytelling with Data - Community Forum.  https://community.storytellingwithdata.com/.
  * tab (2023) Accessed: March 28, 2023.  Tableau - Tableau Official Site.  https://www.tableau.com.
  * voy (2023) Accessed: March 28, 2023.  vega/voyager: Visualization Tool for Data Exploration \- GitHub.  https://github.com/vega/voyager.
  * Alper et al. (2017) Basak Alper, Nathalie Henry Riche, Fanny Chevalier, Jeremy Boy, and Metin Sezgin. 2017\.  Visualization literacy at elementary school. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. 5485–5497.
  * Amar et al. (2005) R. Amar, J. Eagan, and J. Stasko. 2005.  Low-level components of analytic activity in information visualization. In _IEEE Symposium on Information Visualization, 2005. INFOVIS 2005._ 111–117.  https://doi.org/10.1109/INFVIS.2005.1532136
  * Bateman et al. (2010) Scott Bateman, Regan L Mandryk, Carl Gutwin, Aaron Genest, David McDine, and Christopher Brooks. 2010\.  Useful junk?: the effects of visual embellishment on comprehension and memorability of charts. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, 2573–2582.
  * Bigelow et al. (2014) Alex Bigelow, Steven Drucker, Danyel Fisher, and Miriah Meyer. 2014.  Reflections on How Designers Design with Data. In _Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces_ (Como, Italy) _(AVI ’14)_. Association for Computing Machinery, New York, NY, USA, 17–24.  https://doi.org/10.1145/2598153.2598175
  * Borgo et al. (2012) Rita Borgo, Alfie Abdul-Rahman, Farhan Mohamed, Philip W Grant, Irene Reppa, Luciano Floridi, and Min Chen. 2012.  An empirical study on using visual embellishments in visualization.  _IEEE Transactions on Visualization and Computer Graphics_ 18, 12 (2012), 2759–2768.
  * Borkin et al. (2015) Michelle A Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. 2015.  Beyond memorability: Visualization recognition and recall.  _IEEE transactions on visualization and computer graphics_ 22, 1 (2015), 519–528.
  * Börner et al. (2019) Katy Börner, Andreas Bueckle, and Michael Ginda. 2019\.  Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments.  _Proceedings of the National Academy of Sciences_ 116, 6 (2019), 1857–1864.
  * Bostock et al. (2011) Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011\.  D³ Data-Driven Documents.  _IEEE Transactions on Visualization and Computer Graphics_ 17, 12 (2011), 2301–2309.  https://doi.org/10.1109/TVCG.2011.185
  * Boy et al. (2014) Jeremy Boy, Ronald A Rensink, Enrico Bertini, and Jean-Daniel Fekete. 2014.  A principled way of assessing visualization literacy.  _IEEE transactions on visualization and computer graphics_ 20, 12 (2014), 1963–1972.
  * Brehmer and Munzner (2013) Matthew Brehmer and Tamara Munzner. 2013.  A multi-level typology of abstract visualization tasks.  _IEEE transactions on visualization and computer graphics_ 19, 12 (2013), 2376–2385.
  * Cairo (2019) Alberto Cairo. 2019\.  _How charts lie: Getting smarter about visual information_.  WW Norton & Company.
  * Chen et al. (2017) Min Chen, Georges Grinstein, Chris R. Johnson, Jessie Kennedy, and Melanie Tory. 2017\.  Pathways for Theoretical Advances in Visualization.  _IEEE Computer Graphics and Applications_ 37, 4 (2017), 103–112.  https://doi.org/10.1109/MCG.2017.3271463
  * Cleveland and McGill (1984) William S Cleveland and Robert McGill. 1984.  Graphical perception: Theory, experimentation, and application to the development of graphical methods.  _Journal of the American statistical association_ 79, 387 (1984), 531–554.
  * Cook and Thomas (2005) Kristin A Cook and James J Thomas. 2005.  _Illuminating the path: The research and development agenda for visual analytics_.  Technical Report. Pacific Northwest National Lab.(PNNL), Richland, WA (United States).
  * Diehl et al. (2018) Alexandra Diehl, Alfie Abdul-Rahman, Mennatallah El-Assady, Benjamin Bach, Daniel A Keim, and Min Chen. 2018.  VisGuides: A Forum for Discussing Visualization Guidelines. In _EuroVis (Short Papers)_. 61–65.
  * Dimara et al. (2018) Evanthia Dimara, Steven Franconeri, Catherine Plaisant, Anastasia Bezerianos, and Pierre Dragicevic. 2018\.  A task-based taxonomy of cognitive biases for information visualization.  _IEEE transactions on visualization and computer graphics_ (2018).
  * Haroz et al. (2015) Steve Haroz, Robert Kosara, and Steven L. Franconeri. 2015\.  ISOTYPE Visualization: Working Memory, Performance, and Engagement with Pictographs. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (Seoul, Republic of Korea) _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 1191–1200.  https://doi.org/10.1145/2702123.2702275
  * He and Adar (2017) Shiqing He and Eytan Adar. 2017.  VizItCards: A Card-Based Toolkit for Infovis Design Education.  _IEEE Transactions on Visualization and Computer Graphics_ 23, 1 (2017), 561–570.  https://doi.org/10.1109/TVCG.2016.2599338
  * Heer and Bostock (2010) Jeffrey Heer and Michael Bostock. 2010.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In _Proceedings of the SIGCHI conference on human factors in computing systems_. ACM, 203–212.
  * Hu et al. (2019) Kevin Hu, Michiel A Bakker, Stephen Li, Tim Kraska, and César Hidalgo. 2019.  Vizml: A machine learning approach to visualization recommendation. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Inselberg and Dimsdale (1990) A. Inselberg and B. Dimsdale. 1990.  Parallel coordinates: a tool for visualizing multi-dimensional geometry. In _Proceedings of the First IEEE Conference on Visualization: Visualization ‘90_. 361–378.  https://doi.org/10.1109/VISUAL.1990.146402
  * Isenberg et al. (2017) Petra Isenberg, Florian Heimerl, Steffen Koch, Tobias Isenberg, Panpan Xu, Chad Stolper, Michael Sedlmair, Jian Chen, Torsten Möller, and John Stasko. 2017\.  vispubdata.org: A Metadata Collection about IEEE Visualization (VIS) Publications.  _IEEE Transactions on Visualization and Computer Graphics_ 23 (2017).  https://doi.org/10.1109/TVCG.2016.2615308 To appear.
  * Johnson and Shneiderman (1998) Brian Johnson and Ben Shneiderman. 1998.  _Tree-maps: A space filling approach to the visualization of hierarchical information structures_.  Technical Report.
  * Joyner et al. (2022) Shakila Cherise S Joyner, Amalia Riegelhuth, Kathleen Garrity, Yea-Seul Kim, and Nam Wook Kim. 2022\.  Visualization Accessibility in the Wild: Challenges Faced by Visualization Designers. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 83, 19 pages.  https://doi.org/10.1145/3491102.3517630
  * Kim et al. (2016) Nam Wook Kim, Eston Schweickart, Zhicheng Liu, Mira Dontcheva, Wilmot Li, Jovan Popovic, and Hanspeter Pfister. 2016.  Data-driven guides: Supporting expressive design for information graphics.  _IEEE transactions on visualization and computer graphics_ 23, 1 (2016), 491–500.
  * Kim and Heer (2018) Younghoon Kim and Jeffrey Heer. 2018.  Assessing effects of task and data distribution on the effectiveness of visual encodings. In _Computer Graphics Forum_ , Vol. 37. Wiley Online Library, 157–167.
  * Ko (2023) Amy Ko. 2023.  Sustainable peer review via incentive aligned markets.  https://medium.com/bits-and-behavior/sustainable-peer-review-via-incentive-aligned-markets-a64ff726da56 [Online; posted 27-March-2023].
  * Kosara (2016) Robert Kosara. 2016\.  An Empire Built On Sand: Reexamining What We Think We Know About Visualization. In _Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization_ (Baltimore, MD, USA) _(BELIV ’16)_. Association for Computing Machinery, New York, NY, USA, 162–168.  https://doi.org/10.1145/2993901.2993909
  * Kraus et al. (2020) Matthias Kraus, Katrin Angerbauer, Juri Buchmüller, Daniel Schweitzer, Daniel A. Keim, Michael Sedlmair, and Johannes Fuchs. 2020\.  Assessing 2D and 3D Heatmaps for Comparative Analysis: An Empirical Study. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3313831.3376675
  * Kraut and Resnick (2012) Robert E Kraut and Paul Resnick. 2012.  _Building successful online communities: Evidence-based social design_.  Mit Press.
  * Lam et al. (2011) Heidi Lam, Enrico Bertini, Petra Isenberg, Catherine Plaisant, and Sheelagh Carpendale. 2011\.  Empirical studies in information visualization: Seven scenarios.  _IEEE transactions on visualization and computer graphics_ 18, 9 (2011), 1520–1536.
  * Lee et al. (2016) Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. 2016.  Vlat: Development of a visualization literacy assessment test.  _IEEE transactions on visualization and computer graphics_ 23, 1 (2016), 551–560.
  * Liu and Heer (2018) Yang Liu and Jeffrey Heer. 2018.  Somewhere Over the Rainbow: An Empirical Assessment of Quantitative Colormaps. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3173574.3174172
  * Liu et al. (2018) Zhicheng Liu, John Thompson, Alan Wilson, Mira Dontcheva, James Delorey, Sam Grigg, Bernard Kerr, and John Stasko. 2018\.  Data Illustrator: Augmenting vector design tools with lazy data binding for expressive visualization authoring. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. ACM, 123.
  * Lo et al. (2022) Leo Yu-Ho Lo, Ayush Gupta, Kento Shigyo, Aoyu Wu, Enrico Bertini, and Huamin Qu. 2022\.  Misinformed by Visualization: What Do We Learn From Misinformative Visualizations?  _Computer Graphics Forum_ 41, 3 (2022), 515–525.  https://doi.org/10.1111/cgf.14559
  * Meyer et al. (2015) Miriah Meyer, Michael Sedlmair, P Samuel Quinan, and Tamara Munzner. 2015.  The nested blocks and guidelines model.  _Information Visualization_ 14, 3 (2015), 234–249.  https://doi.org/10.1177/1473871613510429 arXiv:https://doi.org/10.1177/1473871613510429
  * Munzner (2014) Tamara Munzner. 2014\.  _Visualization analysis and design_.  CRC press.
  * Parsons (2022) Paul Parsons. 2022\.  Understanding Data Visualization Design Practice.  _IEEE Transactions on Visualization and Computer Graphics_ 28, 1 (2022), 665–675.  https://doi.org/10.1109/TVCG.2021.3114959
  * Potter and Levine‐Donnerstein (1999) W. James Potter and Deborah Levine‐Donnerstein. 1999\.  Rethinking validity and reliability in content analysis.  _Journal of Applied Communication Research_ 27, 3 (1999), 258–284.  https://doi.org/10.1080/00909889909365539
  * Quadri and Rosen (2022) Ghulam Jilani Quadri and Paul Rosen. 2022.  A Survey of Perception-Based Visualization Studies by Task.  _IEEE Transactions on Visualization and Computer Graphics_ 28, 12 (2022), 5026–5048.  https://doi.org/10.1109/TVCG.2021.3098240
  * Roberts et al. (2016) Jonathan C. Roberts, Chris Headleand, and Panagiotis D. Ritsos. 2016\.  Sketching Designs Using the Five Design-Sheet Methodology.  _IEEE Transactions on Visualization and Computer Graphics_ 22, 1 (2016), 419–428.  https://doi.org/10.1109/TVCG.2015.2467271
  * Saket et al. (2018) Bahador Saket, Alex Endert, and Çağatay Demiralp. 2018\.  Task-based effectiveness of basic visualizations.  _IEEE transactions on visualization and computer graphics_ 25, 7 (2018), 2505–2512.
  * Satyanarayan et al. (2017) Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2017.  Vega-Lite: A Grammar of Interactive Graphics.  _IEEE Transactions on Visualization and Computer Graphics_ 23, 1 (2017), 341–350.  https://doi.org/10.1109/TVCG.2016.2599030
  * Scott-Brown (2018) James Scott-Brown. 2018\.  Presenting Visualization Guideline Collections.  _2nd IEEE VIS 2018 Workshop on the Creation, Curation, Critique and Conditioning of Principles and Guidelines in Visualization_ (2018).  https://c4pgv.dbvis.de/Scott-Brown_2018.pdf
  * Sedlmair et al. (2012) Michael Sedlmair, Miriah Meyer, and Tamara Munzner. 2012\.  Design Study Methodology: Reflections from the Trenches and the Stacks.  _IEEE Transactions on Visualization and Computer Graphics_ 18, 12 (2012), 2431–2440.  https://doi.org/10.1109/TVCG.2012.213
  * Stolte et al. (2002) Chris Stolte, Diane Tang, and Pat Hanrahan. 2002.  Polaris: A system for query, analysis, and visualization of multidimensional relational databases.  _IEEE Transactions on Visualization and Computer Graphics_ 8, 1 (2002), 52–65.
  * Wang et al. (2020) Zezhong Wang, Lovisa Sundin, Dave Murray-Rust, and Benjamin Bach. 2020.  Cheat sheets for data visualization techniques. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Zeng and Battle (2023) Zehua Zeng and Leilani Battle. 2023.  A Review and Collation of Graphical Perception Knowledge for Visualization Recommendation.  arXiv:2109.01271 [cs.HC]
  * Zhang et al. (2023) Yixuan Zhang, Yifan Sun, Joseph D. Gaggiano, Neha Kumar, Clio Andris, and Andrea G. Parker. 2023\.  Visualization Design Practices in a Crisis: Behind the Scenes with COVID-19 Dashboard Creators.  _IEEE Transactions on Visualization and Computer Graphics_ 29, 1 (2023), 1037–1047.  https://doi.org/10.1109/TVCG.2022.3209493