# How Good is ChatGPT in Giving Advice on Your Visualization Design?

Nam Wook Kim  Boston College140 Commonwealth AveChestnut
HillMassachusettsUSA02467<EMAIL_ADDRESS>,  Grace Myers  Boston
College140 Commonwealth AveChestnut HillMassachusettsUSA02467
<EMAIL_ADDRESS>and  Benjamin Bach  University of Edinburgh Old College,
South BridgeEdinburghUnited Kingdom<EMAIL_ADDRESS>
(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Data visualization practitioners often lack formal training, resulting in a
knowledge gap in visualization design best practices. Large-language models
like ChatGPT, with their vast internet-scale training data, offer
transformative potential in addressing this gap. To explore this potential, we
adopted a mixed-method approach. Initially, we analyzed the VisGuide forum, a
repository of data visualization questions, by comparing ChatGPT-generated
responses to Human replies. Subsequently, our user study delved into
practitioners’ reactions and attitudes toward ChatGPT as a visualization
assistant. Participants, who brought their visualizations and questions,
received feedback from both Human experts and ChatGPT in a randomized order.
They filled out experience surveys and shared deeper insights through post-
interviews. The results highlight the unique advantages and disadvantages of
ChatGPT, such as its ability to quickly provide a wide range of design options
based on a broad knowledge base, while also revealing its limitations in terms
of depth and critical thinking capabilities.

data visualization, design feedback, ChatGPT, LLM, AI

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Human-centered computing Empirical studies in visualization††ccs: Human-
centered computing User studies

##  1\. Introduction

Visualizations are ubiquitous and widely employed by practitioners across
various disciplines. However, many data visualization practitioners commonly
lack formal training and instead acquire the necessary skills on the go, often
leaning on examples from the internet, online blogs, and other publicly
available resources (Esteves and Neves, 2022; dvs, [n. d.]). They frequently
find themselves in situations where they must navigate intricate and
occasionally contradictory choices (Choi et al., 2023b). In such instances,
they often revert to their instincts, drawing from the experiences and
observations they have accumulated along the way (Choi et al., 2023b; Bako et
al., 2023). Alternatively, they seek feedback from colleagues, often field
experts, who can help provide novel perspectives, validate design choices, and
challenge assumptions (Choi et al., 2023b; Luther et al., 2015). Nevertheless,
not all practitioners have the privilege of accessing such feedback from
experienced colleagues.

This work delves into ways to address the knowledge gap in practical contexts.
We examine the emergence of large language models (LLMs) equipped with
extensive internet-scale training data, which holds transformative potential
(Okerlund et al., 2022). LLM-based chatbots, exemplified by ChatGPT (OpenAI,
[n. d.]), can serve as a design companion to offer tailored suggestions by
analyzing a practitioner’s design choices and considering established best
practices. This paradigm shift can level the playing field, enabling
practitioners with limited knowledge and resources to receive high-quality
guidance in their design journey. Our two specific research questions are as
follows:

  * •

RQ1. Can ChatGPT rival Human expertise in data visualization knowledge?

  * •

RQ2. How would visualization practitioners perceive ChatGPT’s design feedback?

Figure 1. Methodology Overview: The methodology comprises two key phases. In
the first phase, questions answered within a forum space by Human respondents
are explored and then presented to ChatGPT. The second phase involves a
feedback session in which users solicit visualization design feedback from
both ChatGPT and Human experts.

Both phases are represented by a flow diagram chart. On left is the first
phase in which arrows point from question to ChatGPT responses and from
question to Human responses. There is also an arrow from content analysis that
points to both Human responses and ChatGPT responses.In phase two, there are
three parts. The first part is the data viz practitioner who comes to the
session with a design feedback based question. the second part is the two
feedback sessions, one with ChatGPT and one with the Human experts. The third
part is the follow up interview in which the participant shares about their
experience.

We employ a mixed-method approach (Figure 1 to explore the potential of
ChatGPT in offering data visualization design knowledge. First, we
investigated the VisGuides forum, which provides a valuable repository of
questions and responses from practitioners. We filtered out inadequately
formulated questions lacking context, such as vague queries like “how do you
like my design?” while only including the ones with at least one response. Our
final collection comprises 119 questions, each accompanied by an average of
1.54 replies. We fed the questions to ChatGPT and compared its responses to
the Human counterparts across 6 key metrics: coverage, topicality, breadth,
clarity, depth, and actionability. We found that ChatGPT’s performance is
comparable to, and often better than, human responses, especially in providing
clear and comprehensive answers that cover all questions asked.

While the VisGuide forum analysis provided insight into ChatGPT’s response
quality, it fell short of capturing practitioners’ reactions, attitudes, and
overall experiences. Thus, we conducted a subsequent user study wherein we
invited practitioners to partake in a comparative assessment of Human expert
feedback and ChatGPT-generated feedback. Participants were recruited through
the Data Visualization Society and a university mailing list. Before the
study, participants were requested to bring their visualizations along with
questions for feedback. The sequence in which they sought feedback, either
from Human experts or ChatGPT, was randomized to mitigate order effects.
Interestingly, all participants preferred Human experts, valuing their ability
to engage in fluid conversations and provide tailored recommendations.
However, they also appreciated the broad knowledge demonstrated by ChatGPT,
despite noting that ChatGPT’s responses lacked depth.

Synthesizing these two studies, we can draw valuable lessons and identify
future opportunities. For instance, the variation in human responses within
VisGuides could contribute to the observed differences between the studies, as
there was a mix of both high-quality and subpar responses. One significant
limitation of ChatGPT, as pointed out by participants, is its inability to
grasp nuanced visuals and engage in proactive discussions, especially when
compared to human experts. Furthermore, we explore the potential advantages of
integrating ChatGPT into visualization education and its role as a design
knowledge agent within existing visualization tools for practitioners.

##  2\. Related Work

###  2.1. Data Visualization Design Practice

Data visualization has gone mainstream, frequently used to explore and
communicate data in a wide variety of industry sectors. As a result, data
visualization practitioners have diverse professions, such as managers,
scientists, journalists, and analysts, not necessarily having a specific job
title as “visualization designer” or “visualization developer.” (Esteves and
Neves, 2022) These professionals do not typically have formal training or
education in data visualization, while mostly learning necessary visualization
skills on the go (Esteves and Neves, 2022; dvs, [n. d.]). While decades of
empirical research in data visualization have produced fruitful knowledge in
effective visualization design, it is unclear how well it is accessible to
practice (Choi et al., 2021; Diehl et al., 2018). In fact, it has been
reported that many practitioners are not aware of theories and principles from
research (Parsons, 2022).

As with any creative design profession, visualization practitioners run into
challenging situations where they must make decisions among conflicting design
alternatives (Choi et al., 2023b). Such decisions might include selecting
chart types, making competing decisions between aesthetics and functions,
picking appropriate color scales, and addressing conflicts with user and
business stakeholder needs (Choi et al., 2023b). Processes by practitioners to
address such design issues are more nuanced and multi-faceted than theoretical
process models proposed in the research field (Parsons, 2022). They depend on
a kind of situated planning that often relies on forms of experimentation and
responding to what is happening in the moment (Parsons, 2022; Parsons et al.,
2020). They balance readability and engagement and also consider contextual
factors such as whether their visualizations are business-oriented dashboards
or public-facing graphics (Parsons and Shukla, 2020).

Practitioners frequently resort to their intuition or gut instinct while also
looking for examples for inspiration (Choi et al., 2023b; Parsons, 2022).
Moreover, they also seek feedback from their colleagues to improve their data
visualizations (Choi et al., 2023b; Esteves and Neves, 2022). Such feedback is
essential to assess a design and generate revision ideas in the design
process, although the fear of criticism and non-anonymity can make people
uncomfortable receiving feedback from colleagues. Moreover, self-employed
people might struggle to find such colleagues who can provide valuable
feedback (Chalkidis et al., 2019)

In this work, we examine how an LLM-based chatbot, trained with extensive
knowledge in the wild, can serve as a design companion to provide knowledge
and feedback to practitioners.

###  2.2. Evaluating LLMs’ Knowledge Capacity

LLMs, such as BERT (Devlin et al., 2018), GPT (Brown et al., 2020; OpenAI,
2023), and Llama (Touvron et al., 2023), have rapidly gained popularity and
adoption across both industry and academia (Bommasani et al., 2021; Zhao et
al., 2023). Leveraging massive amounts of text data and computational power,
these foundation models have achieved impressive performance on a wide variety
of tasks, from generating human-like text to high-quality images (Radford et
al., 2019, 2018; Wei et al., 2022). However, the scale and complexity of these
models make it difficult to fully understand the scope and limitations of
their language capabilities (Bender et al., 2021).

A number of benchmark tasks have been proposed to assess the general knowledge
and reasoning capabilities of language models in a more holistic manner (Liang
et al., 2022; Chang et al., 2023). These include tasks that require models to
answer broad-coverage factoid questions based on pre-existing knowledge (Liang
et al., 2022). Other benchmarks evaluate numeric, analogical, and logical
reasoning skills across different modalities (Qin et al., 2023; Zhang et al.,
2023), while some others examine the ability of LLMs to generalize knowledge
to new concepts (Wei et al., 2022). Assessing models on such diverse reasoning
skills can provide deeper insight into their adaptability, knowledge gaps, and
limitations. In addition to assessing such capabilities, an important area of
research has focused on evaluating potential harms and biases in large
language models. Some studies have performed targeted adversarial evaluations
to expose problematic biases encoded in the models’ training data (Nadeem et
al., 2021; Tan and Celis, 2019; Gehman et al., 2020).

In addition to generic capabilities, research has been carried out to
investigate LLMs’ discipline-specific knowledge and skills. The inherent
intricacies and nuances of specific domains necessitate tailored evaluation
methods. For instance, researchers explored students’ and teachers’ usage of
and perceptions toward LLMs, investigating opportunities and potential
benefits for education (Kasneci et al., 2023; Baidoo-Anu and Owusu Ansah,
2023; Tlili et al., 2023). Researchers also tested LLMs’ ability to pass exams
in specific domains like medicine (Gilson et al., 2023), law (Choi et al.,
2023a), and business (Mollman, 2023). Others have evaluated LLMs’ ability to
carry out real-world tasks. Examples include analyzing diagnostic abilities by
answering questions about medications, symptoms and treatments (Lee et al.,
2023); producing research hypotheses (Lahat et al., 2023); predicting legal
judgment and recommending case citations (Chalkidis et al., 2019); and
predicting chemical properties (Pan, 2023).

Recent studies have initiated the exploration of Large Language Models’ (LLMs)
applicability in the realm of data visualization. Some investigations have
examined their capacity to generate data visualizations (Paik, 2023;
Hutchinson et al., 2023), facilitate exploratory data analysis (Dibia, 2023),
and create stylistic data graphics (Dibia, 2023), in addition to generating
graph drawings (Bartolomeo et al., 2023). Furthermore, studies have delved
into the usability of LLM-based code generation in coding data visualizations
(Vaithilingam et al., 2022), as well as understanding visualization
practitioners’ perceptions regarding generative text-to-image models
(Schetinger et al., 2023). Our goal in this work is to explore the potential
of LLMs in addressing the knowledge gap among data visualization practitioners
by offering design insights and feedback, thereby shedding light on the
efficacy of LLMs as design companions.

##  3\. Assessing ChatGPT’s Competence in Data Visualization Knowledge

As a step toward assessing ChatGPT’s data visualization knowledge, we were
interested in its ability to respond to real-world data visualization
questions in comparison to Humans.

###  3.1. Data Collection

We relied on VisGuides (Diehl et al., 2018) as our main source for
visualization questions. VisGuides is a discussion forum focused on
visualization guidelines, where practitioners from various backgrounds, such
as scientists, designers, engineers, and students, ask and respond to
questions regarding their visualizations and fundamental visualization
principles.

We identified a total of 226 questions within the VisGuides repository. Among
these, we selected 119 questions for the study based on specific inclusion
criteria. These criteria included the presence of a question in the post, the
requirement for a visualization description to possess sufficient visual
encoding (due to ChatGPT’s inability to process images), the need for
questions to be comprehensible and well-structured (excluding overly generic
inquiries like ’What do you think?’ or ’Is this good enough?’), and the
necessity of having at least one Human response. Two of the authors jointly
reviewed and unanimously approved the inclusion of these 119 questions.

Visguides primarily featured two types of questions, broadly divided into two
categories: design feedback questions (87) and visualization guideline
questions (32). The former aimed to enhance users’ personal visualizations
(see Figure 3 B & C), while the latter sought to comprehend visualization
guidelines or principles (see Figure 3 A). For a more comprehensive taxonomy
of questions on the platform, Diehl et al. provide detailed insights (Diehl et
al., 2021).

After compiling the list of qualifying questions, we presented each query to
ChatGPT, accompanied by a role-playing prompt designed as follows:

    Please act as if you are a visualization expert.
    Can you respond to this question delimited by ///.
     ///
     {Question}
     ///
    Please format your reply as JSON objects with
    keys: Question and Response.
     {
        Question: xxx
        Response: xxx
     }
    There may be more than one question,
    format each as a new object.

To streamline our analysis, we created a spreadsheet where each row included a
question, the corresponding ChatGPT response, and a compilation of Human
responses. Each question had 1.54 Human responses on average ($Std.dev$ =
1.03). The average approximate word count of the Human responses was 235.35
($Std.dev$ = 289.62) with a maximum of 1721 words and minimum of 13 words. The
average approximate word count of the ChatGPT responses was 463.42 ($Std.dev$
= 189.34) with a maximum of 1192 words and minimum of 115 words.

###  3.2. Analysis Method

Two researchers performed an open coding process on a randomly selected 10%
sample of the questions and the corresponding ChatGPT and Human responses.
This process resulted in the development of six evaluation metrics: breadth,
clarity, depth, actionability, coverage, and topicality. These metrics were
assessed using a Likert scale that ranged from 1 to 5. For instance, a rating
of 1 indicated narrow responses, while a rating of 5 denoted very broad
responses in the context of the breadth metric. The researchers initially
assigned scores to the 10% sample, engaging in discussions to resolve any
discrepancies and establish consensus definitions and criteria for each score
within every metric to ensure consistency. Following this, one of the
researchers proceeded to score the remaining question responses.

Given that some questions elicited multiple Human responses while others
received only one, We created a composite score to evaluate the collective
quality of the Human responses. Furthermore, any elements of the Human
responses that extended beyond the capabilities of ChatGPT, such as providing
citations to related literature, links to external resources, visual examples,
or posing follow-up questions, were duly noted. We also categorized whether
each question sought feedback on the questioner’s visualizations or inquired
about general visualization design knowledge.

###  3.3. Results

Figure 2. Comparison of metric scores between Human and ChatGPT responses: the
chart showcases the existence of significant differences for the metrics of
breadth, clarity, and coverage, while the scores for actionablity, depth and
topicality were comparable for ChatGPT and Human response ratings.

Comparing metric scores. This graph presents an evaluation of six key metrics
on the y-axis. The x-axis represents scores, ranging from 0 to 5.0. The
dataset consists of two distinct conditions: Human responses and responses
generated by ChatGPT. For each of the aforementioned metrics, the graph
displays scores for both ChatGPT and Human responses in the form of interval
plots. Additionally, confidence intervals are included to illustrate the
degree of uncertainty in the measurements. In the case of actionability,
depth, and topic focus, the ChatGPT and Human responses exhibit overlapping
score intervals, indicating no significant differences between them. However,
for the metrics of breadth, coverage, and clarity, ChatGPT consistently
achieved higher scores compared to Human responses. Importantly, the intervals
representing these metrics do not intersect, reinforcing the statistical
significance of these differences.

Figure 3. Examples of visguide questions and responses: Each question serves
as a unique example, accompanied by responses from both ChatGPT and a Human
expert. The ChatGPT responses consistently exhibit breadth, topicality, and
coverage. The Human response rating are more varied, as is evident when
comparing the first and last example.

This visual representation showcases three distinct instances of questions
posed within the context of Visguide. The three examples are as follows: 1.
Guideline Question on Color Palettes: The first question pertains to color
palettes and serves as a guideline inquiry. ChatGPT provides general
recommendations for color selection, while the Human expert response focuses
on a specific guideline and cites relevant academic sources. 2. General Design
Feedback Question with Bubble Chart: The second question revolves around
general design feedback, particularly regarding the enhancement of a bubble
chart visualization. The actual bubble chart visualization is presented. In
response, both ChatGPT and the Human expert offer suggestions and insights. 3.
Specific Design Feedback Question with Tabular Heat Map: The third question
addresses specific design feedback, centered on a tabular heat map
visualization. The question itself incorporates the visualization, allowing
readers to contextualize the discussion. Responses from ChatGPT and the Human
expert are provided. These three scenarios exemplify the versatility of
Visguide as an interactive platform for soliciting guidance and feedback, and
the varying nature of responses between ChatGPT and Human expertise.

####  3.3.1. Ratings on Quality Metrics

Figure 4 shows the scores of response quality metrics for both ChatGPT and
Human responses. We describe the findings for each quality metric as below.

##### Coverage:

Both ChatGPT and Humans excelled in covering the entirety of the question,
with ChatGPT slightly outperforming Humans. ChatGPT scored 4.92 ($std.dev$ =
0.33), while Humans scored 4.37 ($std.dev$ = 0.95). The lower score often
stemmed from the Human respondent’s failure to address all facets of the
question. For instance, in Figure 3C, the user raised a two-part question,
“…would you recommend all the states be placed on the Y-axis instead of
grouped as regions? If so, which visual layouts would be effective with
dealing with the large number of states?…”. However, the Human response only
addressed the second part of the question, suggesting a heat map as a useful
visual layout.

##### Topicality:

This score indicates the degree to which the response stays on topic. ChatGPT
averaged 4.87 ($std.dev$ = 0.45), while Humans averaged 4.72 ($std.dev$ =
0.59). There were moments when both ChatGPT and Humans strayed from the main
topic. For instance, when a user inquired about the application of visual
variables in VR visualizations,111https://visguides.org/t/visual-variables-
for-visualizations-in-vr/143 ChatGPT not only addressed this question but also
came up with a hypothetical question asking what factors to consider when
choosing such visual variables. Human respondents also occasionally veered
off-topic by offering unsolicited advice and extending their responses beyond
the original question’s scope.

##### Breadth

: Breadth measures how widely a response explores various ideas, concepts,
options, or perspectives. In this category, ChatGPT outperformed Human
responses by a significant margin. ChatGPT achieved an average score of 4.62
($std.dev$ = 0.61), while Human responses averaged only 3.35 ($std.dev$ =
1.13). For example, in Figure 3A, when the user inquired about color palette
guidelines, ChatGPT provided an extensive list of potential approaches,
encompassing data type, color harmony, accessibility, etc. In contrast, the
Human respondent to this question concentrated primarily on the creation of
domain-specific color maps, offering in-depth insights into the process.

##### Clarity:

The clarity score assesses how easily a reader can comprehend a response,
taking into account conciseness, lack of verbosity, and the organization of
the response structure. Notably, ChatGPT consistently presented its responses
in a well-structured JSON array due to the specific prompt format we employed.
This contributed to ChatGPT receiving a high clarity score, averaging 4.93
($std.dev$ = 0.28). Human responses also demonstrated good clarity with an
average score of 4.30 ($std.dev$ = 0.79). They received lower ratings when the
responses were overly brief, making it difficult to discern the specific
reference points.

##### Depth:

The depth score evaluates the extent of explanations, expertise, and valuable
insights in a response. Human responses averaged a score of 3.52 ($std.dev$ =
1.07), while ChatGPT’s average was 3.44 ($std.dev$ = 0.71). Although the
scores are comparable to each other, a distinct difference in depth between a
ChatGPT response and a Human response is evident in Figure 3B. The ChatGPT
response primarily focuses on general, surface-level improvements applicable
to any visualization, such as adding labels and legends. In contrast, the
Human response often provides more tailored and specific suggestions for
enhancing the visualization, as demonstrated by the recommendation to “put a
label before the numerical attribute like ’Estimated Production: 1000’.” This
detailed guidance directly pertains to the unique aspects of the visualization
in question. However, Human responses also displayed greater variability. For
instance, when asked about color clarity and alternative data representation
to reduce clutter, a Human response simply confirmed the current design
choices are fine222https://visguides.org/t/latitude-distribution-of-solar-and-
wind-farms-uk/815.

##### Actionability:

The actionability score assesses whether the response offers guidance that can
be readily implemented in the visualization. This metric is not applicable to
visualization guideline questions. The actionability scores were quite
similar: Human responses had an average score of 3.86 ($std.dev$ = 0.94),
while ChatGPT achieved an average of 3.99 ($std.dev$ = 0.61). Overall, ChatGPT
received a high score for actionability, as it consistently provided a list of
recommendations. However, the score was occasionally reduced due to the
generic or vague nature of these suggestions. For instance, in Figure 3B,
ChatGPT offered advice for addressing label clutter, selecting colors, and
refining the legend. Nevertheless, these recommendations were explained in a
manner that made them applicable to most visualizations. The actionability
scores for Human responses tend to be low when the Human refrains from
offering recommendations and instead validates all of the user’s
choices333https://visguides.org/t/comparison-of-type-a-b-acute-hepatitis-from-
tycho-dataset/406.

####  3.3.2. Further Observations in Human and ChatGPT Responses

Humans tend to provide external resources—43 out of 119 (36.13%) Human
responses included elements that ChatGPT would not provide unless explicitly
instructed. This additional content encompassed references to academic
research, links to related articles and websites, citations of studies, and
the inclusion of informative video links. For instance, in Figure 3A, asking
for guidelines on choosing color palette, the responses include references to
three different academic research papers including “H. Fang, et al.”. In
another case, when the question was about using blow-apart effects, the Human
respondent embedded an educational video related to the
subject444https://visguides.org/t/the-blow-apart-effect/71.

Multiple Human responses tend to complement each other—Most questions had a
single response (84), while 35 questions had multiple responses. Average
ratings generally favored responses with multiple contributors across all
categories, except for topicality. Significant rating differences were
observed in breadth, depth, and actionability. Single-respondent questions
averaged 3.06 ($std.dev$ = 1.05) in breadth, 3.35 ($std.dev$ = 1.05) in depth,
and 3.72 ($std.dev$ = 0.96) in actionability. Conversely, multi-respondent
questions scored higher, averaging 4.06 ($std.dev$ = 0.98) in breadth, 3.94
($std.dev$ = 0.98) in depth, and 4.26 ($std.dev$ = 0.74) in actionability.
When there’s only one response to a question, the quality of that response
becomes heavily reliant on the individual respondent, leading to significant
variability in quality, as seen in Figure 3A (high quality) versus Figure 3C
(low quality). However, when multiple respondents contribute, they complement
each other and compensate for areas in which one respondent might fall short
(Figure 3B).

Human response rating improves for visualization guideline questions—For
ChatGPT responses, the differences in ratings between feedback and
visualization guideline questions consistently remained small (less than 0.2
rating points). However, Human responses received higher scores for addressing
guideline questions and lower scores for addressing design feedback questions
across all metrics. For instance, Human responses excelled in breadth and
depth for visualization guideline questions, with average ratings of 4.09
($std.dev$ = 0.95) for breadth and 4.25 ($std.dev$ = 0.97) for depth. In
contrast, design feedback questions received lower average ratings, with 3.08
($std.dev$ = 1.06) for breadth and 3.25 ($std.dev$ = 0.97) for depth.

Guideline questions frequently require broader perspectives and readily
accessible design knowledge, with 84.4% (27/32) of their responses citing
external references, in contrast to design feedback questions, where only
18.39% (16/87) of Human responses did so. Conversely, feedback questions
entail an understanding of domain-specific data and tasks, potentially making
it challenging to offer comprehensive insights. For instance, a question like
“Is this color scheme suitable?” requires a deep understanding of the domain.
Conversely, questions such as the one about gendered colors in
visualizations555https://visguides.org/t/use-of-gendered-colours-in-
visualization-a-guideline-or-a-personal-principle/999 can easily elicit a
variety of various viewpoints and existing resources.

Question specificity corresponds with response quality—ChatGPT’s response was
susceptible to how specific and clear the question was. Figure 3B shows such
an example when the user query was “do you believe the graph is clear?” Other
examples of less specific user questions include: “is my color map optimal?”
and “how can my visual design be improved?”666https://visguides.org/t/map-and-
bar-visualization/841. On the other hand, in Figure 3C, the question is more
specific, resulting in more meaningful options and insights offered by
ChatGPT.

ChatGPT’s color vision deficiency is usually not problematic—Color-related
questions were prominent. Out of the 119 questions, 32 were centered on color-
based design feedback (e.g., “Is the choice of the colour scheme appropriate?”
777https://visguides.org/t/area-chart-generation-capacity-over-the-years-for-
each-power-fuel-type/810). Users rarely mentioned the specific colors used in
their visualizations, often providing only visual encoding information. As a
result, ChatGPT’s responses were typically based on general color principles.
Despite this limitation, ChatGPT surpassed its overall average in most
categories, except for depth and actionability.

###  3.4. Takeaways

Our analysis reveals ChatGPT’s ability to respond to data visualization
queries compared to Human counterparts. It’s noteworthy that ChatGPT, despite
lacking vision, demonstrates remarkable reliability in providing answers,
possibly owing to its exceptional language understanding and extensive
knowledge base. While these findings are promising, they don’t provide
insights into how practitioners perceive the value of Human’s responses,
prompting the need for further research.

Participant ID | Role | Years in Data Vis | Gender | Age range | Racial background |  Frequency of ChatGPT usage
---|---|---|---|---|---|---
P10 | Developer | 3-5 years | Male | 25-34 years old | Asian | Daily
P11 | Manager | 1-3 years | Male | 25-34 years old | Caucasian | Occasionally
P12 | Freelancer | 3-5 years | Male | 35-44 years old | Asian | Daily
P13 | Journalist | 1-3 years | Female | 18-24 years old | Hispanic | Weekly
P14 | Consultant | 3-5 years | Male | 25-34 years old | Caucasian | Daily
P15 | Product Designer | 3-5 years | Female | 25-34 years old | Caucasian | Weekly
P16 | Analyst | 5-10 years | Male | 35-44 years old | Caucasian, African American | Weekly
P17 | Scientist | 3-5 years | Female | 25-34 years old | Asian | Weekly
P18 | Scientist | 1-3 years | Female | 25-34 years old | African American | Weekly
P19 | Student | 5-10 years | Male | 25-34 years old | Caucasian | Only once or twice
P20 | Freelancer | ¿ 10 years | Male | 45-54 years old | Caucasian | Occasionally
P21 | Student | 3-5 years | Male | 25-34 years old | Asian | Daily

Table 1. Demographic and experience related information for participants in
user study.

##  4\. Perception of practitioners toward ChatGPT’s utility

Building upon our previous research, we conducted a comparative interview
study involving feedback sessions with ChatGPT and Human experts. We aimed to
gain a deeper understanding of how data visualization practitioners perceive
ChatGPT’s role as a design companion, uncovering their attitudes and
perceptions regarding ChatGPT’s design feedback.

###  4.1. Recruitment

We aimed to engage a diverse group of data visualization practitioners in our
study. We recruited participants through academic mailing lists targeting
students and scientists, as well as the Data Visualization Society’s Slack
channel (dat, [n. d.]). Inclusion criteria required proficiency in English,
experience in data visualization creation, and a willingness to share at least
one of their data visualizations. Initially, 41 individuals responded to the
recruitment survey, from which we selected 12 participants to take part in the
study.

###  4.2. Participants

The study involved 12 participants (P10 to P21), creating a diverse group with
varying professional backgrounds, experience levels, and familiarity with
ChatGPT (see Table 1). Their roles encompassed a broad spectrum, including
developers, managers, journalists, consultants, product designers, analysts,
scientists, and students. Professional experience ranged from less than one
year to over ten years in their respective fields. Furthermore, participants’
engagement with ChatGPT varied, with some being daily users while others
interacted with it only occasionally or as the need arose.

###  4.3. Tasks & Procedures

Before each study session, participants were asked to prepare a visualization
they were comfortable sharing, along with a list of relevant questions. These
sessions were conducted using Zoom. The entire study session took about 60
minutes. Participants were compensated with a $50 Amazon gift card.

Each interview began with a brief introduction by the moderator and was
divided into three segments: a visualization feedback session with ChatGPT, a
similar session with Human expert(s), and an open-ended interview. Six
participants initiated the process with the ChatGPT session, while the other
six commenced with the Human expert session. After each feedback session, we
administered a survey. In five sessions, two visualization experts provided
feedback, while in the remaining seven sessions, we had one expert present.
Both experts are current professors in the field of visualization.

During the ChatGPT feedback session, participants shared their screens with
the moderator and presented their visualizations, accompanied by a brief
explanation. The moderator introduced a predefined input format for ChatGPT,
employing a role-playing structure (Ihwan, [n. d.]). Participants were guided
to furnish details about the visualization’s chart type, textual description,
visual encodings, and any related questions. They input this information into
ChatGPT while screen-sharing. Following ChatGPT’s response, participants could
pose follow-up questions or request clarifications. Afterward, participants
received a survey via the Zoom chat to gather feedback on their ChatGPT
experience.

During the Human expert session, one or two visualization experts joined the
Zoom meeting. Participants shared their screens and presented their
visualization-related questions to these experts. The experts answered queries
and provided additional feedback or further insights. Subsequently, following
this session, the visualization experts left the meeting, and participants
were presented with a Zoom survey to gather feedback about their experience.

In the final segment of the study, open-ended interviews were conducted. In
these interviews, the moderator inquired about participants’ experiences in
both ChatGPT and Human feedback sessions, delving deeper into the rationale
behind their survey responses. Furthermore, participants were prompted to
share their perspectives on the possible integration of ChatGPT into their
data visualization workflow, its constraints, and the future possibilities it
might offer.

###  4.4. Data & Analysis Methods

The final dataset from our user studies comprises participant questions (see
Table 2) and responses, survey data, and transcripts from post interviews. Our
primary focus for analysis was on the interview transcripts. Initially, two
researchers independently examined three interview transcripts, identifying
interesting quotes and assigning meaningful themes. Once we established a high
degree of consistency between the researchers’ findings, one of them proceeded
to review the remaining transcripts. Ultimately, we compiled a categorized
list of quotes based on initial higher-level themes (e.g., Where Human experts
excel). To further refine our analysis, we revisited these quotes and divided
the themes into sub-level categories (e.g., Collaborative and natural
conversations).

###  4.5. Post-Session Interview Analysis Results

P ID | Chart Type | Initial Questions
---|---|---
P10 | parallel coordinate map | Could you suggest an alternative way to visualize this? How can I make this visualization more engaging?
P11 | scatterplot | Does it make more sense to compare X or to compare Y? Would there be a better visualization type to show ___?
P12 | scatterplot | How can I improve this visualization in order to satisfy the customer?
P13 | scatterplot | What is the best way to show ___? what do you think about the current color scheme?
P14 | diverging stacked bar chart | How can we ensure the user looks at ___? How can I put these four bars next to each other in one visualization?
P15 | bubble cluster | How would I enable more properties to be seen? How would I let the user encode the strength of their preference?
P16 | interactive map | Which parts of the data visualization do you think are successful? Are there any areas you believe could be improved for better clarity and impact?
P17 | sankey chart | Is there a way to ascertain what visualization is most effective for communicating the data? How could ___ be more clearly displayed?
P18 | wav file visualization | Would it be visually overwhelming to use ___? Is this an appropriate visualization for demonstrating ___?
P19 | color heatmaps | Would another color map be more suitable? Would you have any suggestions as to how to indicate ___?
P20 | beeswarm, line chart | Does the special encoding for ___ make intuitive sense? Is it clear that ___ represents ___?
P21 | comparative line graph | How can I decrease the number of colors used in this visualization? Would this data be better represented if I ___?

Table 2. Initial questions from participants. These initial questions are
altered to maintain the privacy of the participants. The original questions
provided more contextual information and specificity.

####  4.5.1. Participant Questions and Interaction Dynamics

Four of the participants (P10, P15, P16, and P18) shared interactive
visualizations. The other eight participants shared static visualizations. The
type of visualization, as well as the questions they initially brought, can be
seen in Table 2. To safeguard participant privacy, we refrain from disclosing
their visualizations or any elements of their questions that could potentially
reveal information about participants.

In the ChatGPT section of the interview, the majority of the participants
asked ChatGPT at least one follow-up question. The follow-up questions to
ChatGPT tended to be more specific and technical than the follow-up questions
asked to the Human experts. For example, P11 asked ChatGPT “What are the pros
and cons of having too many bubbles on the scatterplot from having a clear
comparison?”

On the other hand, the follow up questions to the Human experts tended to be
less structured and more conversational. The participant and expert had a
conversation and pulled out different aspects of the question along the way.
There was a lot of stopping to clarify what different elements of the
visualization meant and how they played into the visualization. As a result,
other factors not explicitly mentioned in the question often ended up getting
brought up in these sessions with the Human experts.

####  4.5.2. Post Survey Results

In Figure 4, we present the outcomes of the experience surveys conducted
following each feedback session. The overarching consensus among participants
was a strong preference for Human experts. This preference was underscored by
statistically significant distinctions, as determined through two-tailed Mann-
Whitney U tests. Specifically, participants expressed significantly higher
levels of satisfaction in interacting with Human experts, perceiving their
responses as notably more accurate, helpful, reliable, and adaptable to their
preferences.

Furthermore, respondents indicated that Human experts exhibited a
significantly deeper understanding of the context and requirements of their
queries. Human experts were also acknowledged for their expertise in data
visualization, along with their ability to offer actionable recommendations.
It is worth noting that no significant differences were observed in responses
to questions related to the clarity and conciseness of explanations, concerns
about potential biases, or perceived risks of receiving misleading
information.

Figure 4. User preference for Human expert vs. ChatGPT responses in feedback
sessions: The overarching pattern evident in these charts predominantly
indicates a user preference for Human experts over ChatGPT in the context of
feedback sessions.

This figure is about user preferences for the feedback sessions. 11 grouped
bar charts are presented, each illuminating user feedback collected through
Likert scale questions. These questions gauge user perceptions of the quality
of responses received from ChatGPT compared to those from Human experts during
feedback sessions. Examples of these questions encompass aspects such as
overall satisfaction with interaction and engagement with the feedback
provider, and the level of trust in the accuracy and correctness of
information provided concerning data visualizations. For every question, two
bars stand side by side: one representing the Human response and the other
representing ChatGPT’s response. These bars are further divided into five
distinct sections, reflecting the spectrum of user sentiment—ranging from
”strongly disagreed” to ”strongly agreed” with the respective question.

####  4.5.3. Where Human Experts Excel

All participants were generally more satisfied with Human experts. They
mentioned a variety of reasons for such opinions. The participants highlighted
several key themes in their responses, shedding light on the strengths of
Human experts in the context of data visualization guidance.

##### Bespoke and focused feedback of Human experts

First of all, they liked Human experts’ ability to offer tailored
recommendations that were closely aligned with the specific visuals in
question (P10, P11, P15, P17 - P21). P17 exemplified this sentiment by
stating, “[…] recommendations are a lot more tailored and specific in that
way, versus just like being tossed a list of potential tools we can look.”.
P11 further elaborated on this point by highlighting that Human experts can
envision various options and present the ones they deem most valuable. They
also noted that Human experts could effectively understand the context of
visuals by directly observing them (P11, P17 - P19). P19 expressed this
sentiment by stating, “I felt the feedback was obviously more grounded to the
visualization at hand. Because again, they could see it.” Similarly, P18
commented, “As he’s looking at the pictures, he knows the information I had
access to and how that translates.” Furthermore, participants acknowledged
that Human experts excelled in staying focused on the specific problem at
hand. For instance, P15 articulated, “[the expert] was able to talk about
improvements within scope […] without redesigning it entirely.” Similarly, P10
commented, “The ones that [the expert] gave were within the scope of refining
existing visualizations.”

##### Collaborative and natural conversations

Participants consistently expressed the value of more fluid conversations when
interacting with Human experts (P10, P11, P15, P17, P19, P20). They
highlighted that Human responses felt more free-flowing, contributing to a
sense of genuine interaction (P10, P17). Participants also noted that Human-
expert interactions provided a collaborative and interactive experience (P11,
P15, P19, P20). For instance, P11 highlighted an efficient turn-taking dynamic
by saying, “the ability to ask follow-up questions and clarify I think was
easier to do within the Human interaction context.” Others similarly
emphasized the convenience of seeking clarifications without the need for
precise phrasing or additional context adjustments (P11, P15).

##### Enriched insights through lived experience

Participants felt that Human experts’ responses were often rooted in their
experience and education (P16). They valued the ability of Human experts to
explain the underlying rationale, promoting a more actionable approach to
implementing the suggestions (P10). Furthermore, participants appreciated the
Human experts’ ability to think beyond the obvious; as P13 noted, “I feel like
I was definitely more satisfied with the Human experts, especially because
they thought outside of the box without me asking … they just started thinking
so many things beyond anything I could have think of.” Moreover, participants
conveyed that interactions with Human experts felt personal and supportive.
One participant reflected, “they were really walking me through the different
things […] help you grow as a professional […] really pushing me.”, while
another participant recognized the proactiveness of Human experts by saying
“even if I didn’t ask question, they kind of brought it up.”

####  4.5.4. Where ChatGPT excels

Participants identified the strengths of ChatGPT in contrast to Human experts.

##### Brainstorming and ideation

ChatGPT’s role as a creative catalyst emerged prominently during participant
discussions, underlining its capacity to spark innovative ideas (P11, P15,
P18). Participants like P18 found unexpected and impressive suggestions,
stating, “It did have some cool ideas, I didn’t expect this from ChatGPT. So
pretty. Pretty good.”, while P11 commented, “I think it’s really interesting
to try and kind of break outside the box and get more creative… GPT has the
potential to disrupt that workflow in a positive way.” Likewise, P15 also
acknowledged that the tool may not always provide precise solutions but rather
serve as a springboard for creative exploration. P15 contrasts this to Human
experts by saying, “[expert] feedback was less brainstorming and more actual
critique.”

##### Broad knowledge base

In a similar vein to its brainstorming capability, participants also
highlighted ChatGPT’s capacity to serve as a vast repository of knowledge,
offering a wide array of ideas (P10, P11, P15). P10 stated, “it really does
kind of feel like a really fancy kind of search engine, where it is kinda, I
give it a problem. And it’s like, oh, like, this might be a solution.”, while
P15 said, “great place to go to start […] researching or come up with ideas
you hadn’t thought of.” P11 similarly emphasized that ChatGPT excels at
showcasing a range of data visualization options, making it a valuable
resource during the initial stages of projects. P11 elaborated on the tool’s
potential, envisioning a scenario where metadata about a dataset could be fed
to ChatGPT to receive chart-type recommendations. However, P10 emphasized that
a critical eye is needed to discern the relevant and valuable insights from
its outputs; “I asked to give 20 ideas and I’m […] experienced enough […] to
[…] know that, like, 16 of them are just nonsense. The other four things I
might not have thought of.”

##### Time saved via rapid understanding and response

Participants appreciated ChatGPT’s ability to quickly grasp and provide
information (P10, P11, P13). P10 said ChatGPT was able to quickly grasp
unfamiliar concepts auxiliary to visualization (e.g., explaining Wordle to
ChatGPT), while P11 and P13 felt ChatGPT provided faster and more
comprehensive lists of options, especially for alternative chart types, when
compared to the responses from Human experts. On a related note, P11 said that
Human interactions can sometimes involve a lengthier back-and-forth process to
arrive at the right question.

##### Soft attitude and neutral perspective

Other participants noted ChatGPT’s behavioral traits. For example, P19 pointed
out that ChatGPT often assumed a gentle and agreeable attitude by
acknowledging concerns without offering specific directions, which led to a
perception of excessive alignment. P19 also noted that ChatGPT demonstrated a
neutral stance, presenting a variety of viewpoints without favoring any
particular side. This contrasted with Human experts who might convey their own
biases, thereby making ChatGPT’s approach seem more balanced.

####  4.5.5. Where ChatGPT falls short

Participants also discussed ChatGPT’s several shortcomings in comparison to
Human experts.

##### Lack of depth in responses

Participant feedback indicated a notable lack of depth in ChatGPT’s responses
(P13, P15, P19). For instance, P19 expressed that ChatGPT’s advice seemed to
lack actionable insights and depth that Human experts possess. This sentiment
was echoed by P15, who noted that while ChatGPT’s answers were broad in
knowledge, they lacked depth. P15 further elaborated that ChatGPT often
suggested radical changes (i.e., diverse ideas), while Human experts provided
improvements that were more aligned with refining existing visualizations. P19
similarly commented that ChatGPT’s ideas were not considering the specific
visualization. Participants acknowledged this contrast as a trade-off (P15,
P17), emphasizing that the desired level of response detail and granularity
from either ChatGPT or Human experts depends on user needs.

##### Generic advice, misalignment, and lack of critical thinking

Participants expressed that ChatGPT’s feedback often felt generic and lacked
contextual depth (P12, P16, P20). P20 expressed that ChatGPT’s “knowledge felt
[…] not necessarily backed up by experience.” P16 and P12 similarly noted that
ChatGPT’s responses were not necessarily wrong but were overly generic,
failing to consider the actual outcomes of the provided recommendations. P16
further commented that they “felt the areas of improvement were good
suggestions. But it didn’t feel like you have to do this.”

Several participants highlighted instances of misalignment and a lack of
specificity in ChatGPT’s recommendations (P11, P16, P19). P19 pointed out an
example where ChatGPT suggested color maps that might create confusion rather
than improve visualizations. This lack of alignment was further emphasized by
P11, who noted a communication hurdle hindering the establishment of a common
understanding and precise feedback.

Participants highlighted concerns related to critical thinking (P13, P16). P16
noted, “whereas the ChatGPT felt more like it was telling me what I wanted to
hear if that makes sense.” P16 further elaborated that ChatGPT’s responses
sometimes resembled textbook information rather than offering insightful
suggestions on visualization. P13 shared a similar sentiment, stating, “And I
feel like the issue with ChatGPT typically is that it only responds to your
question. So if you cannot think of the question, it will not give you the
answer.”

##### Efforts required for fluid conversation

Participants highlighted the challenges in having a fluid and interactive
conversation with ChatGPT (P10, P13, P16, P17, P20). For instance, P10
explained, “… kind of hoping that one of them would be what I wanted.”. P17
expressed a similar sentiment, stating, “even though it’s still
conversational, but it still kind of feels a little bit static sometimes where
you’re just typing text into a text box.” P16 further emphasized the
difference in interaction dynamics, stating, “I really liked listening to the
Human experts go back and forth, and talk about different aspects… whereas
ChatGPT just fed that to me, and that was it.” P13 noted that ChatGPT’s
responses often required iterations and follow-ups, P10 found that ChatGPT’s
insights became more specific and interesting when more constraints were
provided.

####  4.5.6. Opinions on Trust Reliability with ChatGPT

Participants had varied opinions on the trustworthiness and reliability of
ChatGPT’s responses, with some expressing low trust (P10, 18, P20). P20
mentioned the limitations in ChatGPT’s understanding of visuals, which relied
heavily on the accuracy of participants’ descriptions. P18 pointed out the
potential for fabricated information due to the algorithmic nature of the
responses. P10 echoed this sentiment, indicating that ChatGPT often sounded
too confident and authoritative, which could mislead users into placing
unwarranted trust. On the other hand, P17 said the two are relatively the
same, highlighting that the key distinction lies in how information is
delivered.

Several other participants pointed to the need for due diligence in evaluating
ChatGPT’s suggestions, especially when the user lacks domain expertise (P10,
P15). For instance, P10 commented: “If I was in a field that I wasn’t familiar
with, I think it’d be really easy to get fooled by it.” P15 drew a parallel
with the early inception of Wikipedia, stating, “ it was like early days of
Wikipedia, I was taught like, never cite Wikipedia, it could be wrong. And I
think the same thing is true of something like a large language model.”

They also made comparisons to Human experts in terms of trust and authority
(P13, P16, P18, P21). They explained how the credentials of Human respondents
influenced their perception (P18, P21), e.g., “Well, the Human respondent is a
professor in visualization.”—P18. Others perceived Human experts as sounding
knowledgeable and educational (P13, P16). In contrast to these, participants
also shared their perceptions of expertise and self-awareness in AI. P16 said,
“Didn’t feel like that the ChatGPT response was incorrect … trusted it, but
not nearly as much as the Humans.”, while P20 expressed, “ChatGPT doesn’t know
when it’s wrong.”

####  4.5.7. Limitations and Opportunities with ChatGPT

Participants expressed an optimistic outlook regarding the future potential of
ChatGPT, despite acknowledging its current challenges (P10, P12, P15, P16).
They discussed aspects they would like to see improved in ChatGPT in the
future.

##### Ability to convey complex visualizations

Participants emphasized the significance of enabling ChatGPT to comprehend and
interpret complex visualizations, highlighting the limitations of text-based
communication when dealing with intricate design problems (P10, P11, P13, P15
- 21). P15 acknowledged the inherent loss of information when translating
between visual and written mediums, while P10 and P11 conveyed the complexity
of describing interactive dashboards and multimedia visualizations via text.
P10 also mentioned video recordings that could convey the interactivity of
designs.

##### Ability to generate visualizations based on feedback

Several participants pondered the prospect of ChatGPT being able to not only
analyze input information but also generate relevant visualizations (P18, P19,
P21). For example, P18 proposed the generation or modification of images based
on textual descriptions. P19 noted the convenience of sharing underlying code
to accompany visual content, emphasizing its value in effectively
communicating complex ideas instead of relying solely on text descriptions.
Similarly, P21 discussed the concept of inputting data and posing questions to
explore the space of chart design. P13 envisioned the integration of ChatGPT
within existing tools, enhancing workflow efficiency and facilitating a more
cohesive user experience.

##### Facilitating Fluid and Truthful Conversations

Participants expressed the need for ChatGPT to initiate and support more fluid
conversations to enhance its usability and value (P11, P16, P21). P16
highlighted the need for AI to prompt users for further details about the
visualization; “I would really like it if it were to ask me questions to kind
of gauge my thought process.” Similarly, P11 noted the value of follow-up
questions in driving valuable insights, while “it was harder to envision what
a follow-up question to that system would be.” On the other hand, P21 raised
concerns about the possibility of ChatGPT providing false or hallucinated
information.

##### The indispensable role of Human feedback

The consensus among participants highlighted the enduring value of Human
feedback in evaluating and improving visualizations (P11, P16, P21). P16
commented, “I think there’s always, at least in my lifetime, going to be a
need for Human interpretation. And the Human experts’ sessions, for me,
reinforce that they offer a lot of really good insight.” P11 further expressed
that despite ChatGPT’s capabilities, the feedback from actual users remains
incomparable, as it encompasses a deeper level of insight into user
experience. Additionally, P11 discussed ideas of experimenting with different
personas for hypothetical user interactions.

##### Issues with privacy and sensitive data

Participants in the interviews expressed significant concerns regarding the
privacy and handling of sensitive data by ChatGPT (P15, P16, P20, P21). They
emphasized that sharing visuals containing private or sensitive information
would be uncomfortable and unlikely unless the data was from a public source
or there was a guarantee that it would not be shared publicly. P15 shared an
illustrative example, stating, “I met a company that I had to sign an NDA
with, and we are not allowed to use ChatGPT with anything that might be
considered confidential information because that information goes onto OpenAI
servers and will live there for eternity.”

##  5\. Discussions

###  5.1. Lessons learned

Human experts were generally perceived as more effective in imparting data
visualization design knowledge compared to ChatGPT. Several factors
contributed to this perception, including Human experts’ ability to engage in
more fluid conversations, their adeptness in turn-tracking, and their capacity
to offer highly contextualized suggestions. Additionally, participants’ trust
in Human experts was influenced by their professional backgrounds. However,
not all feedback regarding ChatGPT was negative. Participants acknowledged its
utility as a brainstorming tool, primarily due to its extensive knowledge
base, which enabled it to generate novel ideas. Furthermore, participants
appreciated ChatGPT’s prompt responses. Nonetheless, concerns were raised
about ChatGPT, particularly regarding the need for some level of expertise to
evaluate the validity and appropriateness of its suggestions and responses
that are at the risk of being hallucinated. Overall, participants expressed
optimism about ChatGPT’s potential, believing that it will continue to improve
and become more valuable in the future.

###  5.2. Comparisons to Human Responses from Visguides

While we did not formally analyze responses in the feedback study, ChatGPT
responses from the study were similar to ChatGPT’s responses in Visguides. In
both instances, ChatGPT provided broad responses and listed ideas. For
responses from Human experts, they differ more from the Human responses in
Visguides. One reason for this is that any member of the Visguide community
can answer questions on the forum, whereas in the feedback sessions, the
questions were answered by experienced visualization experts. These sessions
also happened in real-time, which led to the Human experts not missing parts
of participant questions and led to them being especially attentive and
thorough. The Human experts in the feedback session also sometimes
supplemented their verbal responses by sharing supporting links and related
articles over the Zoom chat. Overall, the responses from Human experts in the
feedback sessions were similar to the high-scoring Human responses in
Visguides.

###  5.3. Improving perceptual and cognitive abilities

One significant challenge encountered while interacting with ChatGPT was its
lack of nuanced image understanding. Consequently, despite the effectiveness
of our template for describing chart types and visual encodings (P10), ChatGPT
struggled to comprehend the intricate visual patterns depicted in the charts.
We noticed that providing additional descriptions of these perceptual patterns
improved ChatGPT’s performance in the VisGuides analysis, although some
participants expressed concerns about the need to manually input such
descriptions. Currently, platforms like Bard and Bing support image input.
However, existing vision systems are primarily trained on natural images and
are ill-equipped to handle synthetic images such as graphic designs and data
visualizations (Bylinskii et al., 2017; Hoque et al., 2022). In a brief test
where we posed ten questions from VisGuides to Bard, their responses did not
seem to surpass ChatGPT’s performance. For instance, Bard consistently
recommended adding elements that were already clearly in the visualization,
such as legends, color encodings, and titles. Additionally, the
recommendations provided by Bard had less breadth to them and were oftentimes
not applicable, such as recommending adding a trend line to a Gantt Chart.

###  5.4. Ability to proactively engage in discussions

Another major challenge reported by participants was the absence of fluid
conversation and turn-tracking. ChatGPT functions as a chatbot that responds
solely to user instructions, and it is also trained to align with users’
intentions (Ouyang et al., 2022). This design makes ChatGPT more passive and
affirmative in comparison to Human experts, who frequently pose clarifying
questions and may even offer unsolicited advice. There exists a potential
trade-off between these two communication styles. On the one hand, there’s a
concern about unsafe or misalignment scenarios where AI chatbots might have
the ability to intervene and provide malicious or irrelevant feedback. On the
other hand, there are situations where practitioners struggle to formulate
questions because they may not yet discern what’s wrong with their
visualizations. Our participants often found themselves in this latter
scenario, where they did not know what kind of follow-up questions to ask. It
would be valuable to conduct further investigations into passive and active
Human-AI alignments within this problem context. For instance, a middle-ground
approach could involve ChatGPT suggesting potential follow-up questions
similar to the current capabilities of Bing.

###  5.5. Integrating into a practitioner’s workflow

Several participants mentioned that they primarily use ChatGPT for tasks
related to writing or debugging code in order to generate visualizations (P10,
P12, P13, P16, P18). The robust code generation and comprehension capabilities
of large language models (LLMs), such as Github Copilot (Git, [n. d.]), are
relatively well-known. On the other hand, data visualization practice rests on
two pillars: design and implementation. Our study sheds light on ChatGPT’s
current state in terms of design knowledge beyond implementation knowledge.
However, existing data visualization tools often fall short of conveying
effective data visualization knowledge to designers. As P13 suggests, the
integration of a ChatGPT-like assistant within current tools would be highly
beneficial. Such a design assistant could assist in suggesting appropriate
chart types (P11, P21) or providing rationales or critiques for generated
visualizations.

###  5.6. ChatGPT for Design Knowledge Education

Some participants noted ChatGPT’s potential for education (P20, P21), such as
assisting with understanding unfamiliar charts or aiding in writing
educational blogs on data visualization. A recent paper identifies challenges
in visualization education (Bach et al., 2023), highlighting AI as a valuable
tool for formal and informal learning contexts. However, concerns include the
consistency and quality of AI-generated content, as differing responses to
students could be problematic. There’s also the risk of students over-relying
on AI instead of engaging critically with visualizations, e.g., through peer
feedback. Future AI systems in education may act as personalized tutors,
tracking progress, guiding learning goals, and fostering both comprehension
and generation of visualizations.

##  6\. Conclusion

In this study, we explored the potential of ChatGPT as a design companion,
focusing on its capacity to provide valuable visualization insights to
practitioners. Our findings reveal that ChatGPT, despite its absence of visual
perception, capitalizes on a vast knowledge repository to generate diverse and
imaginative suggestions. While limitations like the depth of its responses and
interaction dynamics exist, participants expressed optimism about its future
utility. Looking ahead, our future research endeavors will involve
investigating state-of-the-art chart understanding vision models to develop a
multimodal conversational agent. Additionally, we plan to delve into the
design space of integrating ChatGPT into existing data visualization tools,
further enhancing its practical applicability.

## References

  * (1)
  * dat ([n. d.]) [n. d.].  _Data Visualization Society_.  Accessed: March 28, 2023.
  * dvs ([n. d.]) [n. d.].  _Data Visualization Society Surveys_.  https://www.datavisualizationsociety.org/survey-history Accessed on Sep 9, 2023.
  * Git ([n. d.]) [n. d.].  GitHub Copilot.  https://github.com/features/copilot.  Accessed on Sep 9, 2023.
  * Bach et al. (2023) Benjamin Bach, Mandy Keck, Fateme Rajabiyazdi, Tatiana Losev, Isabel Meirelles, Jason Dykes, Robert S Laramee, Mashael AlKadi, Christina Stoiber, Samuel Huron, et al. 2023\.  Challenges and Opportunities in Data Visualization Education: A Call to Action.  _arXiv preprint arXiv:2308.07703_ (2023).
  * Baidoo-Anu and Owusu Ansah (2023) David Baidoo-Anu and Leticia Owusu Ansah. 2023.  Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning.  _Available at SSRN 4337484_ (2023).
  * Bako et al. (2023) Hannah K. Bako, Xinyi Liu, Leilani Battle, and Zhicheng Liu. 2023.  Understanding how Designers Find and Use Data Visualization Examples.  _IEEE Transactions on Visualization and Computer Graphics_ 29, 1 (2023), 1048–1058.  https://doi.org/10.1109/TVCG.2022.3209490
  * Bartolomeo et al. (2023) Sara Di Bartolomeo, Giorgio Severi, Victor Schetinger, and Cody Dunne. 2023.  Ask and You Shall Receive (a Graph Drawing): Testing ChatGPT’s Potential to Apply Graph Layout Algorithms. In _EuroVis 2023 - Short Papers_ , Thomas Hoellt, Wolfgang Aigner, and Bei Wang (Eds.). The Eurographics Association.  https://doi.org/10.2312/evs.20231047
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 610–623.  https://doi.org/10.1145/3442188.3445922
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).  https://crfm.stanford.edu/assets/report.pdf
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Bylinskii et al. (2017) Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. 2017.  Learning Visual Importance for Graphic Designs and Data Visualizations. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_ (Québec City, QC, Canada) _(UIST ’17)_. Association for Computing Machinery, New York, NY, USA, 57–69.  https://doi.org/10.1145/3126594.3126653
  * Chalkidis et al. (2019) Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019\.  Neural Legal Judgment Prediction in English. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 4317–4323.  https://doi.org/10.18653/v1/P19-1424
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023\.  A survey on evaluation of large language models.  _arXiv preprint arXiv:2307.03109_ (2023).
  * Choi et al. (2023b) Jinhan Choi, Changhoon Oh, Yea-Seul Kim, and Nam Wook Kim. 2023b.  VisLab: Enabling Visualization Designers to Gather Empirically Informed Design Feedback. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 813, 18 pages.  https://doi.org/10.1145/3544548.3581132
  * Choi et al. (2021) Jinhan Choi, Changhoon Oh, Bongwon Suh, and Nam Wook Kim. 2021\.  Toward a Unified Framework for Visualization Design Guidelines. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI EA ’21)_. Association for Computing Machinery, New York, NY, USA, Article 240, 7 pages.  https://doi.org/10.1145/3411763.3451702
  * Choi et al. (2023a) Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. 2023a.  Chatgpt goes to law school.  _Available at SSRN_ (2023).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dibia (2023) Victor Dibia. 2023\.  LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_. Association for Computational Linguistics, Toronto, Canada, 113–126.  https://doi.org/10.18653/v1/2023.acl-demo.11
  * Diehl et al. (2018) Alexandra Diehl, Alfie Abdul-Rahman, Mennatallah El-Assady, Benjamin Bach, Daniel Keim, and Min Chen. 2018.  VisGuides: A Forum for Discussing Visualization Guidelines. In _EuroVis 2018 - Short Papers_ , Jimmy Johansson, Filip Sadlo, and Tobias Schreck (Eds.). The Eurographics Association.  https://doi.org/10.2312/eurovisshort.20181079
  * Diehl et al. (2021) Alexandra Diehl, Elif E. Firat, Thomas Torsney-Weir, Alfie Abdul-Rahman, Benjamin Bach, Robert Laramee, Renato Pajarola, and Min Chen. 2021.  VisGuided: A Community-driven Approach for Education in Visualization. In _Eurographics 2021 \- Education Papers_ , Beatriz Sousa Santos and Gitta Domik (Eds.). The Eurographics Association.  https://doi.org/10.2312/eged.20211003
  * Esteves and Neves (2022) Salomé Esteves and Marco Neves. 2022.  “I learned it on the job” Becoming a Data Visualization professional in news media.  _Information Design Journal_ 27, 3 (2022), 309–319.  https://doi.org/10.1075/idj.22004.est
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020.  RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, 3356–3369.  https://doi.org/10.18653/v1/2020.findings-emnlp.301
  * Gilson et al. (2023) Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023\.  How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment.  _JMIR Medical Education_ 9, 1 (2023), e45312.  https://doi.org/10.1101/2022.12.23.22283901
  * Hoque et al. (2022) E. Hoque, P. Kavehzadeh, and A. Masry. 2022.  Chart Question Answering: State of the Art and Future Directions.  _Computer Graphics Forum_ 41, 3 (2022), 555–572.  https://doi.org/10.1111/cgf.14573 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14573
  * Hutchinson et al. (2023) Maeve Hutchinson, Aidan Slingsby, Radu Jianu, and Pranava Madhyastha. 2023.  Towards Visualisation Specifications from Multilingual Natural Language Queries using Large Language Models.  (2023).  https://doi.org/10.2312/evp.20231072
  * Ihwan ([n. d.]) Aris Ihwan. [n. d.].  Role Prompting.  https://www.linkedin.com/pulse/role-prompting-aris-ihwan/?trk=pulse-article_more-articles_related-content-card.  Accessed on Sep 8, 2023.
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (2023), 102274.  https://doi.org/10.1016/j.lindif.2023.102274
  * Lahat et al. (2023) Adi Lahat, Eyal Shachar, Benjamin Avidan, Zina Shatz, Benjamin S. Glicksberg, and Eyal Klang. 2023.  Evaluating the use of large language model in identifying top research questions in gastroenterology.  _Scientific Reports_ 13, 1 (March 2023), 4164\.  https://doi.org/10.1038/s41598-023-31412-2
  * Lee et al. (2023) Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023\.  Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine.  _New England Journal of Medicine_ 388, 13 (2023), 1233–1239.  https://doi.org/10.1056/NEJMsr2214184
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022\.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ (2022).
  * Luther et al. (2015) Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P. Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P. Dow. 2015\.  Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. In _Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing_ (Vancouver, BC, Canada) _(CSCW ’15)_. Association for Computing Machinery, New York, NY, USA, 473–485.  https://doi.org/10.1145/2675133.2675283
  * Mollman (2023) Steve Mollman. 2023\.  ChatGPT passed a Wharton MBA exam and it’s still in its infancy. One professor is sounding the alarm.  https://fortune.com/2023/01/21/chatgpt-passed-wharton-mba-exam-one-professor-is-sounding-alarm-artificial-intelligence/.  Accessed on Sep 7, 2023.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.  StereoSet: Measuring stereotypical bias in pretrained language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, 5356–5371.  https://doi.org/10.18653/v1/2021.acl-long.416
  * Okerlund et al. (2022) Johanna Okerlund, Evan Klasky, Aditya Middha, Sujin Kim, Hannah Rosenfeld, Molly Kleinman, and Shobita Parthasarathy. 2022.  What’s in the Chatterbox? Large Language Models, Why They Matter, and What We Should Do About Them.
  * OpenAI ([n. d.]) OpenAI. [n. d.].  ChatGPT.  https://openai.com/chatgpt.  Accessed on Sep 7, 2023.
  * OpenAI (2023) R OpenAI. 2023.  GPT-4 technical report.  _arXiv_ (2023), 2303–08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Paik (2023) Soonk Paik. 2023\.  How I Created a Data Visualization With Zero Coding Skills, Thanks to ChatGPT.  _Nightingale magazine by the Data Visualization Society_ (4 April 2023).  https://nightingaledvs.com/data-visualization-using-chatgpt-to-code/
  * Pan (2023) Jie Pan. 2023.  Large language model for molecular chemistry.  _Nature Computational Science_ 3, 1 (2023), 5–5.  https://doi.org/10.1038/s43588-023-00399-1
  * Parsons (2022) P. Parsons. 2022\.  Understanding Data Visualization Design Practice.  _IEEE Transactions on Visualization; Computer Graphics_ 28, 01 (jan 2022), 665–675.  https://doi.org/10.1109/TVCG.2021.3114959
  * Parsons et al. (2020) Paul Parsons, Colin M. Gray, Ali Baigelenov, and Ian Carr. 2020\.  Design Judgment in Data Visualization Practice. In _2020 IEEE Visualization Conference (VIS)_. 176–180.  https://doi.org/10.1109/VIS47514.2020.00042
  * Parsons and Shukla (2020) Paul Parsons and Prakash Shukla. 2020.  Data Visualization Practitioners’ Perspectives on Chartjunk. In _2020 IEEE Visualization Conference (VIS)_. 211–215.  https://doi.org/10.1109/VIS47514.2020.00049
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023\.  Is ChatGPT a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_ abs/2302.06476 (2023).
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving language understanding by generative pre-training.  (2018).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Schetinger et al. (2023) V. Schetinger, S. Di Bartolomeo, M. El-Assady, A. McNutt, M. Miller, J. P. A. Passos, and J. L. Adams. 2023.  Doom or Deliciousness: Challenges and Opportunities for Visualization in the Age of Generative Models.  _Computer Graphics Forum_ 42, 3 (2023), 423–435.  https://doi.org/10.1111/cgf.14841 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14841
  * Tan and Celis (2019) Yi Chern Tan and L. Elisa Celis. 2019.  _Assessing Social and Intersectional Biases in Contextualized Word Representations_.  Curran Associates Inc., Red Hook, NY, USA.
  * Tlili et al. (2023) Ahmed Tlili, Boulus Shehata, Michael Agyemang Adarkwah, Aras Bozkurt, Daniel T. Hickey, Ronghuai Huang, and Brighter Agyemang. 2023\.  What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education.  _Smart Learning Environments_ 10, 1 (2023), 15\.  https://doi.org/10.1186/s40561-023-00237-x
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022\.  Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI EA ’22)_. Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages.  https://doi.org/10.1145/3491101.3519665
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022\.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_ (2022).
  * Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023\.  Multimodal chain-of-thought reasoning in language models.  _arXiv preprint arXiv:2302.00923_ (2023).
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023\.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).


# A Proper Scoring Rule for Validation of Competing Risks Models

Zoe Guan1

(1 Department of Epidemiology and Biostatistics, Memorial Sloan Kettering
Cancer Center, New York, NY 10017
)

###### Abstract

Scoring rules are used to evaluate the quality of predictions that take the
form of probability distributions. A scoring rule is strictly proper if its
expected value is uniquely minimized by the true probability distribution. One
of the most well-known and widely used strictly proper scoring rules is the
logarithmic scoring rule. We propose a version of the logarithmic scoring rule
for competing risks data and show that it remains strictly proper under non-
informative censoring.

##  1 Introduction

A probabilistic forecast is a prediction that specifies a probability
distribution over the set of possible outcomes. The quality of probabilistic
forecasts is typically assessed using scoring rules (for an overview, see
Gneiting and Raftery (2007), Chapter 10 of Parmigiani and Inoue (2009), and
Dawid and Musio (2014)). Given a set of outcomes $\mathcal{X}$ and a family of
probability measures $\mathcal{P}$ over $\mathcal{X}$, a scoring rule is a
loss function
$s:\mathcal{X}\times\mathcal{P}\to\mathds{R}\cup[-\infty,\infty]$ that assigns
a number $s(x,Q)$ to each combination of $x\in\mathcal{X}$ and
$Q\in\mathcal{P}$.

A rational forecaster who believes that the true distribution is
$P\in\mathcal{P}$ will report a forecast $Q\in\mathcal{P}$ that minimizes the
expected score under $P$,

$\displaystyle S(P,Q)\coloneqq E_{P}[s(X,Q)].$ (1)

A scoring rule is proper if $S(P,P)\leq S(P,Q)$ for all $P,Q\in\mathcal{P}$,
and it is strictly proper if $S(P,P)<S(P,Q)$ for $Q\neq P$. Strictly proper
scoring rules are desirable because they encourage honesty (i.e. they
encourage forecasters to report their true beliefs) and reward accuracy
(Winkler, 1994).

One of the most well-known and widely used strictly proper scoring rules is
the logarithmic scoring rule proposed by Good (1952):

$\displaystyle s(x,Q)=-\log(q(x))$ (2)

where $q$ is the probability density or mass function corresponding to $Q$.
There are many theoretical and empirical arguments supporting the use of the
logarithmic scoring rule in various prediction problems (Winkler, 1969;
Phillips and Edwards, 1966; Benedetti, 2010). Besides being strictly proper,
the logarithmic scoring rule is also local, which means that it depends only
on the predicted probabilities for observed events and does not use the
predicted probabilities for unobserved events. Moreover, the logarithmic
scoring rule discourages the forecaster from assigning extreme probabilities
to very rare or very frequent events Benedetti (2010), which might be
desirable in settings where overconfident predictions have serious
consequences.

In survival or failure time analysis, interest lies in predicting the time
until the occurrence of a specific event, which might not be fully observed
due to censoring. Dawid and Musio (2014) described a proper scoring rule,
called the survival score, for the classical survival analysis setting with a
single event type and non-informative right censoring. The survival score
gives rise to a variant of the logarithmic scoring rule as a special case. In
this paper, we consider a competing risks setting where there are multiple
mutually exclusive event types. We propose a logarithmic scoring rule for this
setting and show that it remains strictly proper under non-informative right
censoring.

##  2 Competing Risks Notation

Suppose there are $M$ competing causes of failure. Let $T$ be the time to
failure and let $J\in\\{1,\dots,M\\}$ denote the cause of failure. $T$ is
potentially subject to right censoring. Let $C$ be the censoring time, which
is assumed to be independent of $(T,J)$. We observe $Y=\min(T,C)$ and
$\Delta_{j}=I[T\leq C,J=j]$, the indicator for whether failure type $j$ is
observed, for $j=1,\dots,M$. We assume the times are discrete, but similar
results apply for continuous-time data, with minor notation changes.

Let $Q$ be a probability distribution for $(T,J)$ and $G$ a probability
distribution for $C$. Let $f_{j,Q}(t)=Q(T=t,J=j)$, $F_{j,Q}(t)=Q(T\leq
t,J=j)$, and $F_{Q}(t)=\sum\limits_{j=1}^{M}F_{j,Q}(t)$. These functions can
depend on covariates, but for simplicity we omit them from the notation. The
joint probability mass function for $(Y,\Delta_{1},\dots,\Delta_{M})$ is

$\displaystyle\pi_{Q,G}(Y=y,\Delta_{1}=\delta_{1},\dots,\Delta_{M}=\delta_{M})=\prod_{j=1}^{M}f_{j,Q}(y)^{\delta_{j}}(1-F_{Q}(y))^{1-\delta}G(C\geq
y)^{\delta}G(C=y)^{1-\delta}$ (3)

where $\delta=\sum\limits_{j=1}^{m}\delta_{j}$.

##  3 Scoring Rule and Proof of Propriety

We define a logarithmic scoring rule that evaluates a probability distribution
for $(T,J)$ against the observed data $(y,\delta_{1},\dots,\delta_{M})$. We
show that this scoring rule is strictly proper.

######  Theorem 1.

Given a probability distribution $Q$ for $(T,J)$, define

$\displaystyle
s((y,\delta_{1},\dots,\delta_{M}),Q)\coloneqq-\sum_{j=1}^{M}\delta_{j}\log(f_{j,Q}(y))-(1-\delta)\log(1-F_{Q}(y)).$
(4)

This is a strictly proper scoring rule for the distribution of $(T,J)$.

When $M=1$, (4) is equivalent to a special case of the survival score from
Section 3.5 of Dawid and Musio (2014) that is obtained by setting
$\psi(\lambda)=\lambda\log{\lambda}$.

Proof of Theorem 1.

Let $P$ and $Q$ be probability distributions for $(T,J)$. Let $G$ be a
probability distribution for $C$. Define

$\displaystyle S_{G}(P,Q)\coloneqq
E_{P,G}[s((Y,\Delta_{1},\dots,\Delta_{M}),Q)].$ (5)

We will show that for any choice of $G$, $S_{G}(P,Q)$ is uniquely minimized by
$Q=P$.

$\displaystyle S_{G}(P,Q)-S_{G}(P,P)$
$\displaystyle=\sum_{y,\delta_{1},\dots,\delta_{M}}\pi_{P,G}(y,\delta_{1},\dots,\delta_{M})\left(s((y,\delta_{1},\dots,\delta_{M}),Q)-s((y,\delta_{1},\dots,\delta_{M}),P)\right)$
$\displaystyle=\sum_{y,\delta_{1},\dots,\delta_{M}}\pi_{P,G}(y,\delta_{1},\dots,\delta_{M})\log\left(\frac{\prod_{j=1}^{M}f_{j,P}(y)^{\delta_{j}}(1-F_{P}(y))^{1-\delta}}{\prod_{j=1}^{M}f_{j,Q}(y)^{\delta_{j}}(1-F_{Q}(y))^{1-\delta}}\right)$
$\displaystyle=\sum_{y,\delta_{1},\dots,\delta_{M}}\pi_{P,G}(y,\delta_{1},\dots,\delta_{M})\log\left(\frac{\prod_{j=1}^{M}f_{j,P}(y)^{\delta_{j}}(1-F_{P}(y))^{1-\delta}G(C\geq
y)^{\delta}G(C=y)^{1-\delta}}{\prod_{j=1}^{M}f_{j,Q}(y)^{\delta_{j}}(1-F_{Q}(y))^{1-\delta}G(C\geq
y)^{\delta}G(C=y)^{1-\delta}}\right)$
$\displaystyle=\sum_{y,\delta_{1},\dots,\delta_{M}}\pi_{P,G}(y,\delta_{1},\dots,\delta_{M})\log\left(\frac{\pi_{P,G}(y,\delta_{1},\dots,\delta_{M})}{\pi_{Q,G}(y,\delta_{1},\dots,\delta_{M})}\right)$
$\displaystyle=D_{KL}(\pi_{P,G}||\pi_{Q,G})$ where $D_{KL}(p||q)$ denotes the
Kullback-Leibler divergence from $p$ to $q$

Kullback-Leibler divergence is non-negative and $D_{KL}(p||q)=0$ if and only
if $p(x)=q(x)$ for all $x$ (MacKay, 2003), so $S_{G}(P,Q)$ is uniquely
minimized by $Q=P$.

∎

## Acknowledgements

I would like to thank Giovanni Parmigiani for pointing me to relevant
literature and providing helpful suggestions.

## References

  * Benedetti (2010) Riccardo Benedetti.  Scoring rules for forecast verification.  _Monthly Weather Review_ , 138(1):203–211, 2010\.
  * Dawid and Musio (2014) Alexander Philip Dawid and Monica Musio.  Theory and applications of proper scoring rules.  _Metron_ , 72(2):169–183, 2014.
  * Gneiting and Raftery (2007) Tilmann Gneiting and Adrian E Raftery.  Strictly proper scoring rules, prediction, and estimation.  _Journal of the American statistical Association_ , 102(477):359–378, 2007.
  * Good (1952) Irving J Good.  Rational decisions.  _Journal of the Royal Statistical Society, Series B, Methodological_ , 14:107–114, 1952.
  * MacKay (2003) David JC MacKay.  _Information theory, inference and learning algorithms_.  Cambridge university press, 2003.
  * Parmigiani and Inoue (2009) Giovanni Parmigiani and Lurdes Inoue.  _Decision theory: Principles and approaches_ , volume 812.  John Wiley & Sons, 2009.
  * Phillips and Edwards (1966) Lawrence D Phillips and Ward Edwards.  Conservatism in a simple probability inference task.  _Journal of experimental psychology_ , 72(3):346, 1966.
  * Winkler (1969) Robert L Winkler.  Scoring rules and the evaluation of probability assessors.  _Journal of the American Statistical Association_ , 64(327):1073–1078, 1969.
  * Winkler (1994) Robert L Winkler.  Evaluating probabilities: Asymmetric scoring rules.  _Management Science_ , 40(11):1395–1405, 1994\.