# Composing Neural Networks with Direct Manipulation

Thomas Weber  0000-0002-6894-605X LMU MunichFrauenlobstr. 7aMunichGermany80337
<EMAIL_ADDRESS>and  Sven Mayer  0000-0001-5462-8782 LMU
MunichFrauenlobstr. 7aMunichGermany80337<EMAIL_ADDRESS>
# The Unadopted Benefits of Graphical Programming for Data-Driven Software

Thomas Weber  0000-0002-6894-605X LMU MunichFrauenlobstr. 7aMunichGermany80337
<EMAIL_ADDRESS>and  Sven Mayer  0000-0001-5462-8782 LMU
MunichFrauenlobstr. 7aMunichGermany80337<EMAIL_ADDRESS>
# Usability and Adoption of Graphical Data-Driven Development Tools

Thomas Weber  0000-0002-6894-605X LMU MunichFrauenlobstr. 7aMunichGermany80337
<EMAIL_ADDRESS>and  Sven Mayer  0000-0001-5462-8782 LMU
MunichFrauenlobstr. 7aMunichGermany80337<EMAIL_ADDRESS>
###### Abstract.

Software development of modern, data-driven applications still relies on tools
that use interaction paradigms that have remained mostly unchanged for
decades. While rich forms of interactions exist as an alternative to textual
command input, they find little adoption in professional software creation. In
this work, we compare graphical programming using direct manipulation to the
traditional, textual way of creating data-driven applications to determine the
benefits and drawbacks of each. In a between-subjects user study (N=18), we
compared developing a machine learning architecture with a graphical editor to
traditional code-based development. While qualitative and quantitative
measures show general benefits of graphical direct manipulation, the user’s
subjective perception does not always match this. Participants were aware of
the possible benefits of such tools but were still biased in their perception.
Our findings highlight that alternative software creation tools cannot just
rely on good usability but must emphasize the demands of their specific target
group, e.g. user control and flexibility, if they want long-term benefits and
adoption.

data-driven software, data-driven development, graphical programming,
interaction paradigms, neural network, machine learning, user study

††copyright: none††conference: cs.HC arXiv.org; 2023; arXiv.org††ccs: Software
and its engineering Software development techniques††ccs: Software and its
engineering Integrated and visual development environments††ccs: Human-
centered computing Empirical studies in interaction design Figure 1. The
graphical programming interface we created for this study to represent rich
interaction with direct manipulation. It consists of a primary canvas (1) on
which blocks can be placed. These blocks are dragged from the block library
(2) and can have additional properties which can be edited (3). Any output is
displayed in a dedicated area at the bottom of the screen (4).

##  1\. Introduction

Machine Learning, particularly using Neural Networks, has enabled many
improvements in fields which are critical to everyday life in our society,
like healthcare (Cohen and Graver, 2019), communication (Zhu et al., 2020),
and security (Sarker et al., 2021; Bagaa et al., 2020). Therefore, neural
networks are an essential tool for software developers going forward.

Yet, while these systems become increasingly complex and hard to understand,
an issue exacerbated by their opacity (Goebel et al., 2018; Hoffman et al.,
2018), how developers create them is not much different from how other
traditional software has been created for decades: the probably most popular
tool, Jupyter notebooks111https://jupyter.org, and many more are fundamentally
still textual command input. Meanwhile, there are many alternative forms of
interaction for programming, which could be applied here. Graphical
programming in particular is an interesting candidate, given the nature of
Machine Learning systems where data visualization often plays a large role,
but many other systems also rely on some form of visual presentation for
different goals like interpretability (Zhang and Zhu, 2018) and explainability
(Ribeiro et al., 2016; Alicioglu and Sun, 2022), optimization (Cejudo Grano de
Oro et al., 2022; Park et al., 2021), debugging (Afzal et al., 2021; Alsallakh
et al., 2021) and many more. Graphical Programming systems like RapidMiner
(Mierswa et al., 2006), KNIME (Berthold et al., 2009), or Orange (Demsar et
al., 2004) fully embrace this and omit textual programming for the most part.
And yet, most developers still fall back to their text editors or Jupyter
notebooks, which raises the question whether these alternative systems are
inferior or more importantly what they are missing.

In this paper, we investigated the behavior of programmers in these contexts
to determine how and which aspects of graphical programming are beneficial and
when developers prefer textual programming. To this end, we performed a
between-groups user study (N=18), where participants had to create a Machine
Learning system for image classification either traditionally with code or
using a simple graphical programming tool, which we designed and developed
specifically for this investigation. This tool allows for the graphical
composition of Neural Networks via direct manipulation of its layers. We set
this tool in contrast to the de facto standard tool, Jupyter notebooks, in
which participants also designed new neural networks. This allowed us to
understand the differences from a qualitative and quantitative perspective.
Thus, addressing the question of what alternative systems are missing today
and how we can improve them.

Our results show that graphical programming – for our task – is, in fact,
already faster and produces less workload. While it is therefore viewed
positively, participants still expect that they work better with the
traditional, text-based editor. This feedback provides insights into the
potential benefits of graphical programming for data-driven development, but
also highlights a disparity in users’ subjective perceptions. Giving
developers a high degree of control and flexibility over their tools appears
to be necessary to ensure the success of graphical development tools. Based on
our results, we argue that building a hybrid between text editors and
graphical programming may be a way to consolidate the different benefits and
expectations in the future.

##  2\. Related Work

Richer interactions than text input are by no means new for software
development in general or data-driven development in particular. Direct
manipulation as one of such rich interactions was, in fact, originally
proposed by Shneiderman (1981) as an alternative to programming languages and
command-based control of computers. Since then, research and industry have
come up with many different forms of interaction. Yet, many of the ideas
specific to professional software engineering and, more recently, data-driven
development have only found limited adoption in practice. Anecdotal evidence
suggests that particularly more radical changes that fundamentally change how
developers interact with the software they create, fare poorly. One such
drastically different interaction paradigm is graphical programming, which
replaces text-based commands with visual representations.

While not embraced by the broad majority, graphical programming still
succeeded in some niche application domains for professional software
engineering. Often these are areas where the software is and needs to fulfill
very strict requirements in timing, safety, etc. (Riccobene et al., 2006;
Gamatié et al., 2011; Scheidler et al., 1993; Anwar et al., 2019). So from the
outside perspective, the benefits of graphical programming, particularly for
complex and critical systems, are apparent. However, given the adoption rate,
from the inside perspective, the point of view of developers, these benefits
seemingly are not yet convincing.

An early article by Petre (1995) already investigated this disparity and
compared the benefit of graphical programming for novices and experts where
she describes some of the benefits of that arise from the rich interaction
like accessibility, memorability, etc. However, the article also emphasizes
the challenges which arise from the different needs of novices and experts.
While graphical programming is often advocated for beginners, Petre (1995)
instead describes how they would rather benefit from more constrained
interactions. The ability to grasp and utilize graphical representation is
something that also requires some expertise and poorly designed visualizations
can easily be confusing or overwhelming.

Erwig and Meyer (1995) also provide an early discussion of the benefits of
visual languages and their integration with the more common textual
counterpart in heterogeneous visual languages for general programming
activities.

Mohagheghi and Dehlen (2008) later reviewed the state in the early 2000s in
the context of Model-driven development tools and practices, which are often
graphical in nature. At that point they found adoption to be limited and only
at the small scale and evaluations to be lacking. Since then, many projects in
which new graphical programming tools have been created now also conduct user
studies to evaluate them, especially the application for children and in
education (Xiajian et al., 2011; Wang et al., 2013; Horn et al., 2009;
Sapounidis et al., 2019) and end user programming (Barricelli et al., 2019;
Ajaykumar and Huang, 2020) but also beyond.

After all these years, graphical programming remains an active research
interest, as seen in the literature review by Selwyn-Smith et al. (2022) and
recent examples like the work of Homer (2023), who investigates the use of
graphical programming for data-flow programming.

###  2.1. Graphical Programming for Data-Driven Applications

Data-driven applications have a high degree of inherent complexity, so
graphical programming could be a viable route to mitigate this. Consequently,
academia and industry have come up with some systems that try to leverage this
interaction to simplify development.

Tools like RapidMiner exist for two decades now (Ritthoff et al., 2001) and
have been applied in several domains and case studies (e.g., (Michalak et al.,
2016; Lane and Kim, 2015; Nagy et al., 2015; Han et al., 2008)). RapidMiner
and other tools (e.g., (Demsar et al., 2004; Berthold et al., 2009))
frequently visualize the data-flow in data-driven applications as a sequence
of blocks, which represent different steps in the processing. With this
representation, they are able to make a relatively abstract process easier to
grasp, while hiding unnecessary details. Of course, many of these details are
necessary for experienced users, so there are different interaction patterns
for accessing them, from popups to the common “bento box” (DeLine and Rowan,
2010) layout to hierarchical structures. They also often provide additional
visualizations as support, e.g., for data exploration or to display output
metrics. While not specifically targeted at novices or experts, they are still
often considered more suitable for novices (Bjaoui et al., 2020) or non-
programmers (Shastri, 2020).

Weber and Hußmann (2022) provide a further overview of tools for data-driven
software development from the last decade. Based on their literature review,
they also draw the conclusion that graphical programming is one area that
could improve data-driven development, but has not yet managed to gain enough
traction and practical use.

Instead, for developers, Jupyter notebooks so far seem the tool of choice
(Randles et al., 2017). From an interaction design point of view, these
notebooks are still different from the setup common in traditional software
development with either a text editor and tools for compilation and execution,
or the Integrated Development Environment (IDE) which combines them. Jupyter
notebooks rather focus on small code fragments and short execution and
feedback cycles, very similar to a REPL (Read-Evaluate-Print-Loop) or
interactive shell environment. In this sense, Jupyter notebooks can be seen as
a regression towards the interaction as one would have with a terminal with
superior back-tracing and presentation. Still, it is the development tool of
choice particularly in the domain of data-driven applications, so any novel
and improved tools will measure up to it.

Furthermore, the Jupyter environment provides infrastructure for extending it,
which not only contributed to its success across many programming languages,
but has also enabled researchers to extend the fairly conservative forms of
interaction. Kery et al. (2020b) and Zhao et al. (2022) both attempted to
enhance the Jupyter programming environment with visualizations that allow
developers to switch between the traditional and graphical representations. In
the user studies, which they conducted, they found that professional users
enjoy the flexibility of switching between different representations, but they
also received critical feedback indicating that experts struggle with some
interactions. In the ODEN tool by Zhao et al. (2022), for example, the
participants were sometimes confused by the mapping between graphical and
textual representation, which prompted them to implement a “calm mode”.
Clearly, designing tools that are suitable for developers of data-driven
applications is not a trivial task. To ensure that they can provide the
benefit they promise, we must first better understand what interactions are
suitable and what users prefer in a given context.

##  3\. A Graphical Programming Editor for Neural Networks

As existing graphical programming tools have an overwhelming amount of
features, we decided to work with a prototypical implementation of a graphical
programming tool. Therefore, we limit the volume of features and interactions
to a minimal, comparable subset and completely control what kind of
interactions were possible during our study. A custom implementation also
allowed us to instrument the tool, giving us much more detailed information
about the interactions. Nonetheless, the tool is inspired by and based on
established tools like RapidMiner (Mierswa et al., 2006) or KNIME (Berthold et
al., 2009) and their block-based interaction: instead of instruction as lines
of code, the functions for processing the data are displayed as blocks. These
blocks can be arranged in space via drag and drop. These blocks can then be
combined into a sequence of operations. This is done via direct manipulation
by dragging a connection from one block to another. Combining blocks like this
lends itself to the domain of data-driven applications, since a mostly
sequential pipeline is a typical structure.

This is the primary canvas for creating programs (see Figure 1). In addition,
similar to how Jupyter notebooks display their output below each cell, the UI
had a panel for displaying any program output in a consistent location.
Besides this, the tool offers a library of blocks, which we grouped into
categories like Datasets, and Models. Furthermore, each block can have
parameters, just like a function call has parameters. These are displayed in a
third panel as input fields depending on the parameter’s type, e.g.,
checkboxes for boolean parameters.

Since we base the task on a tutorial that uses the Keras layers API, we
implemented a subset of its functionality as blocks in our application. This
was convenient as the TensorFlowJS library, which we used internally for the
UI, uses mostly the same API, which allowed us to use the same instructions
and functions in both experimental conditions. The blocks we implemented
included all the necessary models, layers, and data processing steps for the
tutorial and a few additional distractors. Each of these blocks used the same
parameters as listed in the Keras/TensorFlowJS documentation. Furthermore, we
added the MNIST dataset as a block, similar to how it is available in the
tensorflow_datasets Python package used in our conventional tutorial, and two
blocks to enable interactive prediction of new inputs.

In the Keras API, a typical model is composed of multiple layers, i.e., a
function composed of sub-functions. For this reason, our tool offers the
possibility of composite blocks, i.e., blocks that consist of a sequence of
subordinate blocks. Double-clicking such a composite block opens up a new
editor view for arranging its components. Breadcrumb navigation at the top of
the main canvas allows for navigation along this hierarchy. In our scenario,
we used this type of block to create a sequential model from different types
of layers, like dense or flatten layers.

We implemented this UI as an extension for Microsoft’s Visual Studio Code
editor in TypeScript. To complete the task we provided, the participants only
interacted with our extension and no further aspects of Visual Studio Code.
The source code for the extensions is part of the supplementary material and
is ready to be part of a public code repository.

##  4\. User Study

To understand how different types of tools and interactions paradigms affect
the programmers, we conducted a between-subject study. In this study,
participants had to create a simple machine learning system using a neural
network for classifying the MNIST handwritten digits (LeCun, 1998) based on a
set of instructions and provide feedback on their experience and preferences
using a questionnaire. Additionally, we conducted semi-structured interviews
with the participants to get additional in-depth feedback.

In the study, one group of participants created their neural network using the
established interaction of textual command input, using Jupyter notebooks, the
de-facto standard tool for data science programming. The second group created
the same system using an equivalent set of instructions, but with the
graphical programming tool described above. With this, we were able to focus
our study and the interface to those interactions in the intersection of these
two different tools.

###  4.1. Task

The instructions were designed to follow the general outline of a typical
machine-learning tutorial. For this reason, we took an existing example from
the TensorFlow
documentation222https://www.tensorflow.org/datasets/keras_example and adapted
it, where necessary, for our study.

The most notable change is that we split the training and validation, which in
the original are a single function call, into two separate steps. The tutorial
version for the graphical programming group received some additional changes:
we replaced all code examples with screenshots of our tool in the respective
state, and we had to rephrase some sentences such that they reflect the
difference in interaction, which, for example, meant replacing occurrences of
to type or to write with more suitable verbs.

We also opted to keep the visual presentation of the tutorial as close to the
original as possible, but removed any clutter and unnecessary links from the
site to ensure that participants were focused on the core instructions. Any
external links were also removed to keep participants on the tutorial page.

###  4.2. Procedure

After the participants arrived, we gave them a small introduction and asked
for their consent to record the data. The study starts with a brief
introduction of the tool which participants get to use. For Jupyter Notebooks,
this includes information on how to write and execute code and how to extend
the notebook with new cells. For Graphical Programming, we introduce the
panels described above and the direct manipulation via drag and drop.

After this, participants received access to the instructions, and we asked
them to work through these at their own pace. The whole task was scheduled to
take about 15 minutes. As per the study invitation, participants were aware
that we scheduled the duration of the study at 45 minutes and the programming
task was expected to take roughly 15 minutes, but we did not rush them to
finish the programming task or cut them off after the allotted time.

At the end of the programming instructions, we included a link to the survey,
which constituted the last part of the study for most participants. This
survey included both qualitative and quantitative questions, as we describe
below. A subset of five participants was able to accommodate time for an
additional semi-structured interview after the survey, in which we collected
additional qualitative feedback.

###  4.3. Survey and Interview Guidelines

The survey was separated into three core parts: the first part was concerned
with a consent form and information about data protection, followed by
questions about demographic data and background knowledge and expertise. We
used the scale from Edison and Geissler (2003) to record a general attitude
towards technology. In addition, we queried participants about their
experience with the technologies specific to our study, i.e., Jupyter
notebooks, graphical programming, and different topics about machine learning.

The second part of the survey asked for feedback on participants’ programming
tasks and tools. First, we used quasi-continuous, i.e., fine-grained, Likert
scales with an agree-disagree scale (Chimi and Russell, 2009) for a series of
general statements about the task and the tool. Second, we asked participants
to rate the tool based on several categories based on Nielsen’s usability
heuristics (Nielsen, 2005) where applicable and some additional categories
(see Figure 3). Finally, we used the Systems Usability Scale (Brooke et al.,
1996) and the raw NASA Task Load Index (Hart, 2006) as further metrics.
Participants could also provide additional qualitative feedback using free
text fields regarding the tool’s benefits, limitations, and drawbacks.

After providing this feedback, we showed the participants a screen recording
of their task being executed in the alternative tool, Jupyter, for
participants who used the graphical tool and vice versa. Having watched this,
we asked participants to compare what they had seen to what they had used,
using the same categories as before. Here, too, participants could provide
additional qualitative feedback on what they thought to be better or worse in
this alternative tool.

As mentioned, we discussed their experience with several participants after
they had completed the survey in a semi-structured interview. The interview
questions first focused on the general experience during the study and what
aspects of the task were particularly challenging. We then asked about the
tool, its benefits and drawbacks, and under what circumstance or for which
target group it might be most suitable. Finally, we asked for feedback on how
to make the machine learning programming easier to use in the everyday work
context of the participants.

###  4.4. Apparatus

Participants completed both the survey and the programming task on a computer
provided by us using a mouse and keyboard. This way, we ensured that the
training phase of the machine learning system was using the same hardware,
making it more consistent. The page with the instructions was displayed on a
second screen to minimize the need for switching windows. Aside from the
survey, additional information like the timing of interactions and the task
result was automatically recorded in the background.

###  4.5. Participants

For this study, we recruited participants with prior experience with
programming, particularly of data-driven applications. Knowledge of specific
Machine Learning libraries or methods was not a prerequisite. We recruited
them via a combination of personal and professional contacts and multiple
mailing lists that specifically included people with programming backgrounds.

In total, 20 participants completed our study (4 female, 16 male). With a mean
age of 29 ($SD=5.2$) and six years of programming on average ($SD=3.8$). Half
of the participants had a background in computer science, while the other half
came from STEM areas where data processing and the use of machine learning
methods is common, for example, physics or biology research. The participants
were employed full-time in their positions, typically with a completed
Master’s degree (14), or were currently still pursuing a degree in these
domains but worked part-time in projects involving data-driven applications
(6). Based on their feedback, the participants had a generally positive
attitude towards technology in general (average score: 76 of 100, $SD=11$) and
were at least somewhat familiar with all the specific technologies of our
study, like machine learning or Jupyter notebooks. While they knew of the
MNIST data set, none of them reported to have completed the specific tutorial
upon which our task was based. Four of the participants answered that they use
graphical programming tools in their work, although not for creating data-
driven applications. We scheduled for a session of 45 minutes per participant
for introduction, performing the task as well as completing the questionnaire,
and, if time allowed it, additional feedback. Regardless of whether they
required the whole time, participants were compensated with an equivalent of
10 US$ for their contribution.

Figure 2. While there are no significant differences between the responses to
various statements about the task and the tools, they show that the task was
perceived very similarly.

##  5\. Results

Of the 20 people who participated in our study, we excluded two from the data
set, one from each group, after the first data screening, as their responses
strongly suggested that they misunderstood parts of the instructions. The
remaining data yielded the following insights.

As Figure 2 shows, the task was perceived as easy to complete in both
experimental conditions. Given the overall very similar feedback about the
task, both in the survey and the interviews, we conclude that the task design
was suitable, as it was not notably easier or better supported by any of the
two tools.

The scales asking about the tools show some differences: while the graphical
programming tool received slightly better feedback on enjoyment, how it
supports understanding, and its overall design, these differences are not
significant.

###  5.1. Usability

In terms of usability Both tools scored well on the Systems Usability Scale
(Brooke et al., 1996), 81 of 100 ($SD=11.7$) for the graphical tool and 74
($SD=10.4$) for Jupyter notebooks, see Figure 4. This difference is not
significant.

The workload as measured by the raw NASA TLX (Hart, 2006), meanwhile, is
significantly lower for the graphical tool (Shapiro-Wilk-test: $W=0.934$,
$p=0.287$, t-test: $t(17)=-4.027$, $p<0.001$). Moreover, each of the six
subscales of the NASA TLX is lower for the graphical tool individually,
highlighting the reduction in workload even further (see Figure 4).

Furthermore, the graphical tool was also rated slightly superior in
appearance, ease of understanding and learning in the questions (see Figure
3). The participants also rated the graphical programming UI to be
significantly superior in terms of error prevention ($p=0.023$) and how easy
to learn they considered it ($p=0.025$), see Table 1 for a complete list).

Table 1. Results of the statistical analysis of how participants judged the systems concerning usability heuristics (Single Judgment) and comparing the system they used to the alternative as seen in a screen recording (Comparative Judgment). | Rating after usage | Comparative Rating
---|---|---
| Normality test | Wilcoxon/t-test | Normality test | Wilcoxon/t-test
| W | p | W/t | df | p |  | W | p | W/t | df | p |
Visibility of System Status | $0.973$ | $0.852$ | $0.8$ | $15.991$ | $0.435$ |  | $0.945$ | $0.442$ | $-1.903$ | $9.724$ | $0.087$ |
Familiarity | $0.857$ | $0.011$ | $45.5$ |  | $0.691$ |  | $0.956$ | $0.627$ | $0.157$ | $12.763$ | $0.878$ |
Exploration | $0.910$ | $0.085$ | $1.283$ | $13.907$ | $0.221$ |  | $0.943$ | $0.428$ | $-0.858$ | $11.273$ | $0.409$ |
Consistency | $0.952$ | $0.460$ | $-0.018$ | $12.85$ | $0.986$ |  | $0.950$ | $0.634$ | $0.157$ | $9.644$ | $0.879$ |
Error Prevention | $0.953$ | $0.480$ | $2.480$ | $15.638$ | $0.025$ | * | $0.915$ | $0.141$ | $-1.636$ | $12.345$ | $0.127$ |
Easy to Understand | $0.907$ | $0.077$ | $1.331$ | $14.827$ | $0.203$ |  | $0.955$ | $0.536$ | $-2.089$ | $14.414$ | $0.019$ | *
Flexibility | $0.973$ | $0.848$ | $-0.474$ | $13.387$ | $0.643$ |  | $0.920$ | $0.168$ | $3.212$ | $13.915$ | $0.006$ | **
Efficiency of Use | $0.900$ | $0.050$ | $1.133$ | $15.852$ | $0.274$ |  | $0.964$ | $0.740$ | $-0.556$ | $10.509$ | $0.590$ |
Aesthetic Design | $0.823$ | $0.003$ | $57.5$ |  | $0.145$ |  | $0.935$ | $0.287$ | $-3.305$ | $11.829$ | $0.006$ | **
Assistance | $0.946$ | $0.369$ | $1.034$ | $15.984$ | $0.317$ |  | $0.946$ | $0.462$ | $-3.318$ | $11.604$ | $0.006$ | **
Easy to Learn | $0.951$ | $0.439$ | $2.517$ | $15.279$ | $0.023$ | * | $0.927$ | $0.191$ | $-2.384$ | $14.7$ | $0.031$ | *
Useful | $0.948$ | $0.397$ | $0.564$ | $13.443$ | $0.582$ |  | $0.956$ | $0.557$ | $-0.665$ | $12.401$ | $0.518$ |
Figure 3. Each participant rated the tool they used and the alternative tool
in comparison using the same properties and heuristics about usability (*:
$p<0.05$, **: $p<0.01$)

While these responses showed only moderate differences, we observe a stronger
change when participants were asked to compare the interface which they used
to the one they saw in the screen recording. After seeing the alternative
tool, participants rated the graphical tool significantly more favorable
concerning appearance, how it offered assistance, and how easy it might be to
learn. The Jupyter notebooks, on the other hand, were rated positively for
their perceived increased flexibility compared to the graphical tool. See
Figure 4, Table 1 for additional details. Figure 4 also shows the difference
in opinions for participants for these categories, which further highlights
how the comparison skews the perception of the participants.

Figure 4. Task completion time and NASA TLX results show significant
differences, while the SUS shows positive results for both UIs. (*: $p<0.05$,
**: $p<0.01$)

###  5.2. Task Completion Time

We then compared the recorded task completion time for all participants. As
Figure 4 shows, completing the task using textual input on average takes
longer by a significant amount (Mann-Whitney-U test, $p=0.040$). While this is
in part due to two outliers, the questionnaire responses from these two do not
suggest that they considered the task more challenging or struggled more than
the other participants.

###  5.3. Qualitative Results

Beyond these scales, we also queried participants for qualitative feedback on
what aspects of the tools they saw as positive or as an impediment for
adopting and using them. After an initial screening of the responses to the
survey and the interview, we clustered them into five categories: responses
specific to the task, the tool’s appearance, understanding of the system,
feature requests, and feedback on the interaction paradigms.

All nine participants who commented on the task described it as “simple and
easy (as it was) split into meaningful parts” (P3). This is the case for both
groups. The feedback on the appearance of the graphical tool was positive,
with multiple participants enjoying the visual presentation of the data flow
via connections. The focus on the blocks led one participant to note that the
output can be “easy to overlook” (P15), though. The presentation of output
close to the code as done in Jupyter, on the other hand, was considered to be
convenient but constituted the only comment on the appearance of Jupyter
notebooks.

Issues with understanding what was going on were related to general machine
learning concepts. One participant expressed the opinion that “(Jupyter)
notebooks require prior knowledge” (P6), whereas the graphical UI was
described as especially suitable for beginners. To enable growth and learning,
one participant suggested that “a person learning could switch between” (P6)
graphical and text-based UI to use the presentation that suits them best and
at any point in time. This flexibility was also a point of multiple
participants asking for more information to be displayed on demand in the
graphical UI, particularly for the parameters, as it was not always clear
“what parameters (…) were doing” (P9). In addition to documentation,
participants also favored more and particularly continuous feedback. Similar
to how in Jupyter notebooks, each cell has its output adjacent, our
participants wanted to see information about the success and output of blocks
close to them in the graphical UI. In the Jupyter notebooks, the only
indicator for a complete computation without output is a change in cell
numbering – an indicator that, according to participants, is easily
overlooked, as we also observed during the study. Still, one participant
praised the blocks, into which code is split in a notebook, as they help to
structure longer code. This can lead to other issues, though, as “boxes can be
compiled in any order, meaning that code can crash if compiled in the wrong
order,” so there was a wish for information about data flow and notifications
about changes with dependencies. If something does indeed not work, Jupyter
shows the corresponding error messages. These messages were criticized by four
participants to be “cryptic” (P1) and not helpful. However, this is not per se
an issue of Jupyter, but of the Python programming language. The error message
of the graphical UI, which use natural language, were praised by one
participant to be good for “understanding the way things worked” (P15).

The key difference between the two systems lies in the form of interaction,
though. Thus, participants provided a number of comments on the interaction
after both using the system and seeing the alternative system. Four
participants mentioned the issue of typing errors, which can be hard to notice
and typically lead to immediate errors in the code. Writing code, however,
forces it to adhere to a strict sequence of instructions, which matches how it
is executed. On the other hand, the graphical presentation is not constrained
like that, so more emphasis is put on the connections to indicate the sequence
of execution. However, two participants highlighted that they liked the
presentation with lines as connections. This also enables the tool to display
data flow across steps that are not directly adjacent. In our task, an example
of this is the validation data set. It is used fairly late in the process,
which in code means that there are multiple code cells between its declaration
and usage, whereas the blocks have a direct connection.

The interaction with drag and drop and direct manipulation to create the
connections and the blocks were commented upon positively by seven of the
participants. Grabbing the blocks from a library of predefined blocks was also
mentioned as a desirable feature, as it reduces the need to recall API
functionality from memory. While this does apply to the properties of selected
blocks, two participants voiced confusion, as they were overwhelmed by the
number of available parameters, even though they only needed to change one or
two for the task.

###  5.4. General Feedback

Based on the feedback from the questionnaire and the interviews, we also
collected some additional general feedback and insights from participants how
these tools could become more useful in their work context to create data-
driven systems.

####  5.4.1. For Jupyter Notebook

As described above, the primary area of criticism for the Jupyter notebooks
were its error message. Participants not only wanted easier-to-understand text
explanations, but particularly requested more precise feedback on the error
location. Since cells can be executed independently of each other, errors in
later cells can originate in prior ones. Tracking them back to their true
origin was described as challenging in the current system, particularly when
the number of cells and their size grows. In general, participants voiced
interest in more detailed information about data flow in the notebooks.

Integration was another concern, as some participants saw Jupyter notebooks
more as a tool for experimentation. When the software becomes very large,
keeping it in one notebook becomes unfeasible. Integrating the functionality
from one notebook with other, potentially legacy code, was considered
challenging and needed a simple mechanism. While this can also apply to
graphical programming tools, participants did not mention this concern in this
context.

Inline access to documentation and auto-completion were also mentioned as
essential features to eliminate trivial errors and increase programming
efficiency. While these features are, to a degree, already available in
Jupyter notebooks, the fact that our participants still requested them
suggests that they are not prominent enough or hard to discover and use.

Likewise, the output, even from successful execution, was criticized for being
hard to understand at times, but again, this is more of an issue of the
specific implementation of the TensorFlow/Keras functions, which participants
used, while more structured and natural language output is already possible
and implemented by other languages and libraries.

####  5.4.2. For Graphical Programming

As it was only a prototypical implementation for the purpose of this study,
naturally, the graphical programming tool also received its share of
suggestions and feature requests. While this may be partly due to its lack of
maturity, this can still show the priority of the participants.

Concerning feedback from the code execution, the feedback was that successful
computations need not necessarily be displayed spatially close to the source
code, like below Jupyter cells, but a single output terminal was adequate. For
errors, however, the participants highlighted how an clear visual indicator at
the erroneous location is desirable, to quickly identify the source of the
error, while the output from the error message may not be immediately helpful.

Regarding the adoption of such a graphical tool, the consensus appears to be
that participants were willing to give such a tool a try, but did not expect
it to replace their familiar work environment. One reason they named for this,
was that they expected no dramatic productivity benefit. If participants
already used graphical tools, they typically did so for external motivations,
e.g. because it was required by regulations or a legacy project required their
use. For their everyday work, both for creating and using data-driven systems
and in other scenarios, they were reluctant to use graphical programming
tools. The justified this with the fact that they had a set of tools with
which they are familiar, and adopting a new tool can be a time investment with
uncertain revenue. The participants did suggest two scenarios, though, where a
graphical programming environment may find adoption: first for education or
novices, as is a common idea described above. Second, they saw the high-level
presentation as beneficial for setting up a data processing pipeline. In their
expectation, they would, however, switch back to textual code for fine-tuning.
Thus, they suggested some form of on-demand translation or switching between
the different presentations to use different paradigms in different phases of
the development project.

##  6\. Discussion

This feedback, overall, paints a mixed picture. Both tools, Jupyter notebooks
and our prototype of a graphical programming tool, appear to be adequate for
the task at hand. Still, there are notable differences.

From an objective perspective, the significant differences in task completion
time would suggest that direct manipulation is the superior way to create a
data-driven application. With this paradigm, participants also asked fewer
questions, which, to a degree, also suggests that they may have had less
trouble, which can be a consequence of the more constrained nature of the
graphical UI. And yet, the additional feedback is not as clear.

When judging the tools in isolation, the one category in which we observed a
significant difference was the category of error prevention. While this is
corroborated by fewer questions during use, it is an almost necessary
consequence of the design: text input has absolute flexibility, while the
graphical UI constrains the options. So, it is much easier to type an
incorrect command accidentally. Likewise, when the developer has to remember
all commands, they are more prone to make mistakes or not remember the correct
parameters, values, etc.

However, the perception differed more strongly when we showed the alternative
tool to the participants. While in the graphical tool was rated superior in
appearance and how easy it is to learn, the text-based coding was rated to
offer better flexibility. Notably though, even in other categories, like
consistency or how it supports exploration, the opinion of the participants
shifted considerably, which suggests a strong anchoring bias, i.e., the
perception of a tool is skewed by what the participants compare it to. Outside
our study, this could mean that any new tool needs to measure up to what
developers already use and with which they are very familiar. Therefore, it is
not surprising that the adoption of alternative tools like graphical
programming tools in general software development and data-driven development
is sluggish at best and happens only in small incremental changes.

This is exacerbated by the opinion, as voiced by our participants, but also
beyond that, that graphical programming should be targeted at novices or for
learning, while the traditional text editor is the tool of choice for experts.
Even if participants could identify the potential benefits of different forms
of interaction and presentation, multiple suggested that any such benefit
would not apply to them. In fact, they, for example, expect to be faster with
written code, which the data in our study does not support. This perception
may hold true for the pure input method, but this fails to consider contextual
factors such as potentially difficult debugging, documentation, etc.

Of course, the feedback provided by the participants focuses on the two tools
used in the study, which were deliberately limited in features. Additional
support mechanisms like good auto-completion can make textual input faster,
but the same potential for improvement also applies to other interaction
paradigms beyond typing code. While the support mechanisms for text are
naturally more apparent, some of them may be adapted to other forms like
graphical programming, or they could enable completely new support mechanisms.
Certainly, this is not without challenges and as previously mentioned,
graphical programming can become overwhelming with a growing number of
features. Therefore, striking a balance between offering enough support but
remaining unobtrusive is important.

Generally though, the feedback from our study demonstrates that the lack of
adoption is not a consequence of inherently inferior usability. The recorded
metrics, in fact, show that a graphical programming environment with an
equivalent level of features can be viewed on par and even superior in some
aspects. Yet, even though our participants saw the potential benefits for
productivity and effectiveness, they remain reluctant to use different types
of tools, partially biased by their prior experience and habituation. If we
hope to make the benefits of alternative forms of interaction more accessible,
maintaining an adequate level of usability appears to be only a prerequisite.
Since software developers seem to value some aspects of their user experience
more, particularly flexibility and being in control of their software, this
may be an aspect that needs to be addressed specifically.

A potentially simple way to facilitate this is to give developers more choice:
if a tool supports different forms of interaction to create a data-driven
application, e.g., via graphical and text-based view, developers can select
which is most suitable in a given situation and switch between them. While the
simple availability may convince some, it is still very possible that
developers simply decide to fall back on what they know and do not utilize
their options. The increased functionality also adds complexity, so, as
previously described by Petre (1995), some expertise will be necessary to
leverage this effectively. It also remains yet unclear how such hybrid systems
best consolidate the options to prevent this. More interactions and
functionality easily runs the risk of adding complexity and confusion. There
are many alternatives for combining these forms of interaction, be it side-by-
side, as a mutually exclusive selection, or any form in between. Which of
these or whether any of these have desirable effects should be explored in the
future. Jupyter notebooks could be a viable platform for this in the future.
While the core notebooks currently offer only hybrid output, extensions with
more interactive widgets already exist (e.g., (Kery et al., 2020a)) and adding
to this to allow for fully hybrid interaction certainly seems viable.

Given how used many developers already are to their text editors or IDEs, for
wide adoption, other forms of interaction in software development must be
advertised as a viable alternative for productive, professional use. Combining
this with the opinion that graphical programming may be suitable for
education, early Computer Science education may be a point where such a change
in perception can be fostered. If a new generation of developers is introduced
to not just the traditional text editor or Integrated Development Environments
(IDEs), but also consistently uses other tools in productive use, they may
become normalized and be just one tool in a developer’s toolbox.

Usage for real-world tasks with long-term observations could then yield
insights into how a wider tool choice affects work practices over time and how
this, in turn, can affect tool design. While this is possible in the few
domains who already use graphical tools, observing this in professional,
large-scale data-driven development is, unfortunately, currently not yet
possible for lack of adoption. This area, though, could provide some
interesting insights, particularly where deep neural networks are used. They
are an especially interesting case for other forms of programming and
presentation, since in them a lot of information and effort is highly
condensed in an often opaque fashion, making them hard to grasp. Since people
thus already often communicate about them with visualizations, a graphical
presentation could make them more transparent and explicit. Still, their size
leads to different challenges of presentation. Feasibility considerations
limited our study to use the MNIST example, which also allowed for fairly
controlled observations. While it contains all fundamental steps of the
typical workflow, this constrained example may not elicit all the nuances of
data-driven development, though. With larger systems like deep neural networks
and in field studies, it will be more challenging to compare equivalent
systems implemented using different tools as we did in our study, but the
feedback could still help refine our understanding where the discrepancy in
perception of such tools originates and which features make graphical
programming viable to handle the complexity of large neural networks.

##  7\. Conclusion

This paper contributes insights into the design of software development tools
for data-driven development. Our experiments compare textual code-based
programming to graphical programming as an alternative rich form of
interaction. They show the latter’s benefits, for example, faster task
completion and better error prevention. In addition, the participants in our
study can identify further benefits, especially related to learning and the
ability to grasp the general structure of a complex data-driven application
quickly. In isolation, the perception of this form of interaction is not much
different from that of typing out code, though, but when presented with the
alternative, this difference becomes more pronounced though, indicating an
anchoring bias.

At times, participants’ subjective perception does not reflect the actual
situation, and participants dismiss some potential benefits but dismiss them
as only applicable to others. All this, coupled with an attitude to “never
change a running system”, after all, current methods are capable of producing
working software, could be an indicator why not just in data-driven
development but in software development in general, a large number of tools
with rich interactions are acknowledged but play no significant role in
productive use. Our study also highlights how general usability appears to
play a minor role, while certain aspects like flexibility seem more important.

One attempt to overcome this from the perspective of interaction design may be
to lean into the aspect of flexibility and developers to choose from different
forms of interaction, both traditional and novel, depending on the usage
context and current developer’s needs. How exactly code and other forms of
rich interaction, like direct manipulation in our prototype, can be combined
not just as replacement for each other but in synergy remains an open
question. Since Jupyter, as the popular choice for this kind of programming,
was still perceived relatively positively and allows for some extensibility,
it may be a good starting point. This can also mitigate any overhead which a
completely new tool could bring. It remains to be seen, though, how far we can
take the existing tools, which were designed for text-based input, and
whether, at some point, the ever-growing complexity of data-driven application
becomes sufficient motivation to adopt alternative tool paradigms.

## References

  * (1)
  * Afzal et al. (2021) Shazia Afzal, Arunima Chaudhary, Nitin Gupta, Hima Patel, Carolina Spina, and Dakuo Wang. 2021\.  Data-Debugging Through Interactive Visual Explanations. In _Trends and Applications in Knowledge Discovery and Data Mining_ , Manish Gupta and Ganesh Ramakrishnan (Eds.). Springer International Publishing, Cham, 133–142.
  * Ajaykumar and Huang (2020) Gopika Ajaykumar and Chien-Ming Huang. 2020.  User Needs and Design Opportunities in End-User Robot Programming. In _Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_ (Cambridge, United Kingdom) _(HRI ’20)_. Association for Computing Machinery, New York, NY, USA, 93–95.  https://doi.org/10.1145/3371382.3378300
  * Alicioglu and Sun (2022) Gulsum Alicioglu and Bo Sun. 2022.  A survey of visual analytics for Explainable Artificial Intelligence methods.  _Comput. Graph._ 102 (2022), 502–520.  https://doi.org/10.1016/j.cag.2021.09.002
  * Alsallakh et al. (2021) Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Shubham Muttepawar, Edward Wang, Sara Zhang, David Adkins, and Orion Reblitz-Richardson. 2021.  Debugging the Internals of Convolutional Networks. In _eXplainable AI approaches for debugging and diagnosis._ https://openreview.net/forum?id=0YRkrxe2blh
  * Anwar et al. (2019) Muhammad Waseem Anwar, Muhammad Rashid, Farooque Azam, Muhammad Kashif, and Wasi Haider Butt. 2019\.  A model-driven framework for design and verification of embedded systems through SystemVerilog.  _Des. Autom. Embed. Syst._ 23, 3-4 (2019), 179–223.  https://doi.org/10.1007/s10617-019-09229-y
  * Bagaa et al. (2020) Miloud Bagaa, Tarik Taleb, Jorge Bernal Bernabé, and Antonio F. Skarmeta. 2020.  A Machine Learning Security Framework for Iot Systems.  _IEEE Access_ 8 (2020), 114066–114077.  https://doi.org/10.1109/ACCESS.2020.2996214
  * Barricelli et al. (2019) Barbara Rita Barricelli, Fabio Cassano, Daniela Fogli, and Antonio Piccinno. 2019.  End-user development, end-user programming and end-user software engineering: A systematic mapping study.  _J. Syst. Softw._ 149 (2019), 101–137.  https://doi.org/10.1016/j.jss.2018.11.041
  * Berthold et al. (2009) Michael R. Berthold, Nicolas Cebron, Fabian Dill, Thomas R. Gabriel, Tobias Kötter, Thorsten Meinl, Peter Ohl, Kilian Thiel, and Bernd Wiswedel. 2009\.  KNIME - the Konstanz Information Miner: Version 2.0 and Beyond.  _SIGKDD Explor. Newsl._ 11, 1 (Nov. 2009), 26–31.  https://doi.org/10.1145/1656274.1656280
  * Bjaoui et al. (2020) Mohamed Bjaoui, Houneida Sakly, Mourad Said, Naoufel Kraiem, and Mohamed Salim Bouhlel. 2020\.  Depth Insight for Data Scientist with RapidMiner an Innovative Tool for AI and Big Data towards Medical Applications. In _Proceedings of the 2nd International Conference on Digital Tools & Uses Congress_ (Virtual Event, Tunisia) _(DTUC ’20)_. Association for Computing Machinery, New York, NY, USA, Article 21, 6 pages.  https://doi.org/10.1145/3423603.3424059
  * Brooke et al. (1996) John Brooke et al. 1996\.  SUS-A quick and dirty usability scale.  _Usability evaluation in industry_ 189, 194 (1996), 4–7.
  * Cejudo Grano de Oro et al. (2022) José Eduardo Cejudo Grano de Oro, Petra Julia Koch, Joachim Krois, Anselmo Garcia Cantu Ros, Jay Patel, Hendrik Meyer-Lueckel, and Falk Schwendicke. 2022.  Hyperparameter Tuning and Automatic Image Augmentation for Deep Learning-Based Angle Classification on Intraoral Photographs - A Retrospective Study.  _Diagnostics_ 12, 7 (2022).  https://doi.org/10.3390/diagnostics12071526
  * Chimi and Russell (2009) Carl J Chimi and David L Russell. 2009.  The Likert scale: A proposal for improvement using quasi-continuous variables. In _Information Systems Education Conference, Washington, DC_. 1–10.
  * Cohen and Graver (2019) I. Glenn Cohen and Harry Graver. 2019.  A Doctor’s Touch: What Big Data in Health Care Can Teach Us About Predictive Policing.  _SSRN Electronic Journal_ (2019).  https://doi.org/10.2139/ssrn.3432095
  * DeLine and Rowan (2010) Robert DeLine and Kael Rowan. 2010.  Code Canvas: Zooming towards Better Development Environments. In _Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2_ (Cape Town, South Africa) _(ICSE ’10)_. Association for Computing Machinery, New York, NY, USA, 207–210.  https://doi.org/10.1145/1810295.1810331
  * Demsar et al. (2004) Janez Demsar, Blaz Zupan, Gregor Leban, and Tomaz Curk. 2004\.  Orange: From Experimental Machine Learning to Interactive Data Mining. In _Knowledge Discovery in Databases: PKDD 2004, 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, Pisa, Italy, September 20-24, 2004, Proceedings_ _(Lecture Notes in Computer Science, Vol. 3202)_ , Jean-François Boulicaut, Floriana Esposito, Fosca Giannotti, and Dino Pedreschi (Eds.). Springer, 537–539.  https://doi.org/10.1007/978-3-540-30116-5_58
  * Edison and Geissler (2003) Steve W. Edison and Gary L. Geissler. 2003.  Measuring attitudes towards general technology: Antecedents, hypotheses and scale development.  _Journal of Targeting, Measurement and Analysis for Marketing_ 12, 2 (1 11 2003), 137 – 156.
  * Erwig and Meyer (1995) Martin Erwig and Bernd Meyer. 1995.  Heterogeneous Visual Languages-Integrating Visual and Textual Programming. In _Proceedings 11th International IEEE Symposium on Visual Languages, Darmstadt, Germany, September 5-9, 1995_. IEEE Computer Society, 318–325.  https://doi.org/10.1109/VL.1995.520825
  * Gamatié et al. (2011) Abdoulaye Gamatié, Sébastien Le Beux, Éric Piel, Rabie Ben Atitallah, Anne Etien, Philippe Marquet, and Jean-Luc Dekeyser. 2011.  A Model-Driven Design Framework for Massively Parallel Embedded Systems.  _ACM Trans. Embed. Comput. Syst._ 10, 4 (2011), 39:1–39:36.  https://doi.org/10.1145/2043662.2043663
  * Goebel et al. (2018) Randy Goebel, Ajay Chander, Katharina Holzinger, Freddy Lécué, Zeynep Akata, Simone Stumpf, Peter Kieseberg, and Andreas Holzinger. 2018.  Explainable AI: The New 42?. In _Machine Learning and Knowledge Extraction - Second IFIP TC 5, TC 8/WG 8.4, 8.9, TC 12/WG 12.9 International Cross-Domain Conference, CD-MAKE 2018, Hamburg, Germany, August 27-30, 2018, Proceedings_ _(Lecture Notes in Computer Science, Vol. 11015)_ , Andreas Holzinger, Peter Kieseberg, A Min Tjoa, and Edgar R. Weippl (Eds.). Springer, 295–303.  https://doi.org/10.1007/978-3-319-99740-7_21
  * Han et al. (2008) Jianchao Han, Juan C. Rodriguez, and Mohsen Beheshti. 2008\.  Diabetes Data Analysis and Prediction Model Discovery Using RapidMiner. In _International Conference on Bio-Science and Bio-Technology, BSBT 2008, part of the Second International Conference on Future Generation Communication and Networking, FGCN 2008, Volume 3, Hainan Island, China, December 13-15, 2008_. IEEE Computer Society, 96–99.  https://doi.org/10.1109/FGCN.2008.226
  * Hart (2006) Sandra G. Hart. 2006\.  Nasa-Task Load Index (NASA-TLX); 20 Years Later.  _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ 50, 9 (2006), 904–908.  https://doi.org/10.1177/154193120605000909
  * Hoffman et al. (2018) Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2018.  Metrics for Explainable AI: Challenges and Prospects.  _CoRR_ abs/1812.04608 (2018).  arXiv:1812.04608 http://arxiv.org/abs/1812.04608
  * Homer (2023) Michael Homer. 2023\.  Multiple-Representation Visual Compositional Dataflow Programming.  (2023).
  * Horn et al. (2009) Michael S. Horn, Erin Treacy Solovey, R. Jordan Crouser, and Robert J. K. Jacob. 2009.  Comparing the use of tangible and graphical programming languages for informal science education. In _Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, Boston, MA, USA, April 4-9, 2009_ , Dan R. Olsen Jr., Richard B. Arthur, Ken Hinckley, Meredith Ringel Morris, Scott E. Hudson, and Saul Greenberg (Eds.). ACM, 975–984.  https://doi.org/10.1145/1518701.1518851
  * Kery et al. (2020a) Mary Beth Kery, Donghao Ren, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, and Kayur Patel. 2020a.  Mage: Fluid Moves Between Code and Graphical Work in Computational Notebooks. In _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_ (Virtual Event, USA) _(UIST ’20)_. Association for Computing Machinery, New York, NY, USA, 140–151.  https://doi.org/10.1145/3379337.3415842
  * Kery et al. (2020b) Mary Beth Kery, Donghao Ren, Kanit Wongsuphasawat, Fred Hohman, and Kayur Patel. 2020b.  The Future of Notebook Programming Is Fluid. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI 2020, Honolulu, HI, USA, April 25-30, 2020_ , Regina Bernhaupt, Florian ’Floyd’ Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjøn, Shengdong Zhao, Briane Paul Samson, and Rafal Kocielnik (Eds.). ACM, 1–8.  https://doi.org/10.1145/3334480.3383085
  * Lane and Kim (2015) Jesse Lane and Hak J. Kim. 2015.  Big data: web-crawling and analysing financial news using RapidMiner.  _Int. J. Bus. Inf. Syst._ 19, 1 (2015), 41–57.  https://doi.org/10.1504/IJBIS.2015.069064
  * LeCun (1998) Yann LeCun. 1998\.  The MNIST database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ (1998).
  * Michalak et al. (2016) Marcin Michalak, Katarzyna Dusza, Dominik Korda, Krzysztof Kozlowski, Bartlomiej Szwej, Michal Kozielski, Marek Sikora, and Lukasz Wróbel. 2016.  Application of RapidMiner and R Environments to Dangerous Seismic Events Prediction. In _Proceedings of the 2016 Federated Conference on Computer Science and Information Systems, FedCSIS 2016, Gdańsk, Poland, September 11-14, 2016_ _(Annals of Computer Science and Information Systems, Vol. 8)_ , Maria Ganzha, Leszek A. Maciaszek, and Marcin Paprzycki (Eds.). IEEE, 249–252.  https://doi.org/10.15439/2016F188
  * Mierswa et al. (2006) Ingo Mierswa, Michael Wurst, Ralf Klinkenberg, Martin Scholz, and Timm Euler. 2006.  YALE: rapid prototyping for complex data mining tasks. In _Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006_ , Tina Eliassi-Rad, Lyle H. Ungar, Mark Craven, and Dimitrios Gunopulos (Eds.). ACM, 935–940.  https://doi.org/10.1145/1150402.1150531
  * Mohagheghi and Dehlen (2008) Parastoo Mohagheghi and Vegard Dehlen. 2008.  Where Is the Proof? - A Review of Experiences from Applying MDE in Industry. In _Model Driven Architecture - Foundations and Applications, 4th European Conference, ECMDA-FA 2008, Berlin, Germany, June 9-13, 2008. Proceedings_ _(Lecture Notes in Computer Science, Vol. 5095)_ , Ina Schieferdecker and Alan Hartman (Eds.). Springer, 432–443.  https://doi.org/10.1007/978-3-540-69100-6_31
  * Nagy et al. (2015) Gabor Nagy, Gergo Barta, and Tamás Henk. 2015.  Portfolio optimization using local linear regression ensembles in RapidMiner.  _CoRR_ abs/1506.08690 (2015).  arXiv:1506.08690 http://arxiv.org/abs/1506.08690
  * Nielsen (2005) Jakob Nielsen. 2005\.  Ten usability heuristics.
  * Park et al. (2021) Heungseok Park, Yoonsoo Nam, Ji-Hoon Kim, and Jaegul Choo. 2021\.  HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks.  _IEEE Trans. Vis. Comput. Graph._ 27, 2 (2021), 1407–1416.  https://doi.org/10.1109/TVCG.2020.3030380
  * Petre (1995) Marian Petre. 1995\.  Why Looking Isn’t Always Seeing: Readership Skills and Graphical Programming.  _Commun. ACM_ 38, 6 (1995), 33–44.  https://doi.org/10.1145/203241.203251
  * Randles et al. (2017) Bernadette M. Randles, Irene V. Pasquetto, Milena S. Golshan, and Christine L. Borgman. 2017\.  Using the Jupyter Notebook as a Tool for Open Science: An Empirical Study. In _Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries_ (Toronto, Ontario, Canada) _(JCDL ’17)_. IEEE Press, 338–339.
  * Ribeiro et al. (2016) Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (Eds.). ACM, 1135–1144.  https://doi.org/10.1145/2939672.2939778
  * Riccobene et al. (2006) Elvinia Riccobene, Patrizia Scandurra, Alberto Rosti, and Sara Bocchio. 2006.  A model-driven design environment for embedded systems. In _Proceedings of the 43rd Design Automation Conference, DAC 2006, San Francisco, CA, USA, July 24-28, 2006_ , Ellen Sentovich (Ed.). ACM, 915–918.  https://doi.org/10.1145/1146909.1147141
  * Ritthoff et al. (2001) Oliver Ritthoff, Ralf Klinkenberg, Simon Fischer, Ingo Mierswa, and Sven Felske. 2001\.  Yale: Yet Another Learning Environment.
  * Sapounidis et al. (2019) Theodosios Sapounidis, Stavros Demetriadis, Pantelis M. Papadopoulos, and Dimitrios Stamovlasis. 2019\.  Tangible and graphical programming with experienced children: A mixed methods analysis.  _International Journal of Child-Computer Interaction_ 19 (2019), 67–78.  https://doi.org/10.1016/j.ijcci.2018.12.001
  * Sarker et al. (2021) Iqbal H. Sarker, Md. Hasan Furhad, and Raza Nowrozy. 2021\.  AI-Driven Cybersecurity: An Overview, Security Intelligence Modeling and Research Directions.  _SN Comput. Sci._ 2, 3 (2021), 173.  https://doi.org/10.1007/s42979-021-00557-0
  * Scheidler et al. (1993) Christian Scheidler, Lorenz Schäfers, and Ottmar Krämer-Fuhrmann. 1993.  TRAPPER: A Graphical Programming Environment for Industrial High-Performance Applications. In _PARLE ’93, Parallel Architectures and Languages Europe, 5th International PARLE Conference, Munich, Germany, June 14-17, 1993, Proceedings_ _(Lecture Notes in Computer Science, Vol. 694)_ , Arndt Bode, Mike Reeve, and Gottfried Wolf (Eds.). Springer, 403–413.  https://doi.org/10.1007/3-540-56891-3_32
  * Selwyn-Smith et al. (2022) Ben Selwyn-Smith, Craig Anslow, and Michael Homer. 2022\.  Blocks, Blocks, and More Blocks-Based Programming. In _Proceedings of the 1st ACM SIGPLAN International Workshop on Programming Abstractions and Interactive Notations, Tools, and Environments_ (Auckland, New Zealand) _(PAINT 2022)_. Association for Computing Machinery, New York, NY, USA, 35–47.  https://doi.org/10.1145/3563836.3568726
  * Shastri (2020) Dvijesh J. Shastri. 2020\.  Machine Learning for Non-Programmers. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI EA ’20)_. Association for Computing Machinery, New York, NY, USA, 1–3.  https://doi.org/10.1145/3334480.3375051
  * Shneiderman (1981) Ben Shneiderman. 1981\.  Direct manipulation: A step beyond programming languages (abstract only). In _Proceedings of the Joint Conference on Easier and More Productive Use of Computer Systems, CHI 1981, Part II: Human Interface and the User Interface, Ann Arbor, MI, USA, May 20-22, 1981_ , Gregory A. Marks and Lorraine Borman (Eds.). ACM, 143\.  https://doi.org/10.1145/800276.810991
  * Wang et al. (2013) Danli Wang, Yang Zhang, and Shengyong Chen. 2013.  E-Block: A Tangible Programming Tool with Graphical Blocks.  _Mathematical Problems in Engineering_ 2013 (Feb. 2013), 598547\.
  * Weber and Hußmann (2022) Thomas Weber and Heinrich Hußmann. 2022.  Tooling for Developing Data-Driven Applications: Overview and Outlook. In _Proceedings of Mensch Und Computer 2022_ (Darmstadt, Germany) _(MuC ’22)_. Association for Computing Machinery, New York, NY, USA, 66–77.  https://doi.org/10.1145/3543758.3543779
  * Xiajian et al. (2011) Chen Xiajian, Wang Danli, and Wang Hongan. 2011.  Design and implementation of a graphical programming tool for children. In _2011 IEEE International Conference on Computer Science and Automation Engineering_ , Vol. 4. 572–576.  https://doi.org/10.1109/CSAE.2011.5952915
  * Zhang and Zhu (2018) Quanshi Zhang and Song-Chun Zhu. 2018.  Visual interpretability for deep learning: a survey.  _Frontiers Inf. Technol. Electron. Eng._ 19, 1 (2018), 27–39.  https://doi.org/10.1631/FITEE.1700808
  * Zhao et al. (2022) Chunqi Zhao, I-Chao Shen, Tsukasa Fukusato, Jun Kato, and Takeo Igarashi. 2022.  ODEN: Live Programming for Neural Network Architecture Editing. In _27th International Conference on Intelligent User Interfaces_ (Helsinki, Finland) _(IUI ’22)_. Association for Computing Machinery, New York, NY, USA, 392–404.  https://doi.org/10.1145/3490099.3511120
  * Zhu et al. (2020) Guangxu Zhu, Dongzhu Liu, Yuqing Du, Changsheng You, Jun Zhang, and Kaibin Huang. 2020\.  Toward an Intelligent Edge: Wireless Communication Meets Machine Learning.  _IEEE Communications Magazine_ 58, 1 (2020), 19–25.  https://doi.org/10.1109/MCOM.001.1900103


# ClimateQ&A : bridging the gap between climate scientists and the general
public

Natalia De La Calzada<EMAIL_ADDRESS>Théo Alves Da Costa
<EMAIL_ADDRESS>Annabelle Blangero
<EMAIL_ADDRESS>Nicolas Chesneau
<EMAIL_ADDRESS>
###### Abstract

This research paper investigates public views on climate change and
biodiversity loss by analyzing questions asked to the ClimateQ&A platform.
ClimateQ&A is a conversational agent that uses LLMs to respond to queries
based on over 14,000 pages of scientific literature from the IPCC and IPBES
reports. Launched online in March 2023, the tool has gathered over 30,000
questions, mainly from a French audience. Its chatbot interface allows for the
free formulation of questions related to nature*. While its main goal is to
make nature science more accessible, it also allows for the collection and
analysis of questions and their themes. Unlike traditional surveys involving
closed questions, this novel method offers a fresh perspective on individual
interrogations about nature. Running NLP clustering algorithms on a sample of
3,425 questions, we find that a significant 25.8% inquire about how climate
change and biodiversity loss will affect them personally (e.g., where they
live or vacation, their consumption habits) and the specific impacts of their
actions on nature (e.g., transportation or food choices). This suggests that
traditional methods of surveying may not identify all existing knowledge gaps,
and that relying solely on IPCC and IPBES reports may not address all
individual inquiries about climate and biodiversity, potentially affecting
public understanding and action on these issues.

Note:we use “nature” as an umbrella term for “climate change” and
“biodiversity loss”.

##  1 Introduction

Understanding public perception of climate change and biodiversity loss
involves assessing a population’s awareness and literacy on these issues.
Climate change awareness is characterized by concern and belief in the
phenomenon Jürkenbeck et al. (2021). On the other hand, climate change
literacy, a subset of scientific literacy, denotes the knowledge, skills, and
attitudes necessary for effectively addressing climate change. Literacy in
climate change is essential for informed decision-making, emissions reduction,
and community resilience Dupigny-Giroux and Cole (2018).

Studies in recent years have sought to quantify public awareness and literacy
on nature topics. Whether conducted by academia or international
organizations, the predominant method used is closed surveys and
questionnaires. Comprehensive examples include the Sustainability Literacy
Test (Sulitest) that is aligned with the UN SDGs, and the People’s Climate
Vote developed by the UNDP and Oxford University’s Department of Sociology
United Nations Development Programme, University of Oxford (2021). Other
research groups have examined perspectives from individuals of different age
groups and regions, such as responses to climate change in advanced economies
Bell et al. (2021), climate change beliefs in Latin America Spektor et al.
(2023), the influence of socio-demographic characteristics on climate change
awareness and risk perception across 119 countries Lee et al. (2015), among
many others.

We find that in the cited cases, data is collected through a series of closed
questions asked to the participant, for example:

“”How serious of a threat is global warming to you and your family?’ Response
categories included: ‘Not at all serious’, ‘Not very serious’, ‘Somewhat
serious’, and ‘Very serious’” Lee et al. (2015)

These studies offer valuable insights into the public”s perception of climate
change. They demonstrate that awareness of climate change significantly
differs across geographical regions and demographic groups. While personal
experiences and exposure to phenomena like rising temperatures influence
awareness, educational attainment emerges as its most significant predictor.
Education facilitates access to global discussions on the topic, often
conducted in the language of science and global politics Lee et al. (2015).
Indeed, the primary sources of information are the reports of the IPCC and
IPBES, whose core mandate is to provide policymakers with assessments of
climate and biodiversity. While these documents provide a comprehensive
understanding of nature phenomena, their format poses several barriers to the
general public. These include the use of complex jargon (e.g., uncertainty and
confidence levels) and availability exclusively in the UN-official languages,
which are spoken as a first or second language by only half of the world”s
population. Additionally, a comprehensive understanding of these documents
requires consideration of both the Summary for Policymakers and the Full
Reports, which collectively can span thousands of pages for each publication.
These barriers, combined with growing fake news and skepticism around these
topics, can hinder public awareness and understanding of these critically
important issues.

It is in view of these issues that we decided to create ClimateQ&A
(www.climateqa.com). The tool leverages LLMs to interpret and answer nature-
related questions based solely on the IPCC and IPBES reports. The architecture
of the tool is described in Annex A. Its core functionalities are: a chatbot-
like interface in which users can enter questions in any language, settings
that allow for the selection of specific reports and language modes (general
public, experts and children), and the sources of the information quoted in
the answer, with direct access to the documents in the page where the
information is found. The tool uses more than 14,500 pages of knowledge from
the two institutions, spanning the multiple publications (Full Report, Summary
for Policymakers, Special Reports, Cross-Chapter Documents) to deliver a fact-
based, comprehensive and understandable answer to the user.

Since its launch in May 2023, the tool has attracted significant attention
from the main media outlets in France, namely Libération Monod (2023), Le
Figaro Mediavilla (2023), Les Echos Henno (2023) among others; and several
authors linked to the IPCC have recommended the use of the tool through social
media Masson-Delmotte (2023). This important reach has allowed us to build a
database of user queries, which we analyze in the next section to understand
individual interrogations of climate and biodiversity issues. To our
knowledge, this is the first large-scale study that uses this novel open-ended
method to derive insights on public perception of nature phenomena.

##  2 Method

We extracted a sample of 7,000 questions posed to ClimateQ&A in April 2023,
one month after the tool’s launch. Using open-source NLP algorithms, we
categorized these questions into two groups: (1) salient topics, such as
climate change, nature, and biodiversity, and (2) intent. From this sample, we
excluded questions meeting the following criteria:

  * •

Repeated questions (exact matches), typically those suggested to users on the
platform.

  * •

Unclassifiable questions, including those lacking a specific topic, unrelated
queries, or those not identified by the NLP algorithm.

Subsequently, we conducted clustering based on dense embeddings of the
questions (bge-base-en-v1.5). Initial cluster naming was automated using
BERTopic. Employing this NLP approach, we categorized 3,425 questions across
130 different clusters representing salient topics. We manually reviewed each
cluster to identify recurring categories and sub-categories related to climate
and biodiversity science, along with adjacent topics related to economics and
society. Finally, we characterized each cluster based on whether the user
expressed a general or personal inquiry.

##  3 Results

When conducting a topic analysis, we find that the majority of questions in
the sample relate to Climate Change and Greenhouse Gas Emissions (43.1%).
Within this category, questions cover several sub-topics, including the
general concept of climate change, its causes, consequences, and potential
mitigation or reversal measures. The subsequent largest categories, Earth
System (12.1%), Energy (10.5%), and Citizens & Behaviors (5.1%) are
considerably smaller in comparison to climate-related inquiries. Biodiversity-
related questions, in particular, are notably less prevalent, accounting for
only 2.9% of the total. This disparity may stem from the tool’s name,
ClimateQ&A, and the tool interface”s UX emphasis on climate-related queries.
Additionally, it could be attributed to the comparatively lower coverage of
biodiversity topics in mass media and public discourse compared to climate
change. For a complete breakdown of the questions’ topics, refer to Annex B.

During intent analysis, we observe that up to 25.8% of questions express
personal interrogations regarding nature phenomena. The most prevalent sub-
topic in the dataset, comprising 8.8% of the total, pertains to inquiries
about climate change impact projections in particular geographic areas,
possibly tied to individual user circumstances. Other remarkable categories
encompass the impacts on climate change of specific consumption patterns
(6.7%), personal concerns regarding the physical and economic consequences of
climate change (5.1%), skepticism regarding scientific consensus or the
ramifications of climate change (0.3%), and the efficiency of individual
actions in mitigating its impacts (2.1%).

The implications of these findings are twofold. Firstly, they suggest that
employing social surveys to gauge public perceptions of nature, where users
can freely formulate questions, may help climate scientists identify existing
knowledge gaps and pinpoint areas of interest for the general public. This may
not be allowed by closed-ended questionnaires, where participants choose from
a list of pre-specified options. In our analysis, we find that the public may
not only interested in broader concepts of nature science but also in how
these phenomena will personally affect them. Secondly, it underscores the need
for enhanced efforts to meet this specific demand for climate and
biodiversity-related information. Given the IPCC and IPBES’s mandate to
provide synthesis for policymakers, alternative outreach and dissemination
techniques may be necessary, potentially leveraging tools like ClimateQ&A and
tailored communication strategies suited to local contexts.

##  4 Discussion and future work

This paper investigates public perceptions of climate change and biodiversity
loss using a dataset comprising 3,500 questions extracted from ClimateQ&A.
This subset corresponds to inquiries made in April 2023, shortly after the
tool’s launch for public use, and contains sampling bias stemming directly
from the French audience of the tool. By February 2024, the tool has gathered
over 30,000 questions. A comprehensive analysis of this larger dataset could
provide further insights into knowledge gaps and public demand for specific
topics.

An additional interesting avenue for analysis would involve tracking the
evolution of questions over time and aligning this evolution with important
environmental events (floods, wildfires, heat waves…) or ongoing public
debates to assess whether such events influence public interest in related
topics. For example, we have identified a significant number of questions
associated with large water reservoirs, which may be linked to ongoing related
controversies in France in March/April 2023.

Based on our findings, we emphasize that this method of social surveying
offers novel perspectives on public perceptions of climate change and
biodiversity loss, particularly in terms of awareness and literacy. While
traditional survey methods are valuable for establishing baseline attitudes,
this approach delves into individuals’ personal inquiries about the subject,
potentially informing future research and communication strategies at
organizational, national, and community levels.

In this context, enhancing the tool on multiple fronts would be beneficial.
For instance, enriching the question database with climate change projections
specific to countries or regions would enable users to access tailored
information directly on the platform, addressing their specific queries about
the impacts of climate change in the places where they live or vacation.

Finally, the tool should progressively integrate advancements in the areas of
computer vision, allowing it to better integrate and interpret tables and
graphics, and generative AI. Some ideas of improvements specific to ClimateQ&A
have already been discussed in the literature Anthony Jude De-Gol (2023),
namely improving the prioritization of information, and the addition of expert
judgment.

## References

  * Anthony Jude De-Gol [2023] Adam J. P. Smith Marianne Aubin Le Quéré Anthony Jude De-Gol, Corinne Le Quéré.  Broadening scientific engagement and inclusivity in ipcc reports through collaborative technology platforms.  _Nature_ , 2023.
  * Bell et al. [2021] James Bell, Jacob Poushter, Moira Fagan, and Christine Huang.  In response to climate change, citizens in advanced economies are willing to alter how they live and work.  2021\.
  * Dupigny-Giroux and Cole [2018] Lesley-Ann Dupigny-Giroux and Amanda Cole.  Climate literacy and education.  _Oxford Bibliographies_ , 2018.
  * Henno [2023] Jacques Henno.  Ekimetrics cree un chatgpt de la crise climatique validé avec le giec.  _Les Echos_ , 2023.
  * Jürkenbeck et al. [2021] Kristin Jürkenbeck, Achim Spiller, and Maureen Schulze.  Climate change awareness of the young generation and its impact on their diet.  _Cleaner and Responsible Consumption_ , 3, 2021.
  * Lee et al. [2015] Tien Ming Lee, Ezra Markowitz, Peter Howe, Chia-Ying Ko, and Anthony Leiserowitz.  Predictors of public climate change awareness and risk perception around the world.  _Nature Climate Change_ , 2015.
  * Masson-Delmotte [2023] Valerie Masson-Delmotte.  Interroger les rapports du giec, July 2023.
  * Mediavilla [2023] Lucas Mediavilla.  Ekimetrics, l’intelligence artificielle et la data au service du climat.  _Le Figaro_ , 2023.
  * Monod [2023] Olivier Monod.  Qu’est-ce que climate q&a, la nouvelle ia pour lire les rapports du giec ?  _Liberation_ , 2023.
  * Spektor et al. [2023] Matias Spektor, Guilherme Fasolin, and Juliana Camargo.  Climate change beliefs and their correlates in latin america.  _Nature Climate Change_ , 2023.
  * United Nations Development Programme, University of Oxford [2021] United Nations Development Programme, University of Oxford.  People’s climate vote.  2021\.

##  Appendix A Annex A

ClimateQ&A is built of several technical algorithmic modules, OpenAI GPT 3.5
API being the last one (the generation of an answer). The core of Climate Q&A
consists of three steps, summarized in Figure 1: creation of a structured
dataset, question enrichment and sourcing, and generation & display in an
interface.

Figure 1: Climate Q&A global architecture

###  A.1 Step 1: Creation of a structured dataset from a set of heterogeneous
documents

The first step parses a heterogeneous set of documents (IPCC and IPBES
reports) and extracts information, which is stored in a structured database.
The algorithm uses OCR techniques for all the documents stored in any given
folder. The document structure is preserved, meaning that paragraphs are
linked to section titles, figures are linked to the passages that cite them,
etc. A structured database is thus created, containing all the information
extracted from the documents and the relationships between the different
entries in the database. These entries are then represented as a vector of
finite dimension. In ClimateQ&A, we use SentenceBert, but any type of
representation is possible. This will serve as the basis for a quick search in
the database for step 2.

###  A.2 Step 2: Enrichment of the user’s LLM query

In the second step, the user’s query is reformulated and enriched by searching
for similar passages in the structured database. The relevant question and
passages are encapsulated in a prompt, which also serves to limit the scope of
acceptable answers for ClimateQ&A. Within the step of enriching the user’s
query, the goal is to create a query on an LLM from the question, whose answer
will be formulated from the documents of interest (the structured database
from step 1). First, the user’s question is reformulated in a more
intelligible way for a LLM by asking the model to do it. Then, the new
question is compared to all entries in the database using the Faiss algorithm
developed by Facebook (Meta), known for its execution speed on data corpora up
to billions of entries. The most relevant entries, meaning those with the most
significant similarity to the question, are selected, filtered by a thematic
classification model (in this case, if the entry discusses climate) and
encapsulated in a prompt, necessary for querying an LLM (in our case, GPT-3.5
Turbo). The prompt contains other information, such as not going beyond the
scope of its knowledge and formulating its answer based on the selected
entries. The prompt created is then used to query an LLM.

###  A.3 Step 3 : Displaying the results

The query result (i.e.; the question) is displayed on an interface. The answer
is generated using the 10 most relevant references found in the corpus of
documents. The sources are displayed alongside the answer, so that the user
can verify the information and extract it for a report if necessary. The
display is based on two parts: the LLM’s answer with the notes serving as a
reference, as well as the sources used to formulate the answer. For
ClimateQ&A, we use the OpenAI Azure environment to send the request and get
the response. Just like the ClimateQ&A interface, the user can continue their
search by asking several questions in a row like a real chatbot.

##  Appendix B Annex B

Table 1 presents the results of the breakdown of the questions by topic.

Topics | Number of questions | Share of total
---|---|---
Climate Change & Greenhouse gas emissions | 1477 | 43.1%
Earth system | 416 | 12.1%
Energy | 358 | 10.5%
Citizens & behavior | 173 | 5.1%
Economics | 171 | 5.0%
IPCC-related questions | 170 | 5.0%
Companies & industries | 153 | 4.5%
Food & Agriculture | 104 | 3.0%
Feelings | 101 | 2.9%
Biodiversity | 101 | 2.9%
Politics & Policies | 70 | 2.0%
Circular economy | 67 | 2.0%
Technologies | 35 | 1.0%
Social issues | 29 | 0.8%
Total | 3425 | 100%
Table 1: Breakdown of questions asked to ClimateQ&A by topic

The topics in Table 1 are defined as follows.

Climate change and greenhouse gas emissions: Causes of climate change, how to
act against climate change, what is climate change, climate risks and impacts,
carbon emissions, temperature projections, evidence of climate change, impacts
of climate change in France…

Earth system: Sea level rise, ocean currents, water, forests, permafrost,
icecap melt, glaciers, planetary boundaries.

Energy: Electric vehicles, renewable energies, nuclear energy, carbon capture
& storage, hydrogen, fossil fuels, biochar & bioenergies, batteries.

Citizens & behavior: Individual action, food & transportation habits,
effectiveness of individual action.

Economics: Impact of different sectors of activity, welath & carbon footprint,
degrowth, green economy, financial system.

IPCC-related questions: Questions about the reports & authors, the main
conclusions, the existence of a consensus.

Companies & industries: Sustainability & CSR, role of companies in the
transition.

Food & Agriculture: Agricultural practices, animal products, risks to specific
agricultural products.

Feelings: Death & survival, climate sceptics.

Biodiversity: Link between climate change & biodiversity loss, impacts of
biodiversity loss, mass extinctions.

Politics & Policies:International agreements, political parties & ideologies.

Circular economy: Waste management, recycling & waste.

Technologies: Digital technologies and AI.

Social issues: Gender issues and social justice.

Figure 2: Illustration of the breakdown of the questions by topic and subtopic