# Forms of Understanding of XAI-Explanations

Hendrik Buschmeier<EMAIL_ADDRESS>Heike M. Buhl  Friederike Kern
Angela Grimminger  Helen Beierling  Josephine Fisher  André Groß  Ilona
Horwath  Nils Klowait  Stefan Lazarov  Michael Lenke  Vivien Lohmer  Katharina
Rohlfing  Ingrid Scharlau  Amit Singh  Lutz Terfloth  Anna-Lisa Vollmer  Yu
Wang  Annedore Wilmes  Britta Wrede

###### Abstract

Explainability has become an important topic in computer science and
artificial intelligence, leading to a subfield called Explainable Artificial
Intelligence (XAI). The goal of providing or seeking explanations is to
achieve (better) ‘understanding’ on the part of the explainee. However, what
it means to ’understand’ is still not clearly defined, and the concept itself
is rarely the subject of scientific investigation. This conceptual article
aims to present a model of forms of understanding in the context of XAI and
beyond. From an interdisciplinary perspective bringing together computer
science, linguistics, sociology, and psychology, a definition of understanding
and its forms, assessment, and dynamics during the process of giving everyday
explanations are explored. Two types of understanding are considered as
possible outcomes of explanations, namely _enabledness_ , ‘knowing how’ to do
or decide something, and _comprehension_ , ‘knowing that’ – both in different
degrees (from shallow to deep). Explanations regularly start with shallow
understanding in a specific domain and can lead to deep comprehension and
enabledness of the explanandum, which we see as a prerequisite for human users
to gain _agency_. In this process, the increase of comprehension and
enabledness are highly interdependent. Against the background of this
systematization, special challenges of understanding in XAI are discussed.

###### keywords:

understanding , explanations , explainable AI , interdisciplinarity ,
comprehension , enabledness , agency

††journal: Cognitive Systems Research

[ubi]organization=Bielefeld University, addressline=Universitätsstraße 25,
city=Bielefeld, postcode=33602, country=Germany

[upb]organization=Paderborn University, addressline=Warburger Straße 100,
city=33098, postcode=Paderborn, country=Germany

[trr]organization=SFB/Transregio 318 Constructing Explainability,
city=Paderborn & Bielefeld, country=Germany

##  1 Introduction

Due to recent technological developments, explainability has become a widely
discussed topic in computer science and artificial intelligence, giving rise
to a new subfield called Explainable Artificial Intelligence (or XAI for
short). It is also a highly interesting topic in other disciplines, such as
cognitive science, psychology, (psycho-)linguistics, and education. The focus
of the article is understanding as knowledge resulting from explanation. From
this perspective, understanding is the goal of a successful explanation (Keil,
2006, p. 229) and “recipients of explanations, if the explanations are at all
successful, are expanding their understanding in real time”. Understanding has
implicit parts (Wilson and Keil, 2000, p. 97) regarding its procedural and
non-reflective components. In the course of explanations, levels of
understanding increase, but understanding always remains incomplete (Miyake,
1986) and is not always factive (Páez, 2019), among other reasons because
explanations are often shallow, depending on the knowledge and motivation of
the explainer and explainee (Wilson and Keil, 2000), and they have “relatively
unclear end states” (Keil, 2006, p.242). While the goal of explanation is to
lead to understanding, we know little about how understanding can be
conceptualized, what forms of understanding can occur, and, finally, how it
can be measured. In this article, we propose a four-field model of
understanding that provides conceptual clarification of related terms and
notions, and thus distinguishes between different forms and levels of
understanding. Our goal is to develop a model that can be used as a heuristic
to capture, analyze, and even measure different forms of understanding in the
context of XAI and beyond.

The starting point of our considerations is a view of explanation as a co-
constructive process in which both humans and AI systems contribute to an
explanation Rohlfing et al. (2021). Accordingly, we adopt a broad notion of
‘understanding’ as implemented in and underlying everyday interactions and
experiences with algorithmic technology in socio-technical AI systems. In
contrast to other research on explanation, we do not focus on scientific
explanations but on ’everyday explanations’ because they show a greater
variety with regard to the knowledge they mediate (Keil, 2006). While
scientific explanations are regularly characterized by _why_ -questions (e.g.,
Chi et al., 1994), and thus aim primarily to convey causal knowledge, everyday
explanations often follow other kinds of questions, such as _what_ , _where_ ,
_how does it work_ , etc. (Klein, 2009). Moreover, the quality of everyday
explanations is more difficult to define – they can be very simple,
incomplete, inaccurate, or, in contrast, quite sophisticated (Johnson-Laird,
1983; Norman, 1988).

The article thus aims to provide a synthesis of interdisciplinary perspectives
on ‘understanding’ as a result of explanation, which is highly relevant for
example in tutoring contexts (e.g., Chi et al., 2004; Miyake, 1986), classroom
interaction (e.g., Koole, 2010; van de Pol et al., 2018; Martins and Gresse
von Wangenheim, 2022), and XAI, while acknowledging that ‘understanding’ is
hardly considered as a research topic in its own right in many of the
individual disciplines. For example, understanding is not a topic in
experimental psychology, nor is there a scientific concept of it. The closest
related research areas are knowledge representation/memory and
reasoning/problem solving. However, we would argue that a model that attempts
to define and unify basic concepts is a useful tool across disciplines because
it can provide a starting point for both further empirical analysis and theory
development. While the proposed model is primarily concerned with cognitive
aspects, we will also argue that embodied interaction should be included. With
such a claim, we intend to expand the notion of understanding from a purely
cognitive conception to one that includes bodily aspects. The model will be
enriched by offering a theoretical account of the assumed dynamics of
understanding by sketching how movement between the four domains can be
conceptualized.

###  1.1 Basic concepts and first examples

In this paper, _explaining_ is conceptualized as a process in which an
_explainer_ and an _explainee_ interactively and iteratively _co-construct_ an
_explanans_ of an entity that is being explained (the _explanandum_). The
explainer is an agent, which can be either a human or a machine, that drives
the explanation. Both partners are actively involved in the explanation
process through mutual monitoring and scaffolding (Rohlfing et al., 2021). The
goal of the explanation process is for the explainee to _understand_ the
explanandum with respect to their goals or purposes.

The different forms of understanding that explainers aim at or strive for are
reflected in the heterogeneity that can be found especially in everyday
explanations. In order to deal with this diversity, we have developed a model
that considers two types of understanding as the result or goal of an
explanation: _comprehension_ and _enabledness_. In addition, the model
conceives of these types as further differentiated by the level of
understanding that can be achieved, which we label _shallow_ and _deep_
respectively (cf. Graesser and Person, 1994, p. 127)). It should be noted that
our model, like any model, is simplified (Bailer-Jones, 2009) in order to
provide a systematic examination of different forms.

The term _comprehension_ can be roughly translated as ‘knowing that’, which at
best is a conceptual framework for a phenomenon that goes beyond what is
immediately perceivable. Regarding the distinction between shallow and deep
levels of understanding, we assume that forms of understanding can also be
either very simple/shallow forms, or rather deep.

In contrast, the term _enabledness_ means ‘knowing how’. Again, enabledness
can be very simple (‘to start an Internet search, enter the search term in
Google’) or deep, e.g., knowing how to program a computer game. Thus, a person
can be enabled to perform either a simple routine action or a complex action.
However, complex actions regularly require some form of comprehension to be
enabled. We suggest that comprehension and enabledness are generally
interdependent.

In this article, we will use two examples to illustrate understanding (see
examples 1 and 2): In the following, we would like to explain the terms
introduced above using example 1 (example 2 will be used later on). It looks
at understanding in a ‘technical’ context, namely the use of a stopping
criterion in an iteration loop of a computer program. More specifically,
example 1 is an illustration to learn how to use _while-loops_ in computer
programming. While-loops are constructs of programming languages that execute
the code within the construct as long as the while-condition attached to the
construct holds. In the example, a variable counter is decremented by $1$ as
long as its current value is greater than $1$. The specific application of the
example is to print the well known ‘99 bottles’
song111https://en.wikipedia.org/wiki/99_Bottles_of_Beer (Byrd, 2010). While-
loops are known to cause problems for computer programming students (Qian and
Lehman, 2017).

1:$counter\leftarrow 99$ $counter>0$ 2:$counter$ “ bottles of liquid on the
wall,”  3:$counter$ “ bottles of liquid;”  4:“If one of those bottles should
happen to fall,”  5:$counter\leftarrow counter-1$ 6:$counter$ “ bottles of
liquid on the wall.”  List of examples 1 A simple example illustrating a
while-loop. This example looks at a specific construction of natural language
use, specifically the English _present progressive_ tense. The present
progressive is constructed using “the present tense form of be + lexical verb
in -ing form” (Carter and McCarthy, 2006, p. 598) and is used for, among other
things, “events in progress at the time of speaking”, “repeated events in
temporary contexts”, “processes of change”, and “with adverbs of indefinite
frequency” (Carter and McCarthy, 2006, p. 601f), i.e., it expresses aspect
grammatically. These usage contexts may be intuitive to native English
speakers, who may also be able to see the nuanced differences in usage
compared to the _present simple_ tense. However, learners of English as a
second language, especially speakers of languages where there is no
grammatical aspect (e.g., standard German), need to understand how and when to
use the present progressive and may not even have the concepts (e.g., that
progressivity is grammatically marked) to begin with. List of examples 2
Grammatical aspect in natural language use.

Shallow Comprehension

of the program would entail that it will print out the ‘99 bottles’-song on
the computer screen. Specifically, regarding the while-construct, shallow
comprehension could involve understanding that the indented lines (3–7) are
executed every time the while-loop is entered and that counter $>$ 0 in line 2
is the condition under which the while-loop is executed.

Deep comprehension

would go beyond this and mean to understand that the condition of the while-
loop actively interferes with the while-loop: the variable counter is
decreased and the condition reacts to this. Deep comprehension may also detect
a shortcoming of the program, i.e., that the lines of the song are not printed
correctly when the counter equals one, as the grammatical agreement (number)
between number and noun is violated (*“1 bottles”). Going beyond this program,
deep comprehension would also give the explainee the insight that the while-
condition could also be used for different applications, e.g., to try to
access an internet resource over an unreliable connection until the full
resource has been downloaded.

Shallow enabledness

would allow an explainee to change the program, so that it would, for
instance, start at 77 bottles or decrease by 2 each step or stop early.

Deep enabledness

would allow an explainee to resolve the ‘1 bottles’-problem mentioned above by
rearranging lines and combining multiple while-loops.

Before presenting the model of understanding in full, we assume that the
notion of knowledge should be clarified beforehand. The goal of explanations
is to construct knowledge (Miller, 2019; Quasthoff et al., 2017). To
understand something as a result of an explanation is to know something about
it. Our own attempt to systematize understanding on the basis of the
distinctions introduced above – shallow/deep and comprehension/enabledness –
thus builds on previous scholarly work on knowledge and learning objectives.

##  2 Aspects of Understanding

For our model of understanding, we follow primarily two conceptualizations:
the types and qualities of knowledge as described in detail by de Jong and
Ferguson-Hessler (1996), and the taxonomies of learning from the field of
education (‘Bloom’s taxonomy’; Anderson and Krathwohl, 2001; Krathwohl et al.,
1969).

de Jong and Ferguson-Hessler (1996) were interested in knowledge in
instruction and learning and especially as a prerequisite for problem solving.
They distinguished four types of knowledge: conceptual, procedural,
situational, and strategic. In terms of understanding as a goal and the
achievement of explanations, two types of knowledge are particularly
important: _conceptual knowledge_ is “knowledge of facts, concepts, and
principles” (p. 107), whereas _procedural knowledge_ refers to “actions and
manipulations” that transform one state into another. Thus, in our
terminology, conceptual knowledge describes ‘knowing-that’ and procedural
knowledge refers to ‘knowing-how’. In the context of XAI, we focus on
cognitive forms of procedural knowledge, such as written division and
programming. Actions and skills with a higher motor component, such as cycling
or knitting, may require other forms of more embodied explanations (showing).

Here we focus on ‘understanding’ as a form of knowledge that is the result of
a given explanation. Despite the functional proximity between the notions of
conceptual knowledge and comprehension, and procedural knowledge and
enabledness, we prefer to use both _comprehension_ and _enabledness_ in the
context of everyday explanation to emphasize their association with
explanation. The achievement of explanations often goes beyond a specific
domain and situation because explanations are tailored to the needs of the
explainee and the situation. With the term ‘comprehension’ we also relate to
the work of Kintsch (e.g., 1998), who uses ‘comprehension’ for the process of
comprehension in the area of text processing.

Building on the taxonomy of _qualities of knowledge_ also developed by de Jong
and Ferguson-Hessler (1996), we consider the following characteristics to be
relevant for describing understanding:

Level

Both conceptual and procedural knowledge are described as more or less shallow
or deep. Regarding conceptual knowledge, de Jong and Ferguson-Hessler (1996,
p. 111) illustrate the distinction with symbols or formulas on a shallow level
vs. concepts and their relations on a deep level. Thus, deep comprehension
most likely reflects highly interconnected and elaborated concepts. Shallow
comprehension would correspond to little interconnectedness, but also to few
concepts. In terms of procedural knowledge, there are rules or recipes at a
shallow level in contrast to meaningful actions at a deep level.

Explainees may require different levels of understanding, and explanations
should be tailored to those needs (Wilson and Keil, 2000). Thus, it is not
always necessary – or even annoying – to give or receive a long and complex
explanation that leads to deep understanding. Often the explainer is satisfied
with a single action or a simple fact.

Structure

Knowledge can be more or less structured (isolated elements or meaningful
structure). The distinction between knowledge and understanding of novices vs.
experts (see Section 4.1) can be captured by describing how elements might be
structured: The knowledge structure of experts is more hierarchical and better
organized than that of novices. Although level and structure are closely
related, an explanation can also lead to a false structure at a very deep
level.

Automation

Automation of knowledge is particularly important for understanding and
therefore for explanation. Consider how difficult it is to explain simple
skills like riding a bicycle. This is because there is a strong distinction
between declarative conceptual knowledge and non-declarative compiled
procedural knowledge. Compilation enables automatic processes.

We conceive comprehension as involving conceptual knowledge, i.e., remembering
facts about and experiences with various concepts such as theories and/or
related routine actions. In contrast, enabledness is conceived as the ability
to perform routine and/or complex actions (mentally and physically) in a
competent and flexible manner. Furthermore, whereas comprehension is always
‘explicit knowledge’ that can be acquired through books or explicit verbal
instruction, enabledness consists at least in part of ‘tacit knowledge’ that
is not easily shared (Lim and Ahmed, 2000).

However, conceptual knowledge can also be compiled, e.g., rote learning of
definitions or mnemonic sentences instead of deep comprehension. On the other
hand, deep enabledness requires declarative, mostly deep comprehension. In
addition to implicit learning of procedures as a basis for skills, e.g.,
native language grammar, there is explicit learning, e.g., adult learning of a
foreign language. Explaining how to drive a car is a good example of how
procedures are first explained in a declarative way, and then the steps are
proceduralized and composed over the years (Anderson, 1983). de Jong and
Ferguson-Hessler (1996) explain automation by mentioning tacit knowledge that
cannot be expressed. In order to give an explanation, the explainer needs non-
tacit knowledge, also of procedures. In the following sections, this
intertwining of understanding quality and type will be discussed in detail.

Modality

Regarding modality, we added embodiment to the verbal and pictorial aspects
given by de Jong and Ferguson-Hessler (1996). Embodied knowledge is
conceptually similar to procedural knowledge (Tanaka, 2011, p. 150), and it
can be better performed than explained verbally. Embodiment plays an important
role in learning situations where new members of a (scientific) community are
taught to become skilled practitioners. Visibly embodied displays (e.g., how
to hold a pencil, how to hold a Munsell color chart next to a piece of dirt to
describe its color) are part of the knowledge conveyed in such situations
(Goodwin, 2013).

Generality

Regarding generality, explanation and understanding are domain specific.
However, it is important to keep in mind for the following discussion that an
explainee may be a novice with no understanding in one domain, but an expert
in another. Moreover, the given domain can be very specific (e.g., the
motivation of a particular person in a particular situation) or on a more
general level (e.g., the motivation of people – even living beings).

In addition to this characterization of different types of knowledge and their
manifold qualities, as a further conceptualization, taxonomies of learning
objectives help to describe the types and levels of understanding (e.g.,
Krathwohl et al., 1969; Anderson and Krathwohl, 2001). Anderson and Krathwohl
2001 (summarized in Anderson 2002) systematize educational objectives in two
dimensions, the structure of the knowledge and the cognitive processes that
are to be done with the knowledge.

Structure of knowledge

The structure of knowledge aims at a differentiation of the above mentioned
‘types of knowledge’. It includes factual, conceptual, procedural, and
metacognitive knowledge. While factual knowledge contains the basic elements
of a domain (e.g., definitions), conceptual knowledge contains the structure
of the basic elements, e.g., principles, models, and theories. Comparable to
de Jong and Ferguson-Hessler (1996), procedural knowledge aims at ‘how to do
something’, such as skills and algorithms. The fourth form of knowledge,
metacognition, will be neglected in our paper, because it is only of interest
in special explanations regarding (own) cognitions. Thus, the categories of
knowledge are similar to the types of knowledge discussed by de Jong and
Ferguson-Hessler (1996) with the additional hint that there are very basic
elements of knowledge. In terms of understanding, factual and conceptual
knowledge aim at comprehension, while procedural knowledge, of course, aims at
enabledness.

Cognitive processes

The second dimension of the taxonomy of learning aims at increasing the levels
of _cognitive processes_ that can be done with knowledge, namely remembering,
understanding (note that this is a different use of the term), applying,
analyzing, evaluating, and creating (Chi and Wylie, 2014). This strengthens
the aspect of the level of knowledge introduced by de Jong and Ferguson-
Hessler (1996). There is a gain in complexity from low level cognitive
processes like remembering to applying to high level processes like creating
something new – resulting in a hierarchy from shallow to deep understanding.

Building on these taxonomies, we focus on the two types of understanding
introduced above (comprehension and enabledness) and the different qualities
(levels) associated with them. We will use the terms shallow and deep
comprehension on the one hand, and shallow and deep enabledness on the other.
The conceptualization in the form of four fields – as sketched in the diagram
in Figure 1 – is only for simplicity. We see the main advantages of the model
in its ability to map the process of understanding at the different stages of
the explanation process.

|  Shallow Enabledness |  Deep Enabledness
---|---|---
Shallow Comprehension |  Sect. 3.1 |  Sect. 3.2
Deep Comprehension |  Sect. 3.3 |  Sect. 3.4
Figure 1: Types and levels of understanding as a four field diagram

To avoid misunderstandings of the four-field diagram, it is important to note
that the distinction between shallow and deep understanding is not dichotomous
- there are fluid transitions between the two extremes - nor are comprehension
and enabledness assumed to be independent (orthogonal) categories. The level
of depth depends on the domain, the given explanation, and the goal being
pursued. Accordingly, there is no static definition of deep understanding;
instead, a change in depth is likely to occur as soon as either the goal or
the domain is changed. These characteristics are discussed in the following.

In summary, in explanations, explainers and explainees co-construct
understanding, primarily of the explainee, but also of the explainer. There
are two forms of understanding - comprehension and enabledness. Both forms of
understanding can be on different levels, from very shallow to very deep
understanding. The four resulting combinations (cf. Figure 1) are intermediate
steps or final achievements of explanations with the possible goal of agency,
the ability to control and adapt the (digital) environment (Passey et al.
2018; Section 3.4). The combinations will be characterized in the next
section. On this basis, (in Section 4) we explain dynamics of comprehension in
explanations, taking into account two relevant aspects: We consider
differences between novices and experts in explanations and discuss how in the
course of an explanation comprehension and enablement are intertwined.

##  3 Forms of Understanding

In the following, we will explain how understanding can be characterized for
each of the four ‘fields’ on the basis of examples 1 and 2 introduced above.
The respective knowledge is clarified additionally by means of its assessment.
Due to their subject matter and also the amount of available research on it,
the sections differ in their extent.

###  3.1 Shallow comprehension, shallow enabledness

We see the combination of shallow comprehension and shallow enabledness as a
potential starting point for understanding. Thus, this section will focus on
the mental state before a change in understanding, or here, before
explanation. The first step is to define the underlying conditions and
premises of this stage. Any process of understanding should begin with and be
tailored to the explainee’s mental representation model, which is built from
(prior) own and mediated experiences with the domain and prior knowledge about
familiar things. At the beginning of an explanation, explainers and explainees
need to establish ’common ground’ (Clark, 2014, p. 109), which involves
assessing “what the other person knows at the start, then tailoring one’s own
contributions to add to the relevant common ground for that occasion […]”.
Common ground is usually based on the prior knowledge that explainees bring to
the explanation process. Prior knowledge thus influences the course of the
explanation and the understanding that follows: Explainers can build their
explanation on it, and explainees’ mental processing of new knowledge is
influenced by it (Chi et al., 1994). Therefore, it is particularly interesting
to know what prior knowledge is and which parts of it are relevant for an
explanation, and how the explainer can identify these relevant parts of prior
knowledge. To clarify the situation in this field, we begin with the two
examples.

#### Example 1

An example of people having shallow comprehension and shallow enabledness of a
stopping criterion in an algorithm might be novice programmers. Their level of
understanding might, for example, be characterized by prior knowledge of and
basic familiarity with the syntax of the conditional statement in while-loops
and using it to control for a specific number of iterations (e.g., do this
exactly three times), but not knowing that it can also be used in a more
dynamic way (a < b) (shallow comprehension) or how to set up such conditions
generically (shallow enabledness).

#### Example 2

An example of persons having shallow comprehension and shallow enabledness of
the English present progressive tense could be beginning learners of English
as a second language. Depending on their first language, they (may or) may not
have prior knowledge of the function of the grammatical constructions. Their
level of understanding of the present progressive tense could be characterized
as being able to construct simple sentences using the present progressive
tense of a number of verbs because these verbs have been introduced to them in
the -ing form (shallow enabledness), but they are not aware that it is a
marker of a specific tense and do not know when to use it (shallow
comprehension).

First, we want to describe the dimensions of understanding and the possibility
of going through the field of shallow comprehension and shallow enabledness.
To introduce changes in the model of understanding, we use the two dimensions
of comprehension and enablement as parts of this model.

We can imagine that a novice in the field of shallow comprehension and shallow
enabledness has low comprehension of programming, does not understand the
syntax of lines of code completely, does not know any concepts of iteration,
and is not able to formulate it generally. However, improving the novice’s
understanding requires fundamental knowledge about the meaning of the code or
the ability to interact with the algorithm to observe changes in the result.
At the beginning of an explanation there are cognitive and motivational
prerequisites of understanding.

#### Prior Knowledge

Prior knowledge is thought to influence co-construction between the explainer
and the explainee (Gentile et al., 2021) because the explainer relies on the
explainee’s currently assumed prior knowledge to scaffold the explainee’s
understanding. However, a greater amount of prior knowledge on the part of
both the explainer and the explainee does not necessarily lead to better
explanations (Williams and Lombrozo, 2010).The influence of prior knowledge in
explanations seems to be similar to its influence in learning scenarios: prior
knowledge is not necessarily beneficial for the explanation process
(Simonsmeier et al., 2021). It was found that the effect of explanation and
the amount of prior knowledge on learning are neither independent nor
subadditive (Williams and Lombrozo, 2010). Rather, they appear to have a
nonlinear influence on learning. An explanation guides the focus in acquiring
new knowledge, and prior knowledge helps to identify patterns and constraints.
Not all prior knowledge is used as by explainers or explainees, only parts
relevant to a task are activated (Dochy and Alexander, 1995). However, these
‘relevant’ parts are relative to the individual holding the prior knowledge
(Alba and Hasher, 1983). Thus, the personal relevant knowledge is not
necessarily identical for the explainer and the explainee; neither is the
assumed prior knowledge of the explainee on the side of the explainer
necessarily correct. The assumed knowledge is part of a mental representation
that the explainer has of his interaction partner, the partner model (Brennan
and Hanna, 2009).

#### Assessment of Prior Knowledge

The literature presents a variety of methods for the scientific assessment of
prior knowledge. These include verbal methods such as ‘think-aloud’ tasks,
verbal protocols and audit trails, and performance metrics on problem solving
and troubleshooting or retention over time, as well as methods specific to
human-computer interaction, system usage, user explanations of the system, and
user prediction of system performance (Sasse, 1991). The simplest method is to
query only the domain-specific prior knowledge (Best et al., 2004). However,
this method discards most of the total prior knowledge.

An alternative approach is presented by Cole et al. (1986), who use a
combination of three different questionnaires to assess prior knowledge. The
first questionnaire tests prior knowledge using true/false questions about
domain-specific conceptual knowledge. In addition, they ask respondents to
estimate their prior knowledge, such as “How well do you know _X_?” The last
questionnaire tests prior knowledge by asking about familiarity with the
domain, e.g., “How often have you used _X_?” In addition, the authors provide
a questionnaire that asks about attitudes towards the domain, as they
influence the explanation (Tobias, 1994) as much as the actual knowledge the
participants bring to an explanation.

In addition to prior knowledge, the motivation brought to an understanding
process also influences its course. Wade and Kidd (2019), for example, have
shown a bidirectional relationship between curiosity and learning. Therefore,
the motivational prerequisites for comprehension will be discussed in the
following.

#### Motivational prerequisites

Regarding explanations, the question is what motivates explainees to establish
understanding of a specific domain as a goal. Their motivation can be
decomposed into two aspects: The social motivation to follow the explanation
and the motivation to understand. Consider the situation of students at
school. Certainly, they are not always motivated by interest, achievement, or
usefulness of the subject, but they know that it can be very useful (or even
interesting) to follow the teacher’s explanation.

To identify motivational prerequisites, a classical approach describes
motivation as influenced by expectancy and value. Expectancy refers to
“beliefs about how well they will do at the activity” (Wigfield and Eccles
2000, p. 68, cf. Eccles and Wigfield 2020). The subjective value of
understanding is divided into at least two parts: understanding can have (1)
an intrinsic value of the activity of understanding, arising from an interest
in the explanandum or a relation to the explainee in the social situation, or
(2) understanding the explanation can be useful or valuable for achieving a
goal, and thus has an extrinsic value that does not lie in the action itself.
For example, understanding digital artifacts is useful or even required in the
context of school, university, and the labor market (e.g. Ball et al., 2017).
Expectation as well as value depend on the previous experiences of the
explainee (Bonanati and Buhl, 2021). With the experience that it is difficult
to follow an explanation or that digital artifacts are difficult to
understand, one’s self-efficacy to understand XAI explanations is likely to be
low. On the other hand, an adequate explanation increases the self-efficacy to
use (digital) technology (Weitz 2021; cf. technology acceptance models: Davis
1989, Marangunić and Granić 2014). Both expectancy and value depend on the
explainee’s goals (Eccles and Wigfield 2020, cf. Covington 2000), such as
learning the present progressive tense for a test in school or for living in a
foreign country.

###  3.2 Shallow comprehension, deep enabledness

Although one can think of specific skills or cases of this form of
understanding in its most extreme form, as we show in Examples 1 and 2 below,
we argue in this section that there are not many cases of more complex actions
following explanations that can be performed at the deepest level of
enablement and still have shallow comprehension. Instead, we argue for actions
that build on each other, that an increase in enabledness will lead to an
increase in comprehension, and that these dimensions of understanding are
intertwined. Thus, we also address the transitions between types and levels of
understanding. We begin by describing how deeper enabledness can be achieved
even when comprehension is rather shallow. In order to describe the processes,
we also briefly refer to shallow enabledness, thus adding to the previous
Section 3.1.

#### Example 1

An example of people who have only a shallow comprehension but a deeper form
of enabledness of a stopping criterion in an algorithm could be experienced
self-taught programmers who lack formal training in logic and algorithms, but
who can easily motivate their formulation of the stopping criterion for a
while-loop (shallow comprehension). However, using their experience and
possibly automated knowledge, they are able to get the conditional expressions
right (deeper form of enabledness)..

#### Example 2

An example of people with shallow comprehension and deep enabledness of the
English present progressive tense would be native English speakers who
intuitively know which form of the present tense to use and can construct it
correctly (deep enabledness). However, if they were asked to explain their
choice of the present progressive tense, they would not be able to do so
(shallow comprehension).

Besides the motivational prerequisites already explained (Section 3.1, the
access to external resources (e.g., tools such as web search, chatbots, FAQs,
other people) may be relevant to be enabled to perform an action without
necessarily having a conceptual comprehension of the task. One example is to
program another while-loop and make a query to a chatbot (as an external
resource) to support this task. In this sense, a _deeper enabledness_ is
characterized by flexibility and adaptivity in the use of the external
resources (e.g., novices vs. experts, see Section 4.1). The more flexibly an
explainee uses the resources and adapts their use to different contextual
environments, the deeper is the enabledness. Flexibility here means (i)
various resources are known, and (ii) contextual factors (such as time
restrictions, current (un)availability of an external resource etc.) are taken
into account to choose the appropriate resources in a certain situation.
Adaptivity means the ability to select the appropriate resources in novel
contexts/in different contextual environments. In this respect, deep
enabledness allows someone to use their skills in a wider range of
applications, and without the need for resources being present in the
immediate environment, because they can be simulated using internal mental
representations, e.g., by memorization (see below). However, the deepest form
of enabledness cannot be reached without any comprehension, since choosing
what the best/most helpful resource is in a certain situation, or making a
concrete query when using systems such as web search or chatbots, certainly
presupposes some conceptual knowledge of the problem or task (cf. Graesser and
Person 1994 on the relation between asking deep-level questions and deepening
of conceptual knowledge).

In contrast, _shallow enabledness_ encompasses rather basic actions of an
individual. This has, for example, been described for early infant
development. Mandler (2012) proposes that an infant might be able to identify
an object as belonging to a category but this cognitive operation is based on
perceptual information only and does not require an interpretation of
spatiotemporal information. A basic action is embedded in the external
representations itself and is externally driven (i.e., there is a perfect
match between the physical disposition and the environment; see
Identification/recognition in the listed abilities). In contrast to deeper
enabledness, only a limited number of external resources are used and they are
used context-dependent, in a given way (e.g., using the same query again) and
cannot be transferred to another or a changed context. The _transition from
shallow to a deeper enabledness_ comes by experience, i.e., a higher degree of
flexibility and adaptivity is achieved by richer experience (through
repetition in the same or only slightly changed contexts) and a far transfer
of skill (Gardner, 1999). Enabledness without deeper comprehension may also be
achieved by memorizing a number of steps, procedures or facts (rote learning)
without understanding the concepts behind it. However, this approach to
enabledness, that Boaler and Greeno (2000, p. 9) call a “ritualistic act of
knowledge reproduction”, may hinder progress in deepening comprehension
because explainees may be less engaged with the underlying concepts. Deeper
comprehension, however, requires “thinking practices” (Boaler and Greeno,
2000, p. 9, for mathematical learning). Therefore, we assume that deeper
enabledness as we have defined it, including a flexible use in different
contexts, cannot be accomplished by memorization or rote learning.

To define _comprehension_ , among others we made use of the taxonomies
proposed by Bloom (Krathwohl et al., 1969) and Krathwohl (Anderson and
Krathwohl, 2001; Krathwohl, 2002) to classify learning objectives. They
describe different cognitive processes that a learner is able to perform
within a learning process. However, originally there is no clear distinction
between enabledness and comprehension in these learning taxonomies: Bloom’s
taxonomy (Krathwohl et al., 1969) starts with early processing mechanisms that
we ascribe to enabledness but not comprehension, whereas Krathwohl’s taxonomy
(Anderson and Krathwohl, 2001; Krathwohl, 2002) seems to require
interpretation and memory from the beginning. To make the early steps of
understanding clear, we will therefore refer to Bloom’s taxonomy. Overall, we
recognize that there is a progression from being bound to perceptual
information or the context to decontextualized knowledge that is more flexible
and adaptive. We view the boundedness to external resources not only to be
more shallow in terms of enabledness (see above) but also in terms of
comprehension. Therefore, the abilities described in the first two stages on
Bloom’s taxonomy seem to belong to shallow comprehension/shallow enabledness
(see Section 3.1). It is difficult, however, to make a clear cut between
comprehension and enabledness because the different forms of comprehension
only become observable by abilities, and therefore by forms of enabledness.

Identification/recognition

Krathwohl et al. (1969) identified the following abilities as reflecting the
first steps of comprehension: Repeating, listing, labeling, completing, and
selecting. We propose that all of them afford a more or less prepared context
(i.e., external resources) in which the knowledge is triggered to be applied.
According to Mandler (2012), the operations at this level may be based on
perceptual information only and do not require an interpretation of the
context. For the other abilities, based on the rich context, only little
knowledge or comprehension is required to achieve an act (e.g., to label
something or to fill something out).

Contextualization: Connection to the obvious/present

The second level, called ”understanding” in Krathwohl’s taxonomy (Anderson and
Krathwohl, 2001), represents, among others, the abilities of interpreting,
exemplifying, and summarizing. We propose that they require some knowledge to
be connected to another entity that is obvious or present in the environment
(i.e., contextualization), thus requiring a deeper comprehension than in the
first stage. Contextualization was recognized as a form of interpretation of
an activity based on cues from the obvious/present and related to the process
of inferencing, which itself depends on the context of its occurrence (Auer, ,
p. 30). For our example 1, a reasonable explanation for why the first result
is ‘99 bottles …’ is that counter is initalized with ‘99’. This explanation
simply accounts for what can be given by referring to the present context,
without having to add anything to it. A deeper reason goes beyond the obvious
and offers a connection to what cannot be perceived, such as what the output
would be if the code in line 1 or 6 were changed in a certain way. Making
connections is important in the learning process (Gentner et al., 2003).

Applying (transitional step towards deep comprehension)

We see the third step of comprehension at the intersection of shallow and
deep, i.e., as intermediate, because it requires applying the knowledge to a
context that is not necessarily prepared for it (i.e., the context does not
trigger a specific application). The application of knowledge always leads to
deeper enabledness.

All of the three above-mentioned steps may be time-dependent – comparable to
cognitive mechanisms of mapping discussed for word-learning in infancy
(Wojcik, 2013), i.e., fast-mapping (short-term; less demanding) and slow-
mapping (requiring long-term memory; more demanding), or to different forms of
cognitive engagement of students (Chi and Wylie, 2014, p. 221). Chi and Wylie
(2014) differentiate between recall (what we see as shallow comprehension),
apply (at the intersection from shallow to deep comprehension), transfer and
co-create (deep comprehension).

In the following, we argue that comprehension and enabledness are intertwined.
We presume that most types of deep enabledness would build on some kind of
comprehension, and, vice versa, deeper enabledness might lead to increased
comprehension: (i) comprehension changes depending on enabledness and vice
versa, and (ii) comprehension, even shallow, only becomes observable in
physical and cognitive abilities that are also signs of enabledness (cf.
Section 4.2). Cases of extreme shallow comprehension together with extreme
deep enabledness, or cases of shallow comprehension and intermediate
enabledness are therefore hard to be defined and would contradict all
propositions made above. (iii) There is neurological evidence on language
production, according to which comprehension and enabledness cannot be torn
apart at the cognitive level because of shared underlying neurological
representations, at least from a functional perspective (e.g., Pickering and
Garrod, 2013).

In terms of _assessability of comprehension_ , the question is how
verbalization is related to the degree of comprehension. This question arises
based on studies on gesture use and learning (cf. extensive work by Goldin-
Meadow and colleagues, e.g., Goldin-Meadow 2017 for an overview): If
comprehension is shallow, a person may not be able to verbalize knowledge,
i.e., the knowledge is more implicit. However, Goldin-Meadow and colleagues
have shown that other (communicative) behavior such as a person’s gestures can
indicate some initial comprehension: even though some learners failed to solve
some mathematical problems and were unable to explain their solution, their
co-speech gestures conveyed semantic information that was relevant to
correctly solve the task but that was not verbalized. Thus, it was assessable
nonverbally. The finding that these learners were more likely to benefit from
further explanations was interpreted in terms of them having implicit
conceptual knowledge. Research on implicit conceptual knowledge has further
shown that participants were able to make judgments on whether a problem-
solving task is likely to be solvable or not, even when they were not able to
provide a correct solution themselves (Bowers et al., 1990). When
comprehension becomes deeper, it becomes explicit, that is, it can be
verbalized, and is therefore assessable by asking questions.

#### Intertwinement of shallow comprehension and deep enabledness

As mentioned above, we assume a strong intertwinement between comprehension
and enabledness. However, it should not be understood as a linear increasing
relationship. Instead, the following three cases, see Figure 2, display the
dynamic relation between both forms of understanding.

The first case is a process at the transition from intermediate to deeper
enabledness with alternating more shallow and deeper comprehension (solid line
in Figure 2), that may often occur during automation, e.g., in language (L1)
acquisition. If something becomes automatized, enabledness becomes deeper, but
comprehension may decrease (Anderson, 1996; Boaler and Greeno, 2000), as
during the process of automation conceptual knowledge becomes compiled, and
therefore less accessible (see Section 2). An example could be automation
demonstrated by highly proficient bilinguals in terms of their language
switching abilities in everyday life (Green and Abutalebi, 2013). Research
suggests that initially, during the early years of second language
acquisition, non-selective language activation occurs. However, as both
languages become dominant, the selection rules become automatic. This leads to
automatic concept switching, where bilinguals become unaware of the switching
rules and rely more on external resources like social cues and a partner’s
fluency (Green and Abutalebi, 2013).

In the second case (dotted line in Figure 2), comprehension stays at a rather
intermediate level although enabling increases. This is only possible if
multiple external resources are available, are flexibly used (see definition
of deep enabledness) and are combined by a person (e.g., to solve a
programming problem, a colleague can serve as an external resource, or in
other situations, possibly due to time restrictions, a web search could be
another external resource). However, we think that this scenario allows deeper
enabledness only to a certain extent because deep enabledness requires deep
comprehension. An example representing this could be a situation in which a
lay user without any programming skills can run a script written in a certain
programming language. With already available commands that can be placed on
the correct position of the programming console, the lay user may manage to
complete the task successfully, and may even be enabled to replicate this in a
slightly changed context. Thus, their enabledness becomes deeper (though only
to a certain extent), but comprehension (e.g., programming syntax and meaning)
would remain at a constant, more shallow level if no further explanations are
given.

The third case (dashed line in Figure 2) is linked to the second one. A
progress in enabledness may result in a lagged, nonlinear increase in
comprehension following further explanations, that is, the progression in both
forms of understanding is step wise, similar to the idea of bootstrapping from
developmental research (Morgan and Demuth 1996; cf. Section 4.2). This is
based on research on automated processes in second language (L2)learning (cf.
Hao and Othman, 2021). Although L2-learners might encounter difficulties in
the process of generating new knowledge, for example the present progressive
tense, this process may be supported by establishing relations to previously
acquired, related conceptual knowledge, such as the general distinction
between present and past, as well as knowledge expressed in identification of
(in-)consistent syntactic constructions of the L2 based on their L1 knowledge
(cf. Mandler 2012; Fiorella and Mayer 2015, p. 7). At this stage,
comprehension remains behind the intermediate level, but enabledness may
increase progressively if L2-learners apply their previous conceptual
knowledge by doing and repeating related exercises, e.g., filling missing
auxiliaries ‘to be’ or missing -ing suffixes. Learners may also rely on
external resources such as textbooks, resources for language practice, or a
human tutor to complete the tasks. Learning the different application contexts
of the English present progressive tense would enable learners to use the
present progressive without recalling basic concepts and to apply it to
various similar cases in L2-learning.

To conclude, in our view and because of the intertwinement there are only
limited cases of really shallow comprehension and deep enabledness, and there
are no real life scenarios for the extreme, i.e., shallow comprehension with
intermediate-to-deep enabledness.

#### Assessment

If we follow Krathwohl’s taxonomy, for which we have argued that the first and
second steps require rather shallow comprehension, then tasks such as
summarizing can be used to assess shallow comprehension. To assess different
levels of enabledness, different tasks should be successfully accomplished
with the support of external resources (see above) that are either very
similar to those present during the explanation (more shallow/intermediate
enabledness) or that are modified to assess whether the knowledge can be
flexibly applied to novel contexts or other use cases (deeper level of
enabledness) within the specific domain of what was explained.

Regarding automation, i.e., how efficient an explainee is, different measures
could be used, e.g., processing time, or number of errors in relation to total
performance. Since we have defined automation as compiled knowledge that may
be less accessible to verbalization, the assessment could also consist of two
parts, a task and a questionnaire: if the explainee is able to successfully
accomplish the task but is unable to explain why or how it was done,
comprehension is likely to be shallower. However, this may also be an
indicator of transitional conceptual knowledge (Goldin-Meadow et al., 1993).

Figure 2: Three cases illustrating the strong intertwinement between
comprehension and enabledness.

###  3.3 Deep comprehension, shallow enabledness

With respect to the intertwinement discussed in Section 3.2, deep
comprehension combined with shallow enabledness is a rare case. However, it is
a persistent problem, e.g., in educational contexts, which are often
ultimately aimed at in educational settings. It can, and often does, occur
without a translation into ”knowing how” as an indicator of shallow
enabledness. An example of lacking conditional or conditionalized knowledge is
a student with knowledge about learning strategies who, in a school situation,
does not recognize that a problem is about learning strategies (and not about
motivation, etc.). Another example of a lack of compilation is a person who
has a lot of knowledge about hand spinning (e.g., types and functions of
spinning wheels, properties of fibers, ideas of twist, and the basic knowledge
of how to spin and ply them into a stable thread). Without the procedural
knowledge of how much twist the yarn should have, how fast to draw out the
fiber, how to stop the wheel, and how to counteract errors, this deep
understanding leads only to shallow enabledness.

One reason for deep understanding and shallow enabling is what has been called
‘inert knowledge’ (Whitehead, 1929, p. 5), although it was more concerned with
inert (scientific) ideas than inert knowledge of individuals. More recent
psychological research has demonstrated the prevalence and relevance of this
problem and suggested several explanations for it that are different in nature
and underscore its complexity (Renkl et al., 1996). The relationship between
deep understanding and shallow enabledness is here conceptualized in a very
specific way. According to meta-process explanations, the knowledge would be
applicable, i.e., would result in enabledness, but external prerequisites such
as motivation or the perception of benefits of applying the knowledge are
missing. Also, persons might lack information about when or where to apply
knowledge, which would also show up as shallow enabledness. Other explanations
focus on a deficiency in the structure of knowledge. Important ones that could
also explain the combination of deep understanding and shallow explanations
are a lack of compilation and a lack conditionalized knowledge. Declarative
knowledge has to be compiled in order to be used, and if compilation is
insufficient or if a person lacks knowledge about conditions in which
knowledge can or cannot be applied, shallow enabledness results. Finally, it
should be noted that theories of situated cognition do not regard inert
knowledge as a problem, but rather as due to the fact that knowledge is bound
to certain contexts and situations and their specific conditions (e.g.,
Greeno, 1998).

#### Example 1

Our example of a programmer who fully grasps the stopping criterion of the
algorithm, but is not able to formulate it recursively might fit this
explanation.

#### Example 2

Our example of a second language learner who knows the concepts of the present
progressive and how to build it, but cannot formulate correct utterances
spontaneously in a dialogue might also fit this explanation.

###  3.4 Deep comprehension, deep enabledness

Following the definitions of comprehension and enabledness, the combination of
deep comprehension and deep enabledness is an important prerequisite for what
is often referred to as ‘agency’. In this section, after presenting what deep
comprehension and deep enabledness would be in the case of the two examples,
we elaborate on the theoretical background of the term agency.

#### Example 1

In general, we reason that programmers have ‘agency’ if they can identify and
formulate the stopping criterion effortlessly and correctly. For implementing
the algorithm in a certain context, they apply theoretical knowledge about
iterative algorithms and stopping criteria and consider the demands defined by
the context (comprehension). Furthermore, they apply this knowledge when
defining the stopping criterion in a specific way, which is implied by the
programming language that is used (enabledness). Only if they do both and can
flexibly combine their understanding of the algorithm with their abilities to
write the program in a programming language, a performance of high
understanding and thus agency can be observed.

#### Example2

Scholars of the English language not only know that the progressive is
generally in the form auxiliary verb be + verb ending in -ing, but adhere to
these rules in writing and speech as well, without actively thinking about it
too much. Scholars with a large body of knowledge and practical experience
using it can elaborate why in certain contexts the present progressive tense
is the correct choice, as well as, for example, describe common errors of
native speakers. Simply knowing a handful of contexts and use cases of the
present progressive by heart is not sufficient. Instead, scholars can flexibly
use and apply their knowledge to come up with novel contexts, depending on the
situation (e.g., when asked a question about the rules by English language
novices). The fact that the scholars can flexibly apply (enabledness) in
combination with what they know (comprehension) is a thought-demanding
performance that shines a light on their understanding and agency.

#### Definitions of agency

Agency is defined in various ways, which is apparent as the term itself is
often combined with another term to form an open compounds part of a specific
phrase such as, for example, ‘digital agency’ (Arnold and Clarke 2013, p. 736;
see, e.g., Passey et al. 2018). There is the need for scholars to provide a
precise definition of the term (cf. Ahearn, 2001, p. 130) and the vagueness of
the term is criticized, too (Emirbayer and Mische, 1998). Different
definitions help to illustrate the problem: Passey et al. (2018) define
‘digital agency’ quite broadly as “Digital Agency (DA)—consisting of digital
competence, digital confidence, and digital accountability—is the individual’s
ability to control and adapt to a digital world”. (Couldry, 2014), in his
inaugural lecture, was more explicit: First, agency, by which I mean not brute
acts (of clicking on this button, pressing ‘like’ to this post) but (following
Weber) the longer processes of action based on reflection, giving an account
of what one has done, even more basically, making sense of the world so as to
act within it. Kallinikos (2002, p. 289f) describes how technology may
influence agency “by inviting specific courses of action” that may be shaped
to a certain degree (‘malleability’) and, that agency is “captured by the
distinctive ways by which a technology invites people to frame a delimited
domain of tasks or activities and organize their execution”. Arnold and Clarke
(2013, p. 751) define student agency as “[t]he discursive practice of
positioning oneself or being positioned as responsible”. Emirbayer and Mische
(1998, p. 963) follow the pragmatic tradition of Bourdieu and Giddens, who
define “[…] human agency as habitual, repetitive, and taken for granted—a view
shared by ethnomethodologists, new institutionalists in organizational theory,
and many others”. According to Sharma (2008, p. 300) agency is a “contingently
emergent feature of situated local action”. Thus, humans do not just have
agency, but they can show agency through their actions. Siry and Lang (2010)
describe agency as a feature a person can acquire and develop. This means
humans may acquire agency during childhood and education. Basu (2008, p. 891)
describes agency as purposefully examining and reflecting the actions of one
person; the foundation of this acting is the beliefs and goals one person has.
She says: “[…] I describe agency as purposefully considering and enacting both
small- and large-scale change in personal and community domains, based on
one’s beliefs and goals”. Duranti (2005, p. 453) defines agency more broadly,
as something entities are having. His working definition defines agency as a
property of entities to (i) have a certain degree of control over their
behavior, (ii) whose actions have a certain effect on other entities, and
(iii) whose actions can be evaluated for example for responsibility.

#### Agency in the context of understanding

A mid to high degree of enabledness combined with a mid to high degree of
comprehension are needed for agency. If both are pronounced, an individual is
capable of actively manipulating and controlling various situations flexibly,
by using what they know to reach individual goals. A high degree of
comprehension and enabling is agency as it is a prerequisite for various
properties mentioned in definitions by Passey et al. (2018) (“control and
adapt”), Basu (2008) (“purposefully considering and enacting both small- and
large-scale change in personal and community domains, based on one’s beliefs
and goals”), Duranti (2005) (having a certain degree of control over one’s own
behavior, having an effect on other entities through one’s behaviour) and
Sharma (2008) (“a contingently emergent feature of situated local action”
perceivable within actions”).

The question of how one would assess understanding in specific contexts and
therefore, implicitly, how agency arises, is challenging is the fact that our
concept of understanding remains a latent construct that is not trivial to
define. Especially mental operations like connecting new facts cannot easily
be measured. In other words, understanding cannot be assessed by asking to
recall facts. However, it can be observed when people are engaged in unknown
situations that require the utilization of previously acquired knowledge and a
person’s enablement. In these situations, agency results in a flexible
solution to the given task. This approach adheres to Perkins’ idea of
measuring the ‘performance of understanding’ (Perkins, 1993, 1998). Perkins
recognizes “understanding through a flexible performance criterion” that
“shows its face when people can think and act flexibly around what they know”.
In contrast to this, a signal of lack of understanding is, “when a learner
cannot go beyond rote and routine thought and action” (Perkins, 1998, p. 42).

##  4 Dynamics of Understanding

Understanding is a process that unfolds over time when an explanation is
successful (Keil, 2006). In this section we offer a conceptualization of the
dynamics of understanding over time. We focus on two distinct aspects that
were mentioned in passing above: differences between novices and experts, and
the intertwining of comprehension and enabledness in the transition from a
lower to a higher form of understanding.

###  4.1 Novices and experts

Using the heuristics of our four-field diagram, an explainee with shallow
comprehension and shallow enabledness is considered a novice, while a deeply
enabled explainee with deep comprehension is considered an expert. The purpose
of this section is to outline some fundamental differences between novices and
experts with respect to the two types of knowledge.

First, novices and experts are assumed to differ in terms of procedural and
conceptual knowledge. Experts were found to categorize problems by applying
abstract solution procedures (e.g., major physical principles), whereas
novices build their problem categorization on “the entities contained in the
problem statement” (Chi et al., 1981, p. 15). In terms of problem
representation, experts translate features of the problem, thereby activating
abstract category schemes for a problem type that are organized, for example,
by general laws of nature. In contrast, novices organize their representation
of a problem by schemas for object categories, thus staying on more concrete
levels when formulating the problem.

Second, while domain knowledge is one aspect in which novices and experts
undoubtedly differ, they are also likely to differ in terms of experience,
e.g., in a communicative practice. A study investigating novice tutors’
abilities to assess and monitor students’ knowledge was found to be profoundly
inaccurate (Chi et al., 2004).

In our attempt to go beyond classical cognitive psychology, which
distinguishes between experts and novices by focusing on (domain) knowledge,
we would like to enrich the pair of terms by bringing in constructive learning
theory, which considers context orientation and situatedness as crucial. The
focus here is on the interactive processes of how experts guide novices into
ways of participating in a practice after a shared goal has been established
in a given situation (Lave and Wenger, 1991; Rogoff, 1990). For example,
children are often referred to as novices because they are less knowledgeable.
In conversations, the more knowledgeable participant (i.e., usually the adult)
provides interactive support to help the child perform a aspired task (Wood et
al., 1976). Goodwin (2013) broadens the perspective even further by explicitly
including the body in the conceptual pair, thereby including multimodal
aspects of interaction. He points out that in order to become competent,
novices need to train both their bodies and the perceptually relevant
structures of the objects or surroundings to be scrutinized in order to become
skilled at performing a particular practice. In this sense, Goodwin (2013)
provides an anthropological view of embodied cognition that sees body and mind
as inextricably intertwined. Understanding is therefore displayed through
correct embodied behavior, and teachers could use displays of embodied
behavior to assess students’ knowledge (see also Zukow-Goldring, 1996, for
perceptual structure provided by caregivers). Hindmarsh et al. (2011) shows
how supervisors of dental students rely not so much on talk but on bodily
behavior to assess their students’ ongoing understanding: even though students
may claim to understand a procedure, the way they move their bodies to look
into a patient’s mouth may actually reveal their lack of understanding of the
matter at hand.

Including embodiment in the conceptualization of understanding adds a new
perspective because it may provide a more complete view of understanding than
simply focusing on internal domains of knowledge. Novices would then differ
from experts not only in terms of conceptualization and problem-solving
procedures with respect to a particular domain, but also in that they would
not have the embodied knowledge of how to perform a task (Zukow-Goldring,
1996). It has been pointed out that knowledge and its asymmetries are relevant
in distinguishing between experts and novices. The next section looks at
different types of knowledge in this context.

###  4.2 Intertwinement of comprehension and enabledness: Transitions between
the forms

The idea for the intertwinement of comprehension and enabledness (see Section
3.2) is related to the concept of ‘bootstrapping’ which might be an underlying
mechanism facilitating the transgression from one part of the four-field
diagram to another. Bootstrapping is assumed as being a learning mechanism for
linguistic development by which growing abilities on one linguistic level,
e.g., phonology, supports the progression on other linguistic levels such as
syntax or semantics (Morgan and Demuth, 1996).

Applying this to the different forms of understanding in our four-field
scheme, comprehension can be bootstrapped by enabledness because comprehension
assumably deepens when learning to apply a knowledge domain to a novel
context/situation. Vice versa, enabledness deepens with comprehension, i.e.,
learning more about covert mechanisms of a domain. As an example, enabledness
in programming a while-loop will improve when the person learns more about
programming languages. Thus, bootstrapping could explain the transitions
between different forms and levels of understanding.

This mechanism can be illustrated using the example 1 for the early stages of
understanding (see Section 3.1). The challenge for a novice is to understand
by explanation a stopping criterion of an iterative algorithm, or mainly to
set up a strategy to understand the basic theory of a while-loop in a
programming language. To describe the states of the model of a novice’s
understanding in this example, we are looking at the transitions to other
forms of understanding from the field of shallow comprehension and shallow
enabledness:

(1) Shallow comprehension, shallow enabledness

The starting point depends on the novice’s prior knowledge. A novice with
basic computer science knowledge will start by modifying sample code functions
and examining the results to improve their comprehension of the functions and
their meanings. A novice with no computer science knowledge, but with a higher
conceptual knowledge about the syntax of an iterative function, will start to
reproduce the results manually on paper. In both cases, interaction with the
code will increase the model of understanding in an exploratory way.

(2) Shallow comprehension, increasing enabledness

A novice with less conceptual knowledge about a while-loop, but more knowledge
about how to interact with code, can try out the code example. The process of
reproducing the meaning of the code will increase (i.e., bootstrap) the
understanding of the syntax and meaning of an iterative function. However, the
lack of comprehension of the theoretical background and the repetition of such
a function (e.g., the stopping criterion) can also lead to irritations and
misunderstandings. Thus, an incorrect model for the theory of a while-loop
could be built up.

(3) Increasing comprehension, shallow enabledness

Increased comprehension can lead to an understanding of the code syntax and
the overall goal of the function. For example, a novice knows in theory that a
while-loop needs a stop criterion to let the algorithms finish a computation.
But with shallow enabledness, the novice may not be able to modify the code
independently without generating errors. In addition, the novice may not be
able to apply their comprehension to new contexts outside this specific
example.

(4) Increasing comprehension, increasing enabledness

A novice with an increased comprehension knows the meaning of the symbols in
the code and understands the basic syntax. A novice might be able to
understand the theory of a while-loop, mainly by interpreting the iterative
repetitions in a given example, and reproduce the expected results outside of
that example.

These transitions can, for example, be classified in the field of shallow
comprehension and shallow enabledness as follows: (1) The initial prior
knowledge of a novice, (2) the process of how to access conceptual knowledge
without known theories, (3) understanding the general theories without being
able to apply them, (4) building up a strategy to increase conceptual
knowledge by applying less or no known theories.

##  5 Conclusion

As shown in the previous sections, discussing the cognitive side of XAI, it is
important to consider types and forms of human understanding of explanations
depending on their prior knowledge, motivation, and goals. These aspects are
increasingly considered in the XAI literature.

We suggested aspects and forms of understanding which may occur in
explanations between humans, but also in XAI explanations for human users.
Considering the forms given in our four-field diagram underlines that
explainers as well as explainees pursue many different goals: to be enabled to
do something, to comprehend something, or to achieve full agency. This agency
needs – as we assume – both, deep comprehension and deep enabledness. There
may, however, also be situations, in which humans only strive for shallow
enabledness and/or shallow comprehension. The development of XAI technology
has to consider these possibilities as well.

Generally, understanding is a journey from shallow to deep comprehension and
enabledness, which results in both forms being highly intertwined and in need
of being considered. In the end, it is not possible to reach deep enabledness
without comprehension. For XAI, it follows that the amount of information
required needs to be negotiated between the explainer (which could be an XAI
system) and their explainee. As an important background of our contribution,
Rohlfing et al. (2021) conceptualize explaining as a social practice in which
explainers and explainees are equally involved in (and responsible for) co-
constructing the explanation and its understanding.

We consider these conceptualizations to be a concise theoretical structure for
an interdisciplinary discussion and multiperspectival development and
evaluation of XAI technology, and explaining and understanding more generally.
Two issues that we have not addressed above but that are highly relevant for
explainability are (1) the need to consider that humans often have a
subjective understanding that does not fully align with their actual/objective
understanding and (2) that our cognitive approach of conceptualising
understanding neglects that explanations always take place in a social
context. We close this paper by discussing these two aspects.

###  5.1 Subjective vs. objective understanding

‘Subjective’ understanding in explanatory interactions relates to how well
explainees think they have understood an explanation, whereas ‘objective’
understanding describes the _actual_ understanding explainees have of the
explanans. Discrepancies between an explainee’s subjective claims and their
objective understanding are to be expected (e.g., due to misconceptions or
unknown unknowns). Concerning the forms of understanding proposed in this
paper, an explainee’s subjective comprehension is likely more prone to be
misaligned to their objective comprehension than is the case for enabledness.
The illustration of the assessment of the different forms of understanding in
Section 3 shows that research makes use of subjective self-report measures of
understanding as well as objective testing following learning taxonomies.

The distinction between subjective and objective understanding is also
important in relation to biased responses – as mentioned above. As long as we
want to measure subjective understanding, we have to take into account that
the answers may be biased in some way. This is not a problem in itself, but it
becomes a problem as soon as the bias is caused by the question posed by an
explainer, which is more likely to happen the more complex and detailed the
question becomes.

In XAI, methods and systems that aim at co-constructing the human user’s
understanding, relying on monitoring and adaptation processes for tailoring
explanations to users’ needs (Rohlfing et al., 2021), need to be aware of the
difference between subjective and objective understanding. Depending on the
mode of interaction, evidence of understanding that users provide can simply
be subjective _claims_ of understanding, (e.g., through the production of
verbal acknowledgements signals or indications of satisfaction in surveys) or
_demonstrations_ of understanding that are more revealing of their actual
understanding (e.g., being able to execute specific actions or asking follow-
up questions that are contingent on understanding; cf. Koole 2010). XAI
systems need to be able to identify these different types of evidence, weigh
them accordingly, and assess them critically when monitoring and adapting to
their human interaction partners.

###  5.2 Influences brought by society

So far, the question of the potential impact of social processes on
explanations and related processes of understanding has been excluded.
However, how explainable AI – or any other new technology – is reflected upon
in society has implications for the understanding of the individuals
interacting with it, as humans constitute social reality by interpreting and
creating their world and its objects in interactions (Rosenthal, 2018). Since
the purpose of co-constructing Explainable Artificial Intelligence (XAI) is to
support both machines and humans in their communication with each other, the
understanding capabilities of both should be considered. The considerations
relate to social constructionist approaches, according to which successful
social acceptance, adoption and understanding of newly introduced technologies
are highly dependent on the meaning co-constructed in social interactions,
i.e., when human (or non-human) agents or group members interact. In this
context, group membership (e.g., constituted by gender, class, race, age,
socioeconomic status) plays a key role in the process of understanding
technological novelty and thus in explaining it. Group membership “determines
what is relevant and what is not” (Kronberger, 2015, p. 361). In other words,
goals or relationships may be different for different social groups.

The social constructivist view of human-human interaction is also relevant to
human-machine interaction, and even precedes it. Ehsan and Riedl (2020)
emphasize the importance of a human-centered approach to XAI. Following their
‘Reflective Sociotechnical Approach’, we propose a concept of reflective
understanding that critically reflects the dominant narratives as well as the
explicit and implicit assumptions and practices of the field. Moreover, it is
value-sensitive to researchers, designers, and users of AI, including to those
who tend to be excluded from the conversation and are denied access to the
technology (Ehsan and Riedl, 2020, p. 461). In other words, in order to
understand how and why people act the way they do (e.g., adoption or non-
adoption of a particular new technology) social meanings, interactions and
interpretations cannot be ignored. Accordingly, explanations must also address
how people interpret their world and how they interactively create this world.
Not only the perspectives and stocks of knowledge that we are aware of should
be of interest when designing an explanation, but also implicit knowledge and
the interactive creation of meanings that people (users) are usually not aware
of (Rosenthal, 2018).

In this paper, we have outlined a model of understanding that prioritizes the
individual cognitive aspects of understanding. However, we propose to add a
social dimension that inevitably informs cognitive processes in ways that are
conducive to a conceptualization of co-constructive explanatory processes.
Thus, we argue for a broad notion of understanding that goes beyond a
cognitive model that conceives of comprehension and enabledness as discrete
processes occurring in the minds of explainees, and argue that aspects such as
context, interaction, or group membership, as well as broader socio-technical
relations and references, should also be considered in the development of XAI
technology.

Applying our concept of agency (see Section 3.4), a high degree of
comprehension and enabledness is characterized by healthy distrust, deep
enablement, knowledge of the strengths and power, but also of the limitations,
biases, and potential for and actualization of oppression and exclusion that
AI technology entails. In addition, agency means being empowered to overcome
socially and technologically consolidated marginalization/discrimination that
may be related to group membership. Thus, in order to support agency,
explainers (human or XAI) should take into account explainees’ group-related
inequalities and differences, as well as varying access to resources and
positions/roles in socio-technical systems, because group affiliation
“determines what is relevant and what is not” (Kronberger, 2015, p. 361).
Indeed, we see this as inevitable for AI development because a
conceptualization of human understanding of XAI explanations without context
is insufficient, and concepts of human understanding should be central to the
research and design of XAI methods and technology so that they can do justice
to the humans they aim to empower.

## Author Contributions

Helen Beierling: Writing – Original Draft. Heike M. Buhl: Supervision,
Conceptualization, Writing – Original Draft, Writing - Review & Editing.
Hendrik Buschmeier: Supervision, Conceptualization, Writing – Original Draft,
Writing - Review & Editing. Josephine Fisher: Writing – Original Draft. Angela
Grimminger: Conceptualization, Writing – Original Draft. André Groß: Writing –
Original Draft. Ilona Horwath: Conceptualization, Writing – Original Draft.
Friederike Kern: Supervision, Conceptualization, Writing – Original Draft,
Writing - Review & Editing. Nils Kloweit: Writing – Original Draft. Stefan
Lazarov: Writing – Original Draft. Michael Lenke: Writing – Original Draft.
Vivien Lohmer: Writing – Original Draft. Katharina Rohlfing: Writing –
Original Draft. Ingrid Scharlau: Conceptualization, Writing – Original Draft.
Amit Singh: Writing – Original Draft. Lutz Terfloth: Conceptualization,
Writing – Original Draft. Anna-Lisa Vollmer: Writing – Original Draft. Yu
Wang: Writing – Original Draft. Annedore Wilmes: Writing – Original Draft.
Britta Wrede: Writing – Original Draft.

## Acknowledgements

This research was supported by the German Research Foundation (DFG) in the
Collaborative Research Center TRR 318/1 2021 ‘Constructing Explainability’
(438445824).

## References

  * Ahearn (2001) Ahearn, L.M., 2001.  Language and agency.  Annual Review of Anthropology 30, 109–137.  doi:10.1146/annurev.anthro.30.1.109.
  * Alba and Hasher (1983) Alba, J.W., Hasher, L., 1983\.  Is memory schematic?  Psychological Bulletin 93, 203–231.  doi:10.1037/0033-2909.93.2.203.
  * Anderson (1983) Anderson, J.R., 1983.  The Architecture of Cognition.  Lawrence Erlbaum, Hillsdale, NJ, USA.
  * Anderson (1996) Anderson, J.R., 1996.  ACT: A simple theory of complex cognition.  American Psychologist 51, 355–365.  doi:10.1037/0003-066x.51.4.355.
  * Anderson (2002) Anderson, L.W., 2002.  Benjamin samuel bloom (1913-1999).  American Psychologist 57, 63–63.  doi:10.1037/0003-066x.57.1.63.
  * Anderson and Krathwohl (2001) Anderson, L.W., Krathwohl, D.R., 2001\.  A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives.  Longman, New York, NY, USA.
  * Arnold and Clarke (2013) Arnold, J., Clarke, D.J., 2013\.  What is ‘agency’? perspectives in science education research.  International Journal of Science Education 36, 735–754.  doi:10.1080/09500693.2013.825066.
  * (8) Auer, P., .  Introduction: John Gumperz’s approach to contextualization, in: Auer, P., Di Luzio, A. (Eds.), The Contextualization of Language. John Benjamins, Amsterdam, The Netherlands, pp. 1–39.  doi:10.1075/pbns.22.03aue.
  * Bailer-Jones (2009) Bailer-Jones, D.M., 2009.  Scientific Models in Philosophy of Science.  University of Pittsburgh Press, Pittsburgh, PA, USA.
  * Ball et al. (2017) Ball, C., Huang, K.T., Rikard, R.V., Cotten, S.R., 2017\.  The emotional costs of computers: An expectancy-value theory analysis of predominantly low-socioeconomic status minority students’ STEM attitudes.  Information, Communication & Society 22, 105–128.  doi:10.1080/1369118x.2017.1355403.
  * Basu (2008) Basu, S.J., 2008.  How students design and enact physics lessons: Five immigrant Caribbean youth and the cultivation of student voice.  Journal of Research in Science Teaching 45, 881–899.  doi:10.1002/tea.20257.
  * Best et al. (2004) Best, R., Ozuru, Y., McNamara, D.S., 2004.  Self-explaining science texts: Strategies, knowledge, and reading skill, in: Proceedings of the 6th International Conference on Learning Sciences, Santa Monica, CA, USA. p. 89–96.
  * Boaler and Greeno (2000) Boaler, J., Greeno, J.G., 2000\.  Identity, agency and knowing in mathematical worlds, in: Boaler, J. (Ed.), Multiple Perspectives on Mathematics Teaching and Learning. Ablex, Westport, CT, USA, pp. 171–200.
  * Bonanati and Buhl (2021) Bonanati, S., Buhl, H.M., 2021\.  The digital home learning environment and its relation to children’s ICT self-efficacy.  Learning Environments Research 25, 485–505.  doi:10.1007/s10984-021-09377-8.
  * Bowers et al. (1990) Bowers, K.S., Regehr, G., Balthazard, C., Parker, K., 1990\.  Intuition in the context of discovery.  Cognitive Psychology 22, 72–110.  doi:10.1016/0010-0285(90)90004-n.
  * Brennan and Hanna (2009) Brennan, S.E., Hanna, J.E., 2009\.  Partner-specific adaptation in dialogue.  Topics in Cognitive Science 1, 274–291.  doi:10.1111/j.1756-8765.2009.01019.x.
  * Byrd (2010) Byrd, D., 2010.  Infinite bottles of beer: A cantorial approach to Cantorian arithmetic and other mathematical melodies.  Math Horizons 18, 16–17.  doi:10.4169/194762110X524657.
  * Carter and McCarthy (2006) Carter, R., McCarthy, M., 2006\.  Cambridge Grammar of English.  Cambridge University Press, Cambridge, UK.
  * Chi et al. (1981) Chi, M.T., Feltovich, P.J., Glaser, R., 1981.  Categorization and representation of physics problems by experts and novices.  Cognitive Science 5, 121–152.
  * Chi et al. (1994) Chi, M.T.H., de Leeuw, N., Chiu, M.H., Lavancher, C., 1994\.  Eliciting self-explanations improves understanding,.  Cognitive Science 18, 439–477.  doi:10.1016/0364-0213(94)90016-7.
  * Chi et al. (2004) Chi, M.T.H., Siler, S.A., Jeong, H., 2004.  Can tutors monitor students’ understanding accurately?  Cognition and Instruction 22, 363–387.  doi:10.1207/s1532690xci2203_4.
  * Chi and Wylie (2014) Chi, M.T.H., Wylie, R., 2014\.  The ICAP framework: Linking cognitive engagement to active learning outcomes.  Educational Psychologist 49, 219–243.  doi:10.1080/00461520.2014.965823.
  * Clark (2014) Clark, E.V., 2014.  Pragmatics in acquisition.  Journal of Child Language 41, 105–116.  doi:10.1017/s0305000914000117.
  * Cole et al. (1986) Cole, C.A., Gaeth, G., Singh, S.N., 1986.  Measuring prior knowledge, in: Lutz, R.J. (Ed.), Advances in Consumer Research, Provo, UT, USA. pp. 64–66.
  * Couldry (2014) Couldry, N., 2014.  Inaugural: A necessary disenchantment: myth, agency and injustice in a digital world.  The Sociological Review 62, 880–897.  doi:10.1111/1467-954x.12158.
  * Covington (2000) Covington, M.V., 2000.  Goal theory, motivation, and school achievement: An integrative review.  Annual Review of Psychology 51, 171–200.  doi:10.1146/annurev.psych.51.1.171.
  * Davis (1989) Davis, F.D., 1989.  Perceived usefulness, perceived ease of use, and user acceptance of information technology.  MIS Quarterly 13, 319\.  doi:10.2307/249008.
  * Dochy and Alexander (1995) Dochy, F.J.R.C., Alexander, P.A., 1995\.  Mapping prior knowledge: A framework for discussion among researchers.  European Journal of Psychology of Education 10, 225–242.  doi:10.1007/bf03172918.
  * Duranti (2005) Duranti, A., 2005.  Agency in language, in: Duranti, A. (Ed.), A Companion to Linguistic Anthropology. Blackwell, Malden, MA, USA, pp. 449–473.
  * Eccles and Wigfield (2020) Eccles, J.S., Wigfield, A., 2020\.  From expectancy-value theory to situated expectancy-value theory: A developmental, social cognitive, and sociocultural perspective on motivation.  Contemporary Educational Psychology 61, 101859.  doi:10.1016/j.cedpsych.2020.101859.
  * Ehsan and Riedl (2020) Ehsan, U., Riedl, M.O., 2020\.  Human-centered Explainable AI: Towards a reflective sociotechnical approach, in: Stephanidis, C., Kurosu, M., Degen, H., Reinerman-Jones, L. (Eds.), HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence, Springer, Cham, Switzerland. pp. 449–466.
  * Emirbayer and Mische (1998) Emirbayer, M., Mische, A., 1998\.  What is agency?  American Journal of Sociology 103, 962–1023.  doi:10.1086/231294.
  * Fiorella and Mayer (2015) Fiorella, L., Mayer, R.E., 2015\.  Learning as a generative activity: Eight learning strategies that promote understanding.  Cambridge University Press, Cambridge, UK.
  * Gardner (1999) Gardner, H., 1999.  Intelligence Reframed: Multiple Intelligences for the 21st Century.  Basic Books, New York, NY, USA.
  * Gentile et al. (2021) Gentile, D., Jamieson, G.A., Donmez, B., 2021.  Evaluating human understanding in XAI systems, in: Proceedings of the ACM CHI 2021 Workshop on Human-Centered Explainable AI (HCXAI), Online. p. 25.
  * Gentner et al. (2003) Gentner, D., Loewenstein, J., Thompson, L., 2003.  Learning and transfer: A general role for analogical encoding.  Journal of Educational Psychology 95, 393–408.  doi:10.1037/0022-0663.95.2.393.
  * Goldin-Meadow (2017) Goldin-Meadow, S., 2017.  Using our hands to change our minds.  WIREs Cognitive Science 8, e1368.  doi:10.1002/wcs.1368.
  * Goldin-Meadow et al. (1993) Goldin-Meadow, S., Wagner Alibali, M., Breckinridge Church, R., 1993.  Transitions in concept acquisition: Using the hand to read the mind.  Psychological Review 100, 279–297.  doi:10.1037/0033-295x.100.2.279.
  * Goodwin (2013) Goodwin, C., 2013.  The co-operative, transformative organization of human action and knowledge.  Journal of Pragmatics 46, 8–23.  doi:10.1016/j.pragma.2012.09.003.
  * Graesser and Person (1994) Graesser, A.C., Person, N.K., 1994\.  Question asking during tutoring.  American Educational Research Journal 31, 104–137.  doi:10.3102/00028312031001104.
  * Green and Abutalebi (2013) Green, D.W., Abutalebi, J., 2013\.  Language control in bilinguals: The adaptive control hypothesis.  Journal of Cognitive Psychology 25, 515–530.  doi:10.1080/20445911.2013.796377.
  * Greeno (1998) Greeno, J.G., 1998.  The situativity of knowing, learning, and research.  American Psychologist 53, 5–26.  doi:10.1037/0003-066x.53.1.5.
  * Hao and Othman (2021) Hao, M., Othman, R., 2021.  Automation of function assignment in the models of speech production and second language acquisition.  Education Research International 2021, 1–9.  doi:10.1155/2021/2441598.
  * Hindmarsh et al. (2011) Hindmarsh, J., Reynolds, P., Dunne, S., 2011.  Exhibiting understanding: The body in apprenticeship.  Journal of Pragmatics 43, 489–503.  doi:10.1016/j.pragma.2009.09.008.
  * Johnson-Laird (1983) Johnson-Laird, P., 1983.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness.  Cambridge University Press, Cambridge, UK.
  * de Jong and Ferguson-Hessler (1996) de Jong, T., Ferguson-Hessler, M.G., 1996\.  Types and qualities of knowledge.  Educational Psychologist 31, 105–113.  doi:10.1207/s15326985ep3102_2.
  * Kallinikos (2002) Kallinikos, J., 2002.  Reopening the black box of technology artifacts and human agency, in: Proceedings of the 23rd International Conference on Information Systems, Barcelona, Spain. pp. 287–294.
  * Keil (2006) Keil, F.C., 2006.  Explanation and understanding.  Annual Review of Psychology 57, 227–254.  doi:10.1146/annurev.psych.57.102904.190100.
  * Kintsch (1998) Kintsch, W., 1998.  Comprehension. A Paradigm for Cognition.  Cambridge University Press, Cambridge, UK.
  * Klein (2009) Klein, J., 2009.  Erklären-Was, Erklären-Wie, Erklären-Warum: Typologie und Komplexität zentraler Akte der Welterschließung, in: Vogt, R. (Ed.), Erklären. Gesprächsanalytische und fachdidaktische Perspektiven. Stauffenburg, Tübingen, Germany, pp. 25–36.
  * Koole (2010) Koole, T., 2010.  Displays of epistemic access: Student responses to teacher explanations.  Research on Language & Social Interaction 43, 183–209.  doi:10.1080/08351811003737846.
  * Krathwohl (2002) Krathwohl, D.R., 2002.  A revision of Bloom’s taxonomy: An overview.  Theory Into Practice 41, 212–218.  doi:10.1207/s15430421tip4104_2.
  * Krathwohl et al. (1969) Krathwohl, D.R., Bloom, B.S., Masia, B.B., 1969.  Taxonomy of Educational Objectives: The Classification of Educational Goals, Volume 1: Cognitive Domain.  McKay, New York, NY, USA.
  * Kronberger (2015) Kronberger, N., 2015.  Of worlds and objects: Scientific knowledge and its publics, in: Sammut, G., Andreouli, E., Gaskell, G., Valsiner, J. (Eds.), The Cambridge Handbook of Social Representations. Cambridge University Press, Cambridge, UK, pp. 358–368.  doi:10.1017/CBO9781107323650.029.
  * Lave and Wenger (1991) Lave, J., Wenger, E., 1991.  Situated Learning: Legitimate Peripheral Participation.  Cambridge University Press, Cambridge, UK.
  * Lim and Ahmed (2000) Lim, K.K., Ahmed, P.K., 2000\.  Enabling knowledge management: A measurement perspective, in: Proceedings of the 2000 IEEE International Conference on Management of Innovation and Technology, Singapore. pp. 690–695.  doi:10.1109/icmit.2000.916781.
  * Mandler (2012) Mandler, J.M., 2012.  On the spatial foundations of the conceptual system and its enrichment.  Cognitive Science 36, 421–451.  doi:10.1111/j.1551-6709.2012.01241.x.
  * Marangunić and Granić (2014) Marangunić, N., Granić, A., 2014\.  Technology acceptance model: A literature review from 1986 to 2013.  Universal Access in the Information Society 14, 81–95.  doi:10.1007/s10209-014-0348-1.
  * Martins and Gresse von Wangenheim (2022) Martins, R.M., Gresse von Wangenheim, C., 2022\.  Findings on teaching machine learning in high school: A ten-year systematic literature review.  Informatics in Education doi:10.15388/infedu.2023.18.
  * Miller (2019) Miller, T., 2019.  Explanation in artificial intelligence: Insights from the social sciences.  Artificial Intelligence 267, 1–38.  doi:10.1016/j.artint.2018.07.007.
  * Miyake (1986) Miyake, N., 1986.  Constructive interaction and the iterative process of understanding.  Cognitive Science 10, 151–177.  doi:10.1207/s15516709cog1002_2.
  * Morgan and Demuth (1996) Morgan, J.L., Demuth, K. (Eds.), 1996\.  Signal to Syntax: Bootstrapping from Speech to Grammar in Early Acquisition.  Psychology Press, New York, NY, USA.
  * Norman (1988) Norman, D.A., 1988.  The Psychology of Everyday Things.  Basic Books, New York, NY, USA.
  * Páez (2019) Páez, A., 2019.  The pragmatic turn in explainable artificial intelligence (XAI).  Minds and Machines 29, 441–459.  doi:10.1007/s11023-019-09502-w.
  * Passey et al. (2018) Passey, D., Shonfeld, M., Appleby, L., Judge, M., Saito, T., Smits, A., 2018\.  Digital agency: Empowering equity in and through education.  Technology, Knowledge and Learning 23, 425–439.  doi:10.1007/s10758-018-9384-x.
  * Perkins (1993) Perkins, D., 1993.  Teaching for understanding.  American Educator 17.
  * Perkins (1998) Perkins, D., 1998.  What is understanding, in: Wiske, M.S. (Ed.), Teaching for Understanding: Linking Research with Practice. Jossey-Bass, San Francisco, CA, USA, pp. 39–57.
  * Pickering and Garrod (2013) Pickering, M.J., Garrod, S., 2013\.  An integrated theory of language production and comprehension.  Behavioral and Brain Sciences 36, 329–347.  doi:10.1017/S0140525X12001495.
  * van de Pol et al. (2018) van de Pol, J., Mercer, N., Volman, M., 2018.  Scaffolding student understanding in small-group work: Students’ uptake of teacher support in subsequent small-group interaction.  Journal of the Learning Sciences 28, 206–239.  doi:10.1080/10508406.2018.1522258.
  * Qian and Lehman (2017) Qian, Y., Lehman, J., 2017.  Students’ misconceptions and other difficulties in introductory programming: A literature review.  ACM Transactions on Computing Education 18\.  doi:10.1145/3077618.
  * Quasthoff et al. (2017) Quasthoff, U., Heller, V., Morek, M., 2017.  On the sequential organization and genre-orientation of discourse units in interaction: An analytic framework.  Discourse Studies 19, 84–110.  doi:10.1177/1461445616683596.
  * Renkl et al. (1996) Renkl, A., Mandl, H., Gruber, H., 1996.  Inert knowledge: Analyses and remedies.  Educational Psychologist 31, 115–121.  doi:10.1207/s15326985ep3102_3.
  * Rogoff (1990) Rogoff, B., 1990.  Apprenticeship in Thinking: Cognitive Development in Social Context.  Oxford University Press, Oxford, UK.
  * Rohlfing et al. (2021) Rohlfing, K., Cimiano, P., Scharlau, I., Matzner, T., Buhl, H., Buschmeier, H., Grimminger, A., Hammer, B., Häb-Umbach, R., Horwath, I., Hüllermeier, E., Kern, F., Kopp, S., Thommes, K., Ngonga Ngomo, A.C., Schulte, C., Wachsmuth, H., Wagner, P., Wrede, B., 2021.  Explanation as a social practice: Toward a conceptual framework for the social design of ai systems.  IEEE Transactions on Cognitive and Developmental Systems 13, 717–728.  doi:10.1109/TCDS.2020.3044366.
  * Rosenthal (2018) Rosenthal, G., 2018.  Interpretive Social Research.  Göttingen University Press, Göttingen, Germany.  doi:10.17875/gup2018-1103.
  * Sasse (1991) Sasse, M.A., 1991.  How to T(R)AP users’ mental models, in: Tauber, M.J., Ackermann, D. (Eds.), Mental Models and Human-Computer Interaction 2. Elsevier, Amsterdam, The Netherlands, pp. 59–79.  doi:10.1016/b978-0-444-88602-6.50007-4.
  * Sharma (2008) Sharma, A., 2008.  Making (electrical) connections: Exploring student agency in a school in India.  Science Education 92, 297–319.  doi:10.1002/sce.20246.
  * Simonsmeier et al. (2021) Simonsmeier, B.A., Flaig, M., Deiglmayr, A., Schalk, L., Schneider, M., 2021.  Domain-specific prior knowledge and learning: A meta-analysis.  Educational Psychologist 57, 31–54.  doi:10.1080/00461520.2021.1939700.
  * Siry and Lang (2010) Siry, C.A., Lang, D.E., 2010\.  Creating participatory discourse for teaching and research in early childhood science.  Journal of Science Teacher Education 21, 149–160.  doi:10.1007/s10972-009-9162-7.
  * Tanaka (2011) Tanaka, S., 2011.  The notion of embodied knowledge, in: Stenner, P., Cromby, J., Motzkau, J., Yen, J., Haosheng, Y. (Eds.), Theoretical Psychology: Global Transformations and Challenges. Captus Press, Concord, ON, Canada, pp. 149–157.
  * Tobias (1994) Tobias, S., 1994.  Interest, prior knowledge, and learning.  Review of Educational Research 64, 37–54.  doi:10.3102/00346543064001037.
  * Wade and Kidd (2019) Wade, S., Kidd, C., 2019.  The role of prior knowledge and curiosity in learning.  Psychonomic Bulletin & Review 26, 1377–1387.  doi:10.3758/s13423-019-01598-6.
  * Weitz (2021) Weitz, K., 2021.  Towards human-centered ai: Psychological concepts as foundation for empirical xai research.  it - Information Technology 64, 71–75.  doi:10.1515/itit-2021-0047.
  * Whitehead (1929) Whitehead, A.N., 1929.  The Aims of Education and Other Essays.  The Free Press, New York, NY, USA.
  * Wigfield and Eccles (2000) Wigfield, A., Eccles, J.S., 2000\.  Expectancy–value theory of achievement motivation.  Contemporary Educational Psychology 25, 68–81.  doi:10.1006/ceps.1999.1015.
  * Williams and Lombrozo (2010) Williams, J.J., Lombrozo, T., 2010\.  Explanation constrains learning, and prior knowledge constrains explanation, in: Proceedings of the 32nd Annual Meeting of the Cognitive Science Society, Portland, OR, USA. pp. 2912–2917.
  * Wilson and Keil (2000) Wilson, R.A., Keil, F.F., 2000\.  The shadows and shallows of explanation, in: Wilson, R.A., Keil, F.F. (Eds.), Explanation and Cognition. The MIT Press, Cambridge, MA, USA, pp. 87–114.
  * Wojcik (2013) Wojcik, E.H., 2013.  Remembering new words: Integrating early memory development into word learning.  Frontiers in Psychology 4.  doi:10.3389/fpsyg.2013.00151.
  * Wood et al. (1976) Wood, D., Bruner, J.S., Ross, G., 1976.  The role of tutoring in problem solving.  Journal of Child Psychology and Psychiatry 17, 89–100.  doi:10.1111/j.1469-7610.1976.tb00381.x.
  * Zukow-Goldring (1996) Zukow-Goldring, P., 1996.  Sensitive caregiving fosters the comprehension of speech: When gestures speak louder than words.  Early Development and Parenting 5, 195–211.