# Disentangled Representations for Causal Cognition

Filippo Torresan
School of Engineering and Informatics
University of Sussex
Falmer, Brighton, BN1 9RH
<EMAIL_ADDRESS>
& Manuel Baltieri
Araya Inc., Tokyo, Japan
and
School of Engineering and Informatics
University of Sussex
Falmer, Brighton, BN1 9RH
<EMAIL_ADDRESS>
Corresponding author.

###### Abstract

Complex adaptive agents consistently achieve their goals by solving problems
that seem to require an understanding of causal information, information
pertaining to the causal relationships that exist among elements of combined
agent-environment systems. Causal cognition studies and describes the main
characteristics of causal learning and reasoning in human and non-human
animals, offering a conceptual framework to discuss cognitive performances
based on the level of apparent causal understanding of a task. Despite the use
of formal intervention-based models of causality, including causal Bayesian
networks, psychological and behavioural research on causal cognition does not
yet offer a computational account that operationalises how agents acquire a
causal understanding of the world. Machine and reinforcement learning research
on causality, especially involving disentanglement as a candidate process to
build causal representations, represent on the one hand a concrete attempt at
designing causal artificial agents that can shed light on the inner workings
of natural causal cognition. In this work, we connect these two areas of
research to build a unifying framework for causal cognition that will offer a
computational perspective on studies of animal cognition, and provide insights
in the development of new algorithms for causal reinforcement learning in AI.

_Keywords_ causal cognition $\cdot$ causal reinforcement learning $\cdot$
disentangled representations $\cdot$ disentanglement

##  1 Introduction

Causal cognition, the ability to acquire and exploit causal information about
oneself and the world, is a core aspect of adaptive and intelligent behaviour,
in non-human and human animals [65, 149, 203, 62]. At the same time, in recent
years it has become apparent that artificial systems displaying various forms
of seemingly intelligence behaviour still fall short of performing at the
level of the majority of non-human animals that showcase various kinds of
causal cognitive abilities [124, 162, 38, 195]. It has thus been suggested
that an understanding of the mechanisms of causal cognition will play a
crucial role in cognitive science and artificial intelligence for the next
decades [180, 124, 162, 195, 129, 126, 67, 75].

The study of causal cognition both in human and non-human animals has a long
history, with roots in behavioural studies trying to establish the extent to
which an organism’s behaviour reflects proper causal understanding of the
world instead of simpler forms of associative learning [206, 65, 63, 220, 70,
202, 163, 203]. Some of the most influential studies in this area have
combined theoretical and modelling work based on the formalism of causal
Bayesian networks to account for the cognitive performance of various subjects
in tasks designed for testing causal cognition abilities [66, 64, 63, 220, 71,
70].

Unlike research on learning from reinforcement in behavioural psychology,
which played a pivotal role in the emergence of a mature and rich formal
characterisation of modern reinforcement learning [212], the wealth of
experimental findings in causal learning has not yet been translated into a
set of computational principles for a coherent, pragmatic framework showing
how agents _acquire_ a causal perspective of the world by acting in it. In
other words, most approaches set aside the core question about causal
cognition: how are causal models acquired, or _learnt_ , by agents in the
first place? Mainstream Bayesian frameworks overlook this question because
they tend to describe processes of causal inference _with a model_ [22], i.e.
assuming that the cognitive capacities of agents in a certain context can be
assessed using a model encoding causal variables and relationships postulated
by the scientist. This descriptive approach however fails to provide a clear
(and testable) account of how a causal viewpoint can emerge from an agent’s
first-person experience, i.e. _within a model_ [22].

On the other hand, the agent-perspective focusing on computational models that
implement processes of causal learning and reasoning is emerging as a dominant
area of research in the field of artificial intelligence (AI), following a
surge of interest in causality by the machine learning and deep learning
communities [181, 182, 166]. In these areas, several works have proposed new
unsupervised methods to learn causal structure from data (causal structure
learning or causal discovery) [165, 8, 53, 141, 142, 110], while others have
designed new reinforcement learning algorithms based on some of the principles
and ideas developed in causality research [205, 189, 173, 97, 280, 132, 127,
68, 103]. Others have also recognised the importance of causality for robotics
[21, 84] and new benchmarks and datasets have been introduced to study more
rigorously causal inference in AI [4, 255, 137, 18, 37, 38].

In this work, we introduce a unifying treatment of research in causal
cognition in non-human animals and causal machine learning, with the goal of
laying the foundations for a research program where 1) computational work in
causal machine learning can help us to gain clarity on the principles driving
the acquisition of causal knowledge in adaptive agents, and where 2) causal
cognition can in turn inspire the development of new algorithmic
implementations in causal machine learning. In particular, we will provide an
explicit blueprint for a theoretical and computational framework centred
around the notions of disentanglement and causal representation learning [182,
157, 213], similar to the process of functional specialisation in neural cells
[260, 259], that can form a more rigorous basis for conceptual
characterisations of causal cognition in terms of explicitness, sources and
integration of causal information [208, 263, 264, 262, 265, 62]. Our proposal
will bridge the gap between studies of causal cognition, on one side, and
mathematical and computational models of adaptive behaviour, on the other,
allowing hypotheses about the nature and emergence of different constitutive
blocks of causal learning and reasoning to be tested using the power of modern
causal machine learning models.

In section 2 we will start with an overview of the literature on causal
cognition with a heavy focus on non-human animals, briefly going through some
of its most relevant paradigm shifts leading to the modern proposals
addressing causal understanding. Section 3 will go through a conceptual
framework proposed to unify different accounts of causal understanding and
developed in order to characterise causal understanding in terms of causal
information on a more finely grained scale that includes three dimensions:
explicitness, sources and integration of causal information. Using some of the
formal work presented in section 4, collecting the useful notions of
disentanglement, structural causal models and (partially observable) Markov
decision processes, in section 5 we will see how it is possible to
operationalise computationally causal understanding in modern work on deep
(reinforcement) learning. Section 6 will provide a comparative analysis of
work on natural (animal cognition) and artificial agents
(machine/reinforcement learning) showing common areas of interest between
causal cognition and causal machine learning and highlighting the main
differences between the two. Finally, in section 7 we discuss how these two
lines of research could benefit from each other’s insights moving forward,
speculating on proposals that combine them in new ways.

##  2 Causal cognition

###  2.1 Causal cues and the debate on associative vs. cognitive explanations

Early work on causal cognition in human and non-human animals focused on how
subjects learn about the strength of cue-reward relationships, where some of
the given cues could be attributed the “causal power” of eliciting rewarding
outcomes [206, 113, 32]. A significant part of this early work can be
contextualised within an old debate on whether causal learning is just a form
of associative or contingency learning, the dominant theoretical framework to
study animal learning [172], or a form of learning that requires more
cognition-laden processes.

The crux of this debate was not whether there are causal relationships or
structures in the world, which is more a philosophical type of issue. Granted
that there are, associative accounts have usually tried to show that the
successful performance of some agents in seemingly causal learning tasks can
be modelled, and ultimately explained, by means of purely associative learning
mechanisms. Roughly, these would track the relevant causal associations
between certain variables without the need of invoking more sophisticated
cognitive processes or structures involving a notion of causality [194, 193,
44, 43]. In contrast, other works have highlighted how certain behavioural
responses, especially from humans, are indicative of causal models that the
agent in question would exploit to reason about causal relationships of
various sorts [246, 249, 248, 20].

A paradigmatic example of how this debate has typically unfolded can be found
within the analysis of _backward blocking_ in conditioning experiments with
rats. In these experiments, test subjects are exposed to cues, say
$\mathbf{C}_{1}$, $\mathbf{C}_{2}$, and the compound cue given by their
combination, i.e. $\mathbf{C}_{1}\mathbf{C}_{2}$, that may or may not lead to
a rewarding outcome, indicated by $\mathbf{+}$ (see fig. 1). After some
trials, the subjects, often rats, learn about the relationships between those
cues and the reward, and react accordingly when similar cues are shown. In the
backward blocking scenario, after rats have been trained with cue-outcome
sequences like $\mathbf{C}_{1}\mathbf{C}_{2}\mathbf{+}$,
$\mathbf{C}_{1}\mathbf{+}$ (in that specific order), one finds that they don’t
react to the presentation of $\mathbf{C}_{2}$ alone in subsequent trials. The
response to $\mathbf{C}_{2}$ has been “blocked” by the animal upon witnessing
$\mathbf{C}_{1}\mathbf{+}$, i.e. the causal role of $\mathbf{C}_{2}$ is
reconsidered after understanding that it is not involved in producing the
reward.

Figure 1: An illustration of the phenomenon of backward blocking in rats..
Subjects are conditioned to elicit a response (salivation) to a stimulus
(presence of food) by means of a compound cue (light + acoustic tone) as well
as a single cue (light). When tested with the other cue (acoustic tone), the
rats do not react as strongly as if they understood that in the compound-cue
trials the only cause of the reward was the first cue (light).

This sort of retrospective evaluation (of what happened in a earlier trial) is
a problem for associative accounts because they usually assume that a cue-
outcome association can increase or decrease in strength only when the cue is
present (together or without the reward). However, in backward blocking
scenarios the change in behavioural response to $\mathbf{C}_{2}$ occurs
following the presentation of $\mathbf{C}_{1}\mathbf{+}$ and despite the fact
that $\mathbf{C}_{2}$ has always (or most of the time) appeared at the same
time as the reward $\mathbf{+}$. An advocate for causal models would see
retrospective evaluation as an example of their influence on cue-outcome
learning. Given the evidence represented by $\mathbf{C}_{1}\mathbf{+}$, the
decreased response towards $\mathbf{C}_{2}$ could be explained in terms of a
re-evaluation of the role played by that cue when it appeared as part of the
sequence $\mathbf{C}_{1}\mathbf{C}_{2}\mathbf{+}$. Such evidence would in turn
suggest that $\mathbf{C}_{2}$ was not included as part of a causal
relationship with the rewarding outcome.

At the same time, over the years several revisions of traditional associative
accounts to model retrospective evaluation have been proposed to account for
backward blocking and more complex scenarios involving higher-order relations
between cues, i.e. relations between two cues that never occur together but
that appear in combination with another cue, see for instance [44]. However,
these revisions usually depart from traditional associative principles in
significant ways, e.g. requiring more sophisticated information-processing
capabilities, see for instance the discussion in [163] and references therein.
This thus leaves us with architectures based on, or inspired by associative
principles [42], begging however the question of whether these models are
still associative, or not.

More recently, the whole dichotomy between associative and cognitive
explanations has been put into discussion, as different works argue that
empirical and theoretical grounds for such a distinction are too weak [163,
23, 88, 81]. Indeed, one could point out that the whole debate is a
consequence of a narrow view of what counts as cognitive, e.g. exclusively
representations and processes that have an “internal semantic or propositional
structure” and that shape thought and behaviour via “structured inference”
[193], hence a legacy of cognitivism. It is this narrow and demanding view of
what defines a cognitive representation or process that generate an unhelpful
contrast with simpler and more basic ones in the explanation of mental
faculties.

Focusing on the friction between these two views make us blind of the
possibility that cognition ought to be considered on a spectrum. To witness,
more recent perspectives have often extended the realm of what counts as
cognitive [146, 12, 14] to more basic, low-level biological processes, or have
described cognition, from perception to action and higher-order functions, as
primarily a matter of Bayesian inference on different spatiotemporal scales
[34, 93, 30]. Pursuing this line of reasoning suggests then that the contrast
between associative and cognitive accounts could be simply reframed as that
between lower-level and higher-level cognitive processes placed on a common
spectrum. In this view, the fundamental questions become where processes
regarded as part of causal cognition can be found on the cognitive spectrum,
whether they can be characterised in terms of simpler building blocks, and how
they might interact with each other to realise some form of causal
understanding.

###  2.2 Causal understanding as a building block for causal cognition

Moving past the associative vs. cognitive debate using a more comprehensive
definition of cognition at different levels has however brought forward a
perhaps more fundamental disputes about the presence, or not, of forms of
causal understanding in agents, and what such an understanding amounts to.
This is especially evident in the behavioural research on causal cognition in
non-human animals, where the goal is to design behavioural tasks specifically
intended to try and measure some manifestation of causal understanding [206].
In other words, tasks that would ascertain whether a subject is capable of the
feats of causal cognition, where the assumption is that a solution of the task
requires certain causal, cognitive strategies.

An example of this research is represented by studies on capuchin monkeys
using the trap-tube task [243, 241, 239, 242], where causal cognition is
characterised as the comprehension of key cause-effect relationships within
the task. The trap-tube task consists in pushing a food reward out from a
transparent tube (anchored to the floor) using some kind of tool (e.g. a
stick), by inserting it into one of the tube’s two openings, see fig. 2(b). In
general, capuchin monkeys struggle to learn how to solve the task, either
because they would pick the wrong kind of tool (a stick that could not be
inserted into the tube because of its shape) or because they would pick the
wrong side to put the stick, making the reward fall into a trap positioned
underneath the tube.

The persistent error patterns of the (few) subjects that could solve the task
after extensive trial and error are thought to be evidence of a distinction
between 1) successful performance based on a “stroke of luck” after extensive
active experimentation, and 2) successful performance based on an
understanding of relevant causal variables inherent within the task
requirements [243, 239]. It is in fact well known that capuchin monkeys have a
propensity to produce a wide variety of actions and complex combinations
thereof, even involving tools, to the point that they could be described as
expert tool-users. Because of this, it is unclear whether they have an
appreciation of the causal relationships between their behaviour and the
resulting outcome. In other words, they might learn that using certain tools
is an effective way to achieve certain results, but they may not appreciate
the reasons for why their actions are successful [239].

In contrast, experimental evidence in chimpanzees suggests that they may have
an understanding of the causal relationships between certain actions and their
associated outcomes [134, 155, 186]. The key finding here is that some
subjects, tested with different configurations of the trap-tube task, were
able to select the right side of insertion (almost) immediately, allegedly
displaying an ability to plan their actions according to the different causal
relationships present in the task configuration. Consequently, this evidence
suggests that the successful subjects were not using heuristics such as a
distance-based rule, which would for example determine the correct action
based on the distance of the reward from the tube’s openings without an
understanding of the causal structure of the problem. Instead, subjects
appeared to take into account the causal features of the task configuration
and choose beforehand what action to perform. This would thus amount to a
representational strategy that delineates the key requirements of the task in
advance and results in the correct behaviour without the need of extensive
trial-and-error learning. More specifically, one could argue that those
successful chimpanzees exhibited some kind of causal understanding of the
consequences of pushing the stick inside the tube (but see Martin-Ordas2008,
Seed2009 for opposing views). Unlike for instance the capuchin monkeys of
other experiments [243, 239], where a constant monitoring of the effects of
one’s ongoing action (to check one is on the the right track) and attempting a
variety of actions’ combinations (in the hope to find the right one) was
instead unnecessary.

In a subsequent review paper, [240] argue that an organism can be said to
understand causal relationships, or to posses a causal understanding of (parts
of) the world, if and only if they are able to see or posit some mediating
forces (or variables) between two associated events. This kind of explanatory
attitude is informally described as the key component of causal understanding,
one that helps an organism to envisage and navigate the web of causal
possibilities of the “how” and “why” events at different points in time may be
connected. This kind of understanding has an impact on the behavioural
strategies an agent might pick to reach certain goals, as novel ways of
manipulating the environment are disclosed, targeting those specific mediating
forces [240]. In this view, an organism equipped with causal knowledge is
therefore capable of dealing with unexpected signals from the environment in
ways that take into account different possibilities in a farseeing way [240].
Following this conceptual definition, it is thus worth focusing on the
relation between causal understanding and behavioural strategies, and more
specifically on whether agents can plan ahead how to obtain action-driven
outcomes arising from a mastery of the causal structure of the world as
opposed to a more basic perception-based understanding of only the causal
structure governing one’s observations (cf. the distinction between model-
based and model-free reinforcement in section 6.1.1).

###  2.3 Causal interventions and tool use

A strong candidate for the presence of plans based on action-driven outcomes
is the ability to produce a causal intervention, an action that involves a
causal control on a particular effect [240]. On the one hand, this seems to
provide strong evidence for causal cognition since producing an intervention
requires some form of causal understanding. In particular, it requires an
agent to understand that its actions, in the form of movements of its own
body, could be used as external probes for the causal texture of the world
(_cf_. second rung of the _causality ladder_ in [162]).

On the other hand, the attribution of causal interventions to cognitive agents
appears still controversial because there are only limited reports that hint
at intervention-like abilities in, for instance, rats [20, 128] and primates
[244, 62]. At the same time it is unclear whether tool-use, the ability to
skilfully manipulate objects, common in species like corvids [218, 102],
should count as a form of intervention or not. In general, it is not entirely
obvious what the markers of causal interventions are and how to design
experiments that could determine their presence or absence.

Work on rats, for example, suggests that these animals can learn a common-
cause model, where a light being turned on is perceived as the cause of two
effects, a noisy sound and the release of some food. After exposure to
patterns of causal relationships for a certain number of (training) trials,
the rats enter a test condition characterised by a lever that produces a noise
when pressed. Interestingly, it has been reported that after (accidental)
lever presses, rats exhibit a less resolute search for food (measured by the
number of nose-poking in the cage’s hopper) than when the noise is presented
alone. A possible explanation for this behavioural response would regard these
rats as capable of recognising their action (the level press) as an
intervention, an independent self-generated perturbation on one variable of
the learnt causal model. In fact, an effect (noisy sound) cannot be an
indication that a cause is present (light) when that effect is produced by an
intervention (lever press). Therefore, by conceiving of their action as an
intervention on one variable of the learnt causal model, the rats do not
expect that the other effect (food release) will occur, which then induces a
less vigorous search for food [20]. While these findings are consistent with
the claim that rats can differentiate between predictions based on
observations and predictions based on interventions, they do not exhaustively
prove that rats can produce interventions to activate a certain causal path,
in this case the one leading from the light to the food dispensation (as
discussed by [20] themselves).

Work on corvids on the other hand, see for instance [216], testing New
Caledonian crows with a few variations of the trap-tube task (see fig. 2(b)),
suggests that they possess critical causal understanding abilities, e.g. an
appreciation of causal relationships involving object-hole interaction, on
which their exceptional tool-using skills might be built. Similarly,
Jelbert2014, Logan2014a present experiments on a task (inspired by Aesop’s
fables) in which crows have to learn to drop some objects (e.g. stones) into
the right water-filled tube so that the water displacement brings a floating
reward (e.g. a piece of meat) closer to the tube opening (see fig. 2(a)). The
results here point at the fact that the birds managed to solve the task,
seemingly by attending to the relevant causal information, e.g. the fact that
larger and not hollow objects will produce a bigger water displacement. For a
variation of the task however, where the setup instead consisted of three
water-filled tubes arranged in a row on a wooden board, with some space
between each other, results were less clear. In this task, the baited tube is
the one in the centre and, crucially, it is connected with one of the others
by means of a U-shape tube hidden from view (located underneath the board).
Dropping objects into one of the lateral tubes will have as an effect a water-
level rise in the baited tube. Since the central tube is too narrow to drop
anything in it, to bring the reward on the surface it is crucial to recognise
this counter-intuitive effect and exploit it, i.e. to infer the presence of
and reason about hidden causal mechanisms. Here, all tested birds performed at
chance level, meaning that they dropped objects randomly on either of the two
lateral tubes [105], see also [140] for similar conclusions on a slightly
modified setup.

(a) Floating-reward task. An animal subject learns to drop stones into a
water-filled cylinder to raise the water level and reach a piece of food.
Image credit: [152], adapted under the terms of the Creative Commons
Attribution License.

(b) A variation of the trap-tube task. An animal subject learns to retrieve a
reward from a transparent tube by pushing it out by means of a stick. In this
variation of the task, the crow has to slide the reward over the open hole
(the one on the right). Image credit: [29] under Elsevier user license.

Figure 2: Causal cognition tasks.

All this evidence points at the fact that causal understanding could be a key
notion for a more systematic study of causal cognition. At this stage,
however, debates on its presence, role and features make it unsuitable for
more practical investigations (see, e.g. Hennefield2018, Hennefield2019 for a
critical meta-analysis of the works on corvids described above). After
reviewing some of the main themes driving current research on causal
cognition, moving away from the debate on associative vs. cognitive and
embracing the challenge of determining what constitutes causal understanding
and how to infer it from behavioural experiments (e.g. using tasks involving
causal interventions or tool-use), we thus next look at how a large body of
research in this areas has been recently organised in a new conceptual
framework attempting to capture the more fundamental dimensions of research
works on causal understanding.

##  3 A Conceptual framework for causal cognition

After decades of research in the field, by now it is evident that different
works on causal cognition have often appealed to different conceptualisations
of the subject matter, to the point that a consensus has yet to be formed on
what even constitutes causal understanding, see Penn2007, Sloman2015 for some
reviews, and the contributions to Gopnik2007, McCormack2011 for other
perspectives. As briefly illustrated by the previous section, different lines
of work place causal understanding at different levels of a hypothetical
cognitive spectrum, and have portrayed very diverse views on how to
characterise it in terms of key cognitive functions and behavioural outputs.
Most researchers might agree on the idea that a causal agent has the kind of
behavioural flexibility that is unattainable by agents lacking causal
understanding. Yet, the variety of positions trying to describe what underpins
it appears to only contribute to the confusion.

Some works point to a representational strategy for agents to picture in
advance what the key causal features for solving a task are. This would then
characterise a distinction between performing and understanding, i.e. whether
the solution of a task is achieved via some sort of shortcut, or in a robust
and reliable way [243, 241, 239, 134]. Others are more demanding, and see
causal understanding as the result of some form of causal reasoning, yet
another ambiguous expression that has been described in different ways. For
instance, causal reasoning could involve structural or symbolic (causal)
knowledge abstracted from perceptual cues [187, 168, 149], or in other words,
the ability to search for cause-effect relations that could reveal how and why
two events are connected, or why some actions lead to certain outcomes (i.e.
diagnostic causal reasoning), requiring thus the presence of some causal
beliefs [240, 45]. Others would further maintain that without an ability to
perform causal interventions, perhaps even involving unobservable entities
(see hidden causal mechanisms in section 2.3), it is unlikely that an agent is
able to grasp causality as opposed to just behave _as if_ it does. Going back
to tool-use then, the question of whether adaptive tool use may reveal the
presence of some of the abilities just described or whether it may be a
confounder instead [186, 218, 102] remains unanswered.

In a recent attempt to put causal cognition research on a more precise and
coherent footing, we find different proposals discussing experimental findings
framed with respect to a few recurring themes, drawing attention to key
aspects of causal cognition [263, 264, 262, 265, 208] (see also [62] for a
recent review). [208] in particular dissect the main disagreements over causal
understanding in non-human animals, proposing a more precise way to study
causal cognition using a three-dimensional conceptual framework inspired by
and overall consistent with other conceptual treatments [263, 264, 262,
265]111In the remainder of this work, the main argument will be built around
the particular characterisation found in [208] while references to the other,
previous conceptualisations will be made when a closer analysis makes it
necessary.. The starting point is to regard causal cognition as the processing
of causal information, understood as information about the nature of certain
causal relationships, instead of referring to the more ambiguous and ill-
defined notion of causal understanding. One of the advantages of this move is
to put on one side normative issues, e.g. what really counts as causal
understanding, and instead focus on empirically tractable parts of the matter
[208]. To a first approximation, the main idea is to score the performance of
subjects on causal tasks (see section 2) along three dimensions that have the
potential to cover the full spectrum of causal cognition. With these as a way
to ground the discussion of different empirical results, [208] then suggest
ways to draw a more fine-grained comparison of the extent to which non-human
animals and humans process causal information. More specifically, following
[208] causal information processing can be characterised along three
dimensions:

  1. 1.

the level of explicitness of causal information,

  2. 2.

the sources of causal information, and

  3. 3.

the level of integration of causal information.

###  3.1 The explicitness of causal information

The explicitness dimension, refining some intuitions presented in [262], aims
to capture a spectrum of causal information where on the one end, implicit
models are essentially blind to causal relationships. These models represent
cases where actions and outcomes/rewards are entangled or “fused” [262], i.e.
models based on an associative correlations where the causal structure (see
the web of causal possibilities in section 2.2) is essentially hidden and
inaccessible to the agent. In this class of models, agents cannot necessarily
come up with a complex plan on how to adjust their actions that is sensitive
to the web of causal possibilities in order to achieve a certain goal, since
they lack or have a limited understanding of their own actions and other
variables in the environment as causally relevant for bringing about certain
outcomes or rewards [262]. They can however take actions in a less structured
way, for example using knowledge acquired from repeated trial-and-error in an
associative manner, leading to a continuum of explicitness that is apparent in
several experimental studies as seen in section 2. For instance, associatively
pairing the action of pressing a button (cause) with the presence (very often,
but not always) of some food (end goal, an effect) can be considered as an
example of implicit model.

On the other side of the spectrum we find explicit models, models where the
causal structure is completely unpacked and relations between actions and
outcomes/rewards are disentangled and available for an agent to take advantage
of. Looking at the previous example, we can imagine a different scenario where
an agent realises that a button press will activate a food dispenser mechanism
(some intermediate variable) and that the food will become available if and
only if there is no obstructing object in the mechanism. In this case, the
action of clearing the dispenser from the obstructing piece is an action that
can be said to require a more explicit understanding of the causal structure
of the world, at least compared to the first situation, an operation that is
directed at altering one of the intermediate causal variables, the object
obstructing the food dispenser, so as to obtain the food.

A qualitative description of explicitness thus amounts to establishing what an
agent can do with the acquired causal information, for example by
investigating an agent’s degree of flexibility in using what is has learned in
a causal task (e.g. clearing the dispenser mechanism of the obstructing
object). To a first approximation, the key idea is that the more explicit a
model is, the more causal information is available to an agent, because the
means to reach a certain goal have been recognised as distinct from each other
and from the goal itself (the mediating variables of a certain causal
influence have been identified, _cf_. [161]), thereby leading to a higher
degree of flexibility in behavioural responses.

To see how different degrees of explicitness appear in the animal cognition
literature, we can take a closer look at the research on the trap-tube task
described in section 2.2. Facing the trap-tube, an agent that can only form
implicit models where actions and outcomes are entangled, i.e. where actions
$\mathbf{a}$ leading to states $\mathbf{s}$ are a-causally associated to
outcomes $\mathbf{x}$, has a very limited ability to discern the possibly
relevant intermediate variables that could be exploited to reach the goal.
These include for instance the position of the trap, necessary for an
understanding of whether its opening (the hole) is on the tube’s lower surface
or not, i.e. whether it affects the desired outcome (recall that in the latter
case the tube has been rotated so the trap is ineffective).

As the literature on these experiments suggests that most capuchin monkeys
that were tested do not seem to appreciate the relevance of the trap, and
consequently perform poorly when the tube is inverted, we could say that these
agents rely only on implicit models of the form “ _insert the stick, out comes
the reward_ ” [239]. While one could say that this associative rule encodes
some causal information, it is evident that this only happens in a very
implicit and vague manner, leading to maladaptive behaviour in most other
contexts, especially without retraining.

In contrast, causal representations capture the relevant difference-making
relationships present in the task at hand, e.g. the role that the trap plays
in the trap-tube. Such relationships express fine-grained information that
shows the different causal links between instrumental/intermediate variables
and outcome/reward variables, enabling more flexibility in planning or action
selection.

More in general, explicitness can be evaluated by assessing an agent
performance in contexts where some kind of knowledge transfer is required. One
scenario might involve adjusting a learned strategy to solve a similar task.
For example, an agent might have learned that a rake can be used to fetch a
coconut that fell into a pond without getting wet. But the rake could also be
used to detach a coconut from a palm tree without having to climb to its top,
and make it fall on the ground for easy retrieval and tasting. One could also
imagine a more drastic context change requiring completely different means for
performing successfully in the same task, e.g. if the rake is not available,
the subject could look for a similar or different object that may play the
same functional role.

Another scenario might instead require using previous knowledge or learned
strategies to solve new tasks, e.g. exploiting the same means for a different
end. For example, if an agent has learned that the rake can be used to bring
desirable items closer, the same agent should be able to use the rake in other
contexts, e.g. to retrieve other types of food. Finally, one might think of
scenarios where knowledge transfer also demands paying attention to different
functional properties of the same means (or others) because such properties
are now relevant to the solution of a different problem problem. In this case,
according to [208], the cognitive capacities in question would amount to some
form of insight learning. For example, after having retrieved the coconut, our
agent could use the rake to discourage greedy conspecifics or other animals
from stealing the just-earned meal.

###  3.2 Sources for learning causal information

The sources dimension of [208] is also inspired by [262], where causal
cognition is proposed to be best explained in terms of some key, distinct
abilities, namely, egocentric and non-egocentric sources of causal
information.

Egocentric causal information captures the idea of an agent that can acquire
an understanding of the causal structure of the world from its own behaviour,
focusing on the performance of their own actions and how they can can reliably
and robustly result in certain outcomes. For example, a subject could learn
that performing action $\mathbf{a}_{1}$ makes a difference for obtaining
outcome $\mathbf{x}_{1}$ but not for $\mathbf{x}_{2}$. This is the realm of
instrumental conditioning (or learning) investigated extensively in animal
research [262].

Non-egocentric causal information can, on the other hand, be obtained from two
main sources: the behaviour of other agents and the unfolding of natural
events. Social causal information implies that an agent can learn about
important action-outcome contingencies by paying attention to other agents’
behaviour. For instance, observations of a conspecific performing action
$\mathbf{a}_{1}$ and reliably obtaining outcome $\mathbf{x}_{1}$, but not
$\mathbf{x}_{2}$, provide important causal information for a subject that is
aiming at outcome $\mathbf{x}_{1}$ (or $\mathbf{x}_{2}$).

Natural causal information similarly suggests that events in the natural world
can disclose ecologically important causal relationships. For instance,
observing a piece of fruit falling from a tree-branch shaken by the wind could
reveal to an attentive observer important causal information on how to get
some food, as long as it is capable of performing a causal analysis of the
situation [233]. Compared to the acquisition of the previous types of causal
information, natural information imposes, arguably, a higher cognitive load on
the subject because the event in question does not tell the subject what
action may be (causally) relevant and for what reason(s). In other words,
there is an additional cognitive effort that the subject needs to make to
appreciate that, given certain observations, action $\mathbf{a}_{1}$ can
produce outcome $\mathbf{x}_{1}$. [208] remark that empirical evidence so far
suggests that egocentric causal learning is the most widespread in the animal
kingdom whereas social causal learning and observational causal learning (as
they call the two more sophisticated forms of causal information acquisition)
are fully present in adult human beings only. Following [262], they agree with
the idea that in principle these forms of learning could dissociate but they
also add that it is not clear the extent to which these abilities are
independent from one another, or whether they form a hierarchy of cognitive
processes.

###  3.3 The integration of causal information

Integration appears more ambiguously in [208] but can be understood, in
general, as consisting of operations of update, combination, extension etc. of
one source of information with another one to form a coherent structure. More
concretely, in our framework integration will be later framed in terms of
meaningful combinations of different sources of information (egocentric,
causal and natural) that can describe if and when agents are capable of
translating observations from different perspectives into their own
(egocentric) perspective, forming new sources of egocentric + social causal
information, egocentric + natural causal information or  complete causal
information (egocentric + social + natural), or whether social and natural
information can be integrated without a direct effect on the agent own
egocentric perspective to form social + natural causal information.

While laying the foundations for formal characterisation of causal cognition
across the animal kingdom, the inherent ambiguity of not only integration, but
of explicitness and sources too, mixed with the more general focus on high-
level discussion over a clear operationalisation, pushes the idea that the
conceptual framework proposed by [208] is in several ways still heavily
relying on interpretative work to be done by the reader. In the next section
we fix some core ideas in a mathematical language that will form the basis of
our proposal to formalise [208]’s work in a pragmatic way in section 5.

##  4 A mathematical framework for causal cognition

###  4.1 Disentanglement in machine learning

A key proposal in modern approaches to deep (reinforcement) learning is that
of disentanglement [16], roughly stating that in order to acquire an
understanding of the causes behind some given observations, it is necessary to
interpret those causes as distinct (high-level) factors, and recognise the
different causal power they exert when giving rise to observations [182]. For
example, if we see a red ball made of rubber bouncing on the ground, what
makes it bounce? While different factors including colour, shape, and
material, are intertwined and together produce observations captured by our
eyes, some of these factors have no causal influence on the bouncing
behaviour, i.e. colour. According to the disentanglement hypothesis, the
ability to discern different factors is thus a crucial step in a theory of
causal understanding.

Similarly to other influential proposals, disentanglement has been used to
characterise a general intuition based, however, on different implementations
and interpretations. Following [278]’s review, we thus look at some of the
common structure behind different definitions of disentanglement. To do so, we
focus in particular only on sets and functions between them. This is
technically equivalent to stating we are working in the category of sets and
functions, $\mathbf{Set}$ [147], and while in various ways limiting, this
allows us to focus on the central parts of a our proposal to connect
disentanglement to explicitness down the line using a relatively simple
mathematical toolkit (sets and functions, without introducing more advanced
tools from category theory).

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$

Figure 3: Disentanglement for a bouncing red ball. Following the example
describe in the text, we sketch here (some possible) factors, observations and
codes for a system disentangling factors from observations into codes. For
such a system to understand what make a red ball bounce, we consider shape and
colour as factors generating observations about different balls (round and
punctured balls, and of different colours, red and blue), inside the dashed-
line oval shape at the centre of the figure. Other observations can be part of
the standard repertoire of observables for our system (say, dogs or trees),
but their factors are not explicitly drawn as we only focus on (some aspects)
of a system capable of disentangling factors that generated observations of
different kinds of balls. A disentangled representation is one that
“faithfully” maps factors to codes via the given observations, according to
the assumptions provided in the main text, see definition 4.1.

Following [278] we start by defining $S,X,Z$ as state-spaces, represented as
sets of (generative) factors, observations and codes respectively. The sets of
factors and codes are further assumed to be (Cartesian) products of
$n\in\mathbb{N}$ factors and $l\in\mathbb{N}$ codes:

$\displaystyle S$ $\displaystyle=S_{1}\times S_{2}\times\dots\times S_{n}$
$\displaystyle Z$ $\displaystyle=Z_{1}\times Z_{2}\times\dots\times Z_{l}$ (1)

At a high level, the main idea driving this framework is that representing
distinct factors, i.e. having disentangled codes that faithfully map to
disentangled factors, is the starting point for acquiring a causal
understanding of the world: an agent with knowledge of what factors generated
its observations is an agent that understand what data-generating mechanisms
brought observed data to its sensory peripheries, see fig. 3 for a way to
frame the example of the red ball in this initial setup, to be unpacked next.
The setup for disentangled representation can then be captured, in a compact
form but with more specific constraints to be imposed below, by the following
commutative diagram:

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$${S}$${X}$${Z}$${\begin{array}[]{l}S:\text{factors}\\\
X:\text{observations}\\\
Z:\text{codes}\end{array}}$$\scriptstyle{g}$generating$\scriptstyle{m}$modularity$\scriptstyle{f}$encoding$\scriptstyle{i}$informativeness
(2)

We introduce next the formal definition of disentanglement that we will be
referring to throughout this work, following [278].

######  Definition 4.1 (Disentangled representations).

A disentangled representation is a product of codes $Z_{i}$ for
$i\in\mathbb{N}$ defined through the following:

  * •

a generative process or data generating process, as a function

$\displaystyle g:S\to X$ (3)

that gives rise to observations $X$ from the product of $n$ factors
$S=S_{1}\times S_{2}\times\dots\times S_{n}$; this function is assumed to be
injective since we ask that if two observations are identical, they must be
mapped to by two identical products of factors, but also allow for the
possibility that some observations may not be mapped to from any factor (i.e.
$g$ is not surjective, which can be seen as stating that the state-space of
observations can be of much higher dimensionality compared to the state-space
of factors; intuitively, the space of observations could include all the
possible scenes a subject has ever been exposed to, but the factors $S$
producing data, e.g. about a ball bouncing around, only map to a small part of
all possible observations (all possible variations of the red bouncing ball,
while for example saying nothing about a tree whose leaves are moved by the
wind, see fig. 3),

  * •

an encoding, as a function

$\displaystyle f:X\to Z$ (4)

that maps observations $X$ to the product of $l$ codes $Z_{1}\times
Z_{2}\times\dots\times Z_{l}$ while also requiring $f$ to be injective on
$g$’s image, $g(S)$; for a more intuitive interpretation we consider the
following scenarios:

    * –

$l\geq n$, the case considered here (and in [211], see next section), meaning
that we assume there are potentially more codes than factors and thus all
factors can be encoded by different codes (independently, after we impose more
structure below),

    * –

$l=n$, the case considered by [278] for a slightly simpler treatment, as this
doesn’t anyway imply that factors $S$ and codes $Z$ are the same (and in
practice they often aren’t, since we rarely know what factors $S$ generated
observations in a given dataset),

    * –

$l<n$, relevant for practical implementations (see for instance variational
autoencoders in section 5.1), where we may not have (or want, for reasons
including for instance lossy compression) enough codes to model all factors
due to constraints or wrong modelling choices, in this case we need to
consider approximations to disentanglement, either mathematically [279] or via
some (deep) reinforcement learning implementation, which will be discussed in
section 5.1,

  * •

a modularity map,

$\displaystyle m:S\to Z$ (5)

defined as the composition $m=f\circ g$222Think of $f\circ g$ as
$f(g(\mathbf{s}))$ for some element $\mathbf{s}\in S$., i.e. using $n$ factors
$S$ to generate observations $X$ and then encoding these observations, $g(S)$,
in $n(\leq l)$ codes $Z$, the constraint can also be visually expressed in the
following (string diagrammatic) form to be read bottom to top

(6)

stating that in the case of $l\geq n$333If $l<n$, we can simply replace $n$
with $l$., $m=m_{1,1}\times m_{2,2}\times\dots\times m_{n,n}$, meaning that
the $n$-th code only encodes the $n$-th factor, note that $m$ is injective as
the composition of an injective function, $g$, and a function injective under
its image $g(S)$, $f$,

  * •

an informativeness requirement, modelled as the existence of as a function

$i:Z\to S$ (7)

that maps codes to factors in such a way that $i\circ
m=\mathsf{id}_{S}$444Technically, we say that $m$ is a split monomorphism
[147]. for the identity map on factors $\mathsf{id}_{S}$, see the
following555Notice that since we treat modularity and informativeness as
separate criteria, we don’t include the modular/factorised version of $m$ in
eq. 8.

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=

(8)

i.e. $i$ is a left inverse for $m$, or in other words (post)composing $m$ with
its left inverse $i$ means that factors can be recovered (we have identities
on the right hand side of eq. 8), it doesn’t however tell us that they will be
disentangled ($i$ is not itself modular/factorised),

  * •

a disentanglement requisite on informativeness, this ensures that the codes
are _independent_ faithful representations of different factors and can be
stated using the following diagram

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$i$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$Z_{1}$$Z_{2}$$Z_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$

(9)

where $i=i_{1,1}\times i_{2,2}\times\dots\times i_{n,n}$; this condition is
particularly relevant because there may be cases where, for instance, without
a proper causal understanding of the factors generating a bouncing ball, one
might mistakenly assume material and colour, together, are a relevant factor:
if all the balls an agent ever saw bouncing were red and made of rubber,
colour and material could be assumed to be _jointly_ necessary factors to
understand how the ball bounces666Further constraints, of general importance
for a more precise definition of disentangled representations that can deal
with undesirable special cases (e.g. redundancy of information in the codes)
can be found in [278] but will not be further discussed here..

Combining these conditions, we obtain finally the following diagram for
disentangled representations $Z_{1}\times\dots\times Z_{n}$ satisfying the
following equation

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$i$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$Z_{1}$$Z_{2}$$Z_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=

(10)

In this view, a disentangled representation is thus one that captures or
reflects in a meaningful way the expressivity of data-generating factors
underlying some given observations. While useful as a general guideline for a
more precise notion of disentanglement, it is however still unclear at this
stage how to relate this notion of disentanglement to different dimensions of
causal understanding [208] in a more formal way. So far, we haven’t in fact
discussed how this idea can be connected to formal accounts of causality. To
form the basis of this connection we thus introduce next some key concepts
from causal representation learning. Due to the probabilistic setup underlying
the following definition(s), one could argue that the notion of
disentanglement we reviewed here (using sets and functions) can’t be applied
directly to the models in the following sections. However, relatively
straightforward generalisations of the notion of disentanglement to categories
other than $\mathbf{Set}$, including (Markov) categories handling
probabilistic reasoning in a “function-like” manner, have already been put
forward in [278]. With this in mind, we will from here onwards use
“disentanglement” in a slightly looser way, taking it capture different
flavours of disentangled representations with the same class of structural
desiderata but applied to different, e.g. probabilistic, setups.

###  4.2 Causal representation learning

As highlighted by [278], disentanglement has been described in various
different ways across the literature. In one of the most influential accounts,
disentanglement can be viewed as a component of causal models recovering the
causal factorisation of a process generating a collection of observations of
interest [182, 252]. In this view, disentanglement is thus a crucial part of
the answer to the question of how causal model are acquired, providing a way
to operationalise a process deemed necessary for an agent to _learn_ a causal
characterisation of the world it acts in [182, 252]. As we will see, taking
this perspective allows us to relate explicitness, one of the dimensions of
causal cognition (see section 3.1), to disentanglement, in light of the
nascent field of causal machine learning.

Causal machine learning is a collection of methods and applications based on
the notion that exploiting causal information in data can lead to a more
robust, accurate, and efficient kind of (data or system) modelling, thereby
viewing causality as a fundamental notion to move past some of the limitations
of machine learning methods based on statistical learning (from now on we will
refer to these methods as “traditional machine learning”) [181, 107, 227].

Within this line of research, the subfield of causal representation learning
can be regarded as a way to recover disentangled representations from data.
Traditionally, representation learning has been conceived as the task of
learning a generative model in the form of a low-dimensional feature vector
(codes) of high-dimensional data (observations) produced by a generative
process whose features (factors) remain hidden. The idea driving this approach
is that if those codes capture key, informative, aspects of a dataset, they
would aid in solving downstream tasks (i.e. predicting a label) [16]. However,
these models have often sidestepped questions regarding the origins of
particular datasets, overlooking structural knowledge of the data-generating
process that could have produced them. This in turn affects what a generative
model can account for, often limiting its scope to only statistical
correlations with little to no causal power.

Causal representation learning extends these ideas by bringing into
representation learning (and, more generally, deep learning) some of the
principles, methodologies, and objectives of classic causal inference research
[161, 166, 87], with the goal to learn a low-dimensional vector of _causal_
codes from high-dimensional observations generated by _causal_ factors777Note
that in the causal representation learning literature, both causal codes and
causal factors are usually addressed as simply “causal variables”. Here we
will instead maintain an explicit distinction. [182, 107, 17]. In this
paradigm, the data-generating process can be formalised as a structural causal
model capturing the causal relationships between factors underlying the data
distribution. Importantly, recalling the distinction between generative
process and generative model, learning a generative model means to represent
something about the generative process described as a structural causal model.
In the best case scenario, a generative model recovering the full gamut of
causal information assumed to exist in the generative process can itself be
described as a structural causal model of the same form as the generative
process. This particular scenario assumes however _causal sufficiency_ , i.e.
that there are no hidden common causes (also referred to as hidden
confounders) on factors in the generative process, meaning that every common
cause of any two or more variables is already accounted for and included in
the model (see [207, Ch. 10] and [166, Ch. 9]). More often, we will instead
only look at cases where hidden confounders are present, thereby violating
causal sufficiency, which will then allow us to distinguish between weak and
strong disentanglement in section 5.1888Note that a causal factor or code does
not have to be hidden to be a confounder. Any common cause of two or more
variables has a confounding effect on certain causal relationships (see the
definition of confounding in [166, 113]).. To better understand the role
confounders can play, we define next a structural causal model as the
following.

######  Definition 4.2 (Structural causal model of the generative process).

Given the following:

  * •

a collection $S=(S_{1},\dots,S_{n})$ of $n\in\mathbb{N}$ causal variables, or
causal factors,

  * •

a collection $X=(X_{1},\dots,X_{d})$ of $d\,(\geq n)\in\mathbb{N}$
observables,

  * •

a collection $C=(C_{1},\dots,C_{d})$ of $m\in\mathbb{N}$ confounders,

  * •

a collection $N^{S}=(N^{S}_{1},\dots,N^{S}_{n})$ of $n$ noise variables on
causal variables,

  * •

a collection $N^{X}=(N^{S}_{1},\dots,N^{S}_{d})$ of $d$ noise variables on
observables,

and assuming that all the noise variables are jointly independent, a
structural causal model $\mathcal{C}$ [161] of the data-generating process is:

  * •

a collection $(h_{1},\dots,h_{n})$ of $n$ structural assignments, each
assigning a value to a corresponding causal variables $S_{j}$ for
$j\in\\{1,\dots,n\\}$ based on

    * –

its parents (direct causes) in the set

$\displaystyle\mathbf{PA}_{j}\subset\\{S_{\setminus j},C\\}\quad(\text{where
}S_{\setminus j}\coloneqq S\setminus S_{j})$ (11)

and,

    * –

the noise variable $N^{S}_{j}$,

  * •

and an emission map (or mixing function) $g$ generating observables $X$, cf.
the generating process in definition 4.1,

that is:

$\displaystyle S_{j}$ $\displaystyle=h_{j}\,(\mathbf{PA}_{j},N^{S}_{j}),\qquad
j\in\\{1,\dots,n\\}$ $\displaystyle X$ $\displaystyle=g\,(S,N^{X}).$ (12)

Importantly, one can show that a structural causal model $\mathcal{C}$ entails
a corresponding directed acyclic graph, $\mathcal{G}$, and a unique
probability distribution999For the use of probabilistic language from now
onwards, the following conventions are adopted:  1. $P$ stands for a
probability distribution or function, and $P_{X}$ is the probability
distribution for a random vector $X$ (if one-dimensional, $X$ should be
understood as a random variable), 2. $P(X=x)$ is shortened by simply writing
$P(x)$, i.e. using only the value the random variable takes, 3. $p(x)$ is
either the probability mass function or probability density function evaluated
at $x$ for the probability distribution $P_{X}$, 4. the subscript of $P_{X}$
will be usually omitted if the random variable is clear from context.
$P_{S,X}$ defined over factors $S$ and $X$ [166]. Concisely, a structural
causal model induces a causal Bayesian network, where the conditional
distributions relating causes to effects, e.g. $P(X|S),P(S_{i}|S_{j})$, etc.,
are often referred to as causal mechanisms, capturing ways in which causes
produce effects (the underlying structural assignments of the structural
causal model describe those mechanisms in greater detail, providing a
functional specification). Conversely, we can also think that any empirical
probability distribution has an associated structural causal model that
induces it. In this case, however, it can be shown that such a structural
causal model is not unique. It is nonetheless possible to define an
equivalence class of structural causal models consistent with that same
distribution [166]. Starting from a joint probability distribution $P_{S,X}$
rather than a specific model $\mathcal{C}$, there is a corresponding
equivalence relation, or partition, of directed acyclic graphs with respect to
which the $P_{S,X}$ can be factorised. A particular graph can be chosen, then,
to reflect the particular causal mechanisms of the true structural causal
models.

While crucial for unpacking a notion of disentangled representations with
connections to the literature on causal representation learning, the tools we
introduced so far remain fundamentally rooted in a body of work in machine
learning that does not take into account how agents make use of these tools
for decision making over time. To introduce a notion of agent as a system with
goals capable of solving a particular class of problems we thus look at the
framework of reinforcement learning [212] and review some of its basic
components that will be later involved in our account of disentanglement for
agents. From the perspective of [278], one can reasonably expect that the
definition of disentanglement given in section 4.1 can also be generalised to
classes of models with dynamics (i.e. time-dependent models)101010For example,
by using monoids instead of groups in [278]., however at this stage the
possibly non-trivial interplay between the categorical (in the sense of being
based on category theory) disentanglement of [278] and a comparable
categorical reinforcement learning setup [83] remains admittedly
unclear111111For instance, we can consider bisimulations of POMDPs, described
in section 6.1.2 as a possible implementation to achieve a high degree of
disentanglement, and whether the definition of equivalence classes of states
it embodies includes rewards or not. If it doesn’t, one can obtain equivalence
classes of states for all actions with transition dynamics leading to the same
equivalence classes of states, i.e. a task-irrelevant partition of states
consistent with their dynamics that does not depend on the reward signal
perceived by a particular agent. If it includes rewards on the other hand, one
in general obtains a finer partition that considers states _having the same
reward_ from which transitions for all actions lead to the same equivalence
classes of states, i.e. a partition of states sensitive to instantaneous
rewards and thus to task-relevant aspects of a problem for a particular
agent..

###  4.3 (Causal) Reinforcement learning

Reinforcement learning (RL) provides a natural avenue for ways to combine work
from machine learning and decision making in agents [212]. Similarities
between classical RL and causality have been put forward in previous works
[57, 59, 58], however a clear-cut notion of causality appears to be missing
[107]. Here we provide some background for standard RL implementations, which
will be then placed in context and used once we overview recent work in causal
RL in section 6.

A typical reinforcement learning setup involves the definition of a problem in
terms of (a model of) an environment represented by a (discrete-time) Markov
decision process (MDP).

######  Definition 4.3 (Markov decision process (MDP)).

A Markov decision process is a tuple $(S,A,T,\gamma,r)$, where:

  * •

$S$ is the state space121212Our choice of using $S$ to represent states of the
environment conforms with the distinction between factors $S$ of a generative
process and codes $Z$ representing states of an agent’s generative model.,

  * •

$A$ is the action space,

  * •

$T:S\times A\to P({S})$ is the transitions dynamics, such that for a given
state $\mathbf{s}$ and $\mathbf{a}$, $T(\mathbf{s},\mathbf{a})$ gives a
probability distribution of states $P({S})$ an agent can transition to from
state $\mathbf{s}$ while taking action $\mathbf{a}$, often written as
$P({\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}})$,

  * •

$\gamma\in[0,1)$ is called the discount factor,

  * •

$r:S\times A\to\mathbb{R}$ is the reward function, giving a reward every time
a transition is taken.

Alternatively, it is also common to define a problem as a partially observable
Markov decision process (POMDP), where information of the environment is only
indirectly available through some observations.

######  Definition 4.4 (Partially observable Markov decision process (POMDP)).

A partially observable Markov decision process is a tuple
$(S,A,X,T,M,\gamma,r)$, where $S,A,T,\gamma,r$ follow the definition of an MDP
and

  * •

$X$ is the observation space,

  * •

$M:S\to P({X})$ is the observation or measurement map.

The goal of agents in an RL setup is to select sequences of actions that
maximise expected cumulative discounted reward, also known as expected return,
based on past and current experiences acquired through meaningful interactions
with the environment. Action policies representing sequences of actions are
defined by the following

######  Definition 4.5 (Action policy).

Given an MDP $(S,A,T,\gamma,r)$, a policy $\pi$ is defined as either

  * •

a deterministic function $\pi:S\to A$, or

  * •

a stochastic map $\pi:S\to P({A})$ assigning a distribution of actions to each
state in $S$.

For a POMDP $(S,A,X,T,M,\gamma,r)$, a policy $\pi$ is usually defined instead
as either

  * •

a deterministic function $\pi:B\to A$, or

  * •

a stochastic map $\pi:B\to P({A})$,

where _beliefs_ $B$, encoding probabilities on states given histories of
observation-action sequences $\tau\coloneqq[X_{0},A_{0},\dots,X_{T},A_{T}]$,
$P({S|\tau})$, play the role of sufficient statistics of these histories. In
the case of MDPs, these beliefs are trivialised by the full observability of
the process and the Markov property, such that states observed $S$ at a
specific time step are sufficient statistics of histories of state-action
sequences
$\tau\coloneqq[\mathbf{s}_{0},\mathbf{a}_{0},\dots,\mathbf{s}_{T},\mathbf{a}_{T}]$.
As we will see, in practice policies are often parameterised by the weights
${\bm{\omega}}$ of a neural network that outputs the action probabilities
after processing $\mathbf{s}$, and in this case, they will be defined using
the following notation: $\pi_{{\bm{\omega}}}$.

For probabilistic state transitions and action policies, the expected
cumulative discounted reward is defined as follows.

######  Definition 4.6 (Expected cumulative discounted reward).

Let $S_{t}\subseteq S,A_{t}\subseteq A$ be state and action subspaces of their
respective spaces, indexed by time $t\in T$. We define time-indexed rewards
$R_{t}\subseteq\mathbb{R}$ for $t\in T$ as

$\displaystyle R_{t}\coloneqq r(S_{t},A_{t}),$ (13)

which, in turn, can be used to define the cumulative discounted reward, or
return $G_{t}$, indicated by

$\displaystyle G_{t}\coloneqq\sum_{i=t}^{T}\gamma^{i-t}R_{t}$ (14)

with $i=1$ for rewards over a full trajectory, and $i>1$ for rewards over a
partial one, respectively (the former appears in the next equation, while the
latter will be used in eq. 22).

The _expected_ cumulative discounted reward is then defined as

$\displaystyle J({\bm{\omega}})=\mathbb{E}_{\tau\sim
p_{\pi_{{\bm{\omega}}}}(\tau)}\left[G_{1}\right]$ (15)

with respect to a probability distribution over trajectories,
$p_{\pi_{{\bm{\omega}}}}(\tau)$.

After introducing all the necessary, mathematical background for
disentanglement and causality as treated in this work, in the next section we
will look at how these notions have been described, implemented and studied in
deep (reinforcement) learning. This step will in particular help to bridge the
gap between the formal, but somewhat simplified definition presented in this
section (with agents modelled with, or without, disentangled representations
and in a causal or simply a-causal way) and measures on degrees of
disentanglement that can tell us _how much_ disentanglement and causal
understanding can be found in different systems, i.e. how far subjects are
from the ideal disentangled and causal representations introduced in this
section.

##  5 A computational framework for causal cognition

###  5.1 Explicitness as degrees of disentanglement

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$i$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$Z_{1}$$Z_{2}$$Z_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$X$$Z$$X^{{}^{\prime}}$$q_{\bm{\phi}}(\mathbf{z}|\mathbf{x})$$p_{\bm{\theta}}(\mathbf{x}|\mathbf{z})$ObservationsCodesReconstructionsEncoderDecoder
Figure 4: Variational autoencoder. An intuitive representation of a
variational autoencoder, combining an encoder
$q_{\bm{\phi}}(\mathbf{z}|\mathbf{x})$ taking observations $\mathbf{x}\in X$
as inputs and producing $\mathbf{z}\in Z$ as outputs, these outputs are then
used by a decoder $p_{\bm{\theta}}(\mathbf{x}|\mathbf{z})$ providing
reconstructions of observations $\mathbf{x}\in X^{\prime}$ with the goal of
making them “as close as possible” to the original observations.

One of the main practical instantiations of disentanglement originates with
models built on the architecture of variational autoencoders (VAEs) [118,
117], where a disentangled representation is defined as one where single
latent units (codes) of a VAE are independently responsive to single factors
generating observations [89].

Following the notation in section 4.1, the goal of a VAE is to learn, given a
dataset $\mathcal{D}$ of observations $X$, a probabilistic generative model
that can approximate factors $S$ using hidden variables (codes) $Z$. To do so,
in a standard VAE architecture we find two neural networks, aptly named
encoder and decoder, see fig. 4. An encoder with weights $\bm{\phi}$
parameterises a distribution $Q^{\bm{\phi}}_{Z\mid X}$, while a decoder
network with weights $\bm{\theta}$ parameterises a distribution
$P^{\bm{\theta}}_{X\mid Z}$. The encoder plays the role of the function $f$ in
definition 4.1, mapping observations to codes, while the decoder is a clever
construction that corresponds to a map

$\displaystyle k:Z\to X^{\prime}$ (16)

where $X^{\prime}$ can be seen as reconstructions of $X$, i.e. observations
that should be “as close as possible” to the original ones according to some
measure, in this case given by the VAE optimisation function provided below.
Notice that, since we only required $f$ to be injective for $g(S)$, $k$ need
not be a function and thus is not well-defined for the simple setup of sets
and functions we adopted in section 4.1. It is however a map that can easily
be defined for more general setups [278]131313One could also just impose
further restrictions on $f$, for example making it surjective so that $k$
becomes its right inverse. Alternatively, one could see $k$ simply as a
_partial multi-valued_ function and work in the category of sets and
relations, or partial multi-valued functions, however this may not be the best
choice for an introductory treatment of formal notions of disentanglement as
the presence of monoidal (and not Cartesian) products complicates the
definition of disentanglement [278].. Combining these, a VAE is trained to
learn weights $(\bm{\phi},\bm{\theta})$ so as to maximise the _evidence-lower
bound_ (ELBO):

$\displaystyle\mathcal{L}(\bm{\phi},\bm{\theta})=\mathbb{E}_{\mathbf{z}\sim
q_{\bm{\phi}}(\mathbf{z}|\mathbf{x})}\Big{[}\log
p_{\bm{\theta}}(\mathbf{x}|\mathbf{z})\Big{]}-D_{KL}[q_{\bm{\phi}}(\mathbf{z}|\mathbf{x})||p(\mathbf{z})]$
(17)

via stochastic gradient descent using (batches of) observations sampled from a
dataset [48, 117]:

$\displaystyle(\bm{\phi},\bm{\theta})\leftarrow(\bm{\phi},\bm{\theta})+\frac{1}{|\mathcal{D}|}\sum_{\mathbf{x}\in\mathcal{D}}\nabla\mathcal{L}(\bm{\phi},\bm{\theta}).$
(19)

Looking at the ELBO more closely, one can notice how the VAE is tasked with
competing objectives. Trying to maximise the first term, usually called the
reconstruction loss, amounts to tweaking the weights of both the encoder and
the decoder, $\bm{\phi}$ and $\bm{\theta}$ respectively, in such a way that
the latent variables (codes $Z$) inferred by the former are more likely given
a certain input (observations $X$) and that the decoder can then use those
variables to reconstruct it (reconstructed observations $X^{\prime}$). In
other words, weight values that lead to bad inferences and/or that do not
afford a good reconstruction of the original inputs will be penalised. The
second term is negative (because the KL divergence, measuring closeness
between distributions, is always greater than or equal to zero) so its
maximisation tries to bring the divergence to zero, i.e. bringing the
posterior distribution close to the prior. This is asking the VAE to map the
inputs to latent variables that are as close as possible to the given prior
distribution. Overall, the VAE should learn hidden variables, or codes,
$\mathbf{z}$ that lead to good reconstructions _and_ whose probability can be
brought close to the one indicated by the prior.

To improve the disentanglement performance of VAEs, one usually adds a
hyperparameter $\beta>1$ to the KL divergence term of the original objective,
leading to what is usually addressed as $\beta$-VAE [89]. Despite the
empirical confirmation of disentanglement, however, the rationale for the
$\beta$ tweak is not entirely clear. The intuition is that by putting more
emphasis on learning statistically independent variables in the latent
representation, one could also get a disentangled representation [26, 89].
Other proposals, based on the VAE, to improve disentanglement have seen the
introduction of different architectures or regularisation terms to the
optimisation objective [116, 31, 177, 174, 49]. However, fundamental
limitations remain because (1) without some supervision or crucial inductive
bias disentangled representation learning cannot be achieved in practice [138,
139, 235, 196, 114] and (2) there is no general agreement on how to
quantitatively measure disentanglement in the units of the latent
representation [190, 47, 252].

Some insights can nonetheless be obtained by looking at the optimisation
objective of the $\beta$-VAE using the information bottleneck principle, a
method to define a trade-off between compression and descriptiveness of a
model given some data [229, 228] with connections to fundamental laws of
thermodynamics [209, 39]. More formally, [6] shows that one can derive a
version of the $\beta$-VAE objective from an unsupervised variation of the
information bottleneck principle. In this version, instead of having a mutual
information term involving a latent representation of codes $Z$ for labels in
a dataset $\mathcal{D}$ to be maximised, we have one capturing how much
information the latent representation encodes about a single given data point
from the dataset, to be minimised because the generative model is supposed to
learn a class of codes compatible with all the data points, not just one.
Using this principle, one can also understand the KL-divergence term
$D_{KL}[q_{\bm{\phi}}(\mathbf{z}|\mathbf{x})||p(\mathbf{z})]$ modulated by
$\beta$ [89] as imposing an information bottleneck limiting the amount of
information that can be encoded from the data into the latent representation,
or in information theoretic terms: as limiting the channel capacity of the
encoder [26].

As explicitly argued in section 4.2 treatments of causal representation
learning can be regarded as attempts at learning a _causal generative model_
that identifies causal factorisations of a _causal generative process_.
Depending on the goals and features of a specific implementation however, we
will see that different architectures recover different kinds of latent
representations. To highlight what we believe to be the biggest difference, we
will define two macro categories, of weak and strong disentanglement, based on
whether a model can recover only factors and their relations to observables or
factors and causal mechanisms involving observables and other factors of a
generative process, respectively.

######  Definition 5.1 (Weak disentanglement).

Weak disentanglement captures the idea of learning a mapping between
observations $X$ and causal codes $Z$, without requiring that the causal
relationships among the factors (potentially involving confounders $C$) are
also recovered (see definition 4.2). This approach appears for example in
[211], where confounders in the generative process cannot be encoded by the
generative model, and there are no causal mechanisms between causal codes or
between confounders and causal codes, thus rendering all assignments $h_{j}$
trivial (i.e. identities) (cf. eq. 11), see fig. 5:

$\displaystyle\mathbf{PA}_{j}\subset\\{\\},\qquad j\in\\{1,\dots,n\\}.$ (20)

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$i$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$Z_{1}$$Z_{2}$$Z_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=Generative
process$C$(structurally exact)(structurally
approximate)$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1}$$Z_{2}$$Z_{3}$$Z_{4}$$X^{\prime}$$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1,2}$$Z_{3}$$Z_{4}$$X^{\prime}$FactorsCodesConfoundersObservationsCausal
mechanisms:from factorsto other factorsLegendfrom confoudersto factorsfrom
factors/codesto observationsfrom codesto other codesGenerative modelWeak
disentanglement

Figure 5: Weak disentanglement. Weak disentanglement as a variant of causal
representation learning concerned only with learning causal codes $Z$ from
high-dimensional inputs (observations $X$) generated by causal factors $S$,
and the identification of causal mechanisms that map causal factors to
observations (no causal mechanisms between causal codes). A generative model
can be considered structurally approximate (with respect to the assumed
generative process) if it fails to recover completely disentangled codes (some
codes remain entangled, e.g. the code $Z_{1,2}$ suggestively standing in for
two factors $S_{1},S_{2}$) and/or if it leaves out causal mechanisms relating
codes to observations. Note that in an actual implementation, like the VAE,
entanglement might manifest as correlations among two or more codes in the
latent representation, all capturing the same factors at the same time. In
other words, the node $Z_{1,2}$ may in practice represent a groups of nodes
with possibly bidirectional influences amongst each other.

In weak disentanglement approaches, causal factors are conceived of as
elementary ingredients that independently influence the observations through
the mapping $g$141414If confounders are not present (or not considered), see
left panel in fig. 5, then the problem of weak disentanglement resembles that
of independent component analysis (ICA) [100], for connections between ICA and
causal representation learning, see [99, 69, 256].. The goal of an agent is
then to learn a causal generative model that reflects this scenario, e.g. by
relying on a measure of robustness with respect to interventions [211] or by
enforcing an independence (orthogonality) constraint on the Jacobian of the
appropriate version of the $g$ map, whose elements quantify the influence of
each causal factor on observations from the dataset $\mathcal{D}$ [69].
Further, if observations provide information only about a subset of the causal
factors (partial observability), thereby introducing potential confounders, a
sparsity constraint imposed on the latent representation can help to recover
the ground-truth causal factors [268]. Despite the absence of any reference to
causality, practical examples of disentangled representation learning
illustrated by means of the $\beta$-VAE earlier achieve something similar
insofar as one prepares a dataset with known causal factors and usually shows
that their method can recover all those factors in the latent representation
(in this case the correspondence is expected to be one-to-one, in contrast
with [211] where it is assumed to be one-to-many).

######  Definition 5.2 (Strong disentanglement).

Strong disentanglement aims to recover not only the causal factors in the
latent representation, but also confounders and the causal relationships
present among them all (i.e. the causal mechanisms). In other words, in strong
disentanglement we consider the more general scenario described in section 4.2
where the causal relations derived from the sets of parents $\mathbf{PA}_{j}$
of each variable are not assumed to be only confounders (see

$\displaystyle\mathbf{PA}_{j}\subset\\{S_{\setminus
j},C\\}\quad(=\lx@cref{creftype~refnum}{eqn:parents}).$ (21)

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$i$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$Z_{1}$$Z_{2}$$Z_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=Generative
process$C$(structurally exact)(structurally
approximate)$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1}$$Z_{2}$$Z_{3}$$Z_{4}$$X^{\prime}$$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1,2}$$Z_{3}$$Z_{4}$$X^{\prime}$FactorsCodesConfoundersObservationsCausal
mechanisms:from factorsto other factorsLegendfrom confoudersto factorsfrom
factors/codesto observationsfrom codesto other codesGenerative modelWeak
disentanglementGenerative
process$S_{1}$$S_{2}$$S_{3}$$S_{4}$$C$$X$StaticDynamicFactorsCodesConfoundersObservationsCausal
mechanisms:from factorsto other
factorsLegend$Z_{1}$$Z_{2}$$Z_{3}$$Z_{4}$$C$$X$$S_{1}^{t}$$S_{2}^{t}$$S_{3}^{t}$$C^{t}$$X^{t}$$S_{1}^{t+1}$$S_{2}^{t+1}$$S_{3}^{t+1}$$C^{t+1}$$X^{t+1}$$Z_{1}^{t}$$Z_{2}^{t}$$Z_{3}^{t}$$C^{t}$$X^{t}$$Z_{1}^{t+1}$$Z_{2}^{t+1}$$Z_{3}^{t+1}$$C^{t+1}$$X^{t+1}$from
confoudersto factorsfrom factors/codesto observationsfrom codesto other
codesGenerative modelStrong disentanglement

Figure 6: Strong disentanglement. Strong disentanglement as a variant of
causal representation learning concerned with learning causal codes and causal
mechanisms of different kinds, to observations and to other causal codes. In a
static setting (left panel), strong disentanglement involves discovering the
causal mechanism among causal factors and, potentially, the structural
assignments of the underlying structural causal model. In a dynamic setting
(right panel), strong disentanglement denotes finding causal structure in the
transition dynamics (e.g. of an MDP), determining causal mechanisms between
causal factors at different time steps. Importantly, we don’t include causal
mechanisms within the same time slice as we assume that there can be no
instantaneous interactions among factors. Note that even when we include
dynamics there is still an underlying SCM that can be partitioned into sets of
causal factors “over time”, i.e. causal factors are process extended over
time, or sets of variables (in this case conveniently given the same name but
a different time index) belonging to different time steps. Intuitively, in the
dynamics setting more causal structure is uncovered because one is considering
causal mechanisms over more causal factors (assigned to different time
slices).

This is implemented for instance by the CausalVAE architecture [269], where a
modified VAE is augmented with a mask layer, in the form of an adjacency
matrix, trained and applied to the latent vector of codes $Z$ to implement the
structural assignments defining the causal mechanism among the codes
(effectively, this step amounts to sampling from a structural causal model),
see fig. 6. The vanilla CausalVAE architecture is however limited to static
data, e.g. images, and does not thus lend itself to treatments of causality
for dynamical processes where causal factors can be causally related in time,
typical for example of reinforcement learning treatments of decision making in
agents. To tackle this, building on the CausalVAE and others similar works,
[270] proposed an extension handling temporal data, see fig. 6. Their VAE-
based architecture is more elaborate, and designed 1) to encode and decode
features of objects (the causal factors) from observations using convolutional
neural networks, CNNs, 2) to integrate temporal information in the latent
representation with gated recurrent units, GRUs, 3) to infer the causal codes
while enforcing constraints on their latent causal dynamics (e.g. that the
noise terms are mutually independent or follow a particular distribution) to
ensure that the causal generative process is identifiable from data (see
section 4.3).

At a high level, differences between the two approaches can be described in
terms of causal understanding of properties vs objects:

  * •

models with weak disentanglement can essentially account for _properties_ , as
in the red bouncing ball example where these would colour, shape and material,
such properties are by definition independent as we wouldn’t want or expect
them to posses causal relationships among them (if they did, they would not be
different properties), while

  * •

models of strong disentanglement aim to describe _objects_ , in the red
bouncing ball example these could be wind, the ground and a player, and unlike
for properties, for objects we would also want a model to capture causal
relations among them, in a way that makes them noticeable and relevant for an
interacting agent.

Overall, in the spectrum of models implementing disentanglement, from weak to
strong, a causal representation that only identifies causal factors and their
relations to observations can be said to be less explicit than one also
recovering the causal mechanisms that exist among them. The latter is in turn
less explicit than a causal representation that identifies the functional form
of the causal mechanism, etc. In other words, a more fine-grained and detailed
causal representation encodes more causal information, thereby describing more
explicitly the causal structure of a certain domain. This is in line with the
informal depiction of explicitness offered by section 3.1, where the notion
was related to the idea of having more causal information to operate in a
certain context, e.g. knowing what means can be manipulated to reach a certain
goal, and will be used hereafter as the foundation of our proposal to relate
work on causality in animal cognition and deep (reinforcement) learning (see
section 6.1).

###  5.2 Trajectories as sources of causal information in RL

In online learning an agent uses the current policy to perform an action in
the (training) environment, which responds with a reward signal, at each time
step. Trajectories of state-action
$\tau\coloneqq[\mathbf{s}_{0},\mathbf{a}_{0},\dots,\mathbf{s}_{T},\mathbf{a}_{T}]$
or observation-action sequences
$\tau\coloneqq[\mathbf{x}_{0},\mathbf{a}_{0},\dots,\mathbf{x}_{T},\mathbf{a}_{T}]$,
often called histories, see section 4.3, can be stored in a replay
buffer151515This procedure is also known as _experience replay_ and was part
and parcel of one of the first breakthroughs of deep reinforcement learning,
see [154]. as sequences of tuples together with their respective rewards at
each time step, e.g, $(\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1})$
or $(\mathbf{x}_{t},\mathbf{a}_{t},r_{t},\mathbf{x}_{t+1})$, where one
sequence corresponds to a trajectory. This information forms an agent’s
experience: the state the agent was in, $\mathbf{s}_{t}$, the action it
performed, $\mathbf{a}_{t}$, the reward it collected, $r_{t}$, and the next
state $\mathbf{s}_{t+1}$ reached from $\mathbf{s}_{t}$ by performing
$\mathbf{a}_{t}$ [135, 154, 153].

Concretely, this experience can be used to obtain an estimate of the expected
cumulative discounted reward in eq. 15 based on trajectories sampled from the
replay buffer, which is regularly updated and acts as a rudimentary database
of memories for the agent. To see this idea in action, we look at a popular
class of approaches represented by actor-critic methods for which the
approximate gradient of the RL objective (see eq. 15), used to update policy
parameters ${\bm{\omega}}$, is computed as follows [212]:

$\displaystyle\nabla_{\bm{\omega}}\hat{J}({\bm{\omega}})=\frac{1}{N}\sum_{n=1}^{N}\sum_{t=1}^{T}\nabla_{{\bm{\omega}}}\log\pi_{{\bm{\omega}}}(\mathbf{a}_{n,t}|\mathbf{s}_{n,t})\biggl{(}\sum_{i=t}^{T}\gamma^{i-t}r(\mathbf{s}_{n,i},\mathbf{a}_{n,i})-V^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i})\biggr{)}$
(22)

where $N$ is the number of trajectories sampled from the replay buffer;
$\pi_{{\bm{\omega}}}$ is the current policy whose parameters ${\bm{\omega}}$
will be updated with the computed gradient;
$\sum_{i=t}^{T}\gamma^{i-t}r(\mathbf{s}_{n,i},\mathbf{a}_{n,i})$ is the
discounted sum of rewards ($G_{t}$, cf. eq. 14) evaluated for the (partial)
sampled trajectory $n$ acquired from time step $t$ until the terminal state
$T$, and $V^{\pi_{\bm{\omega}}}$ is a (state) value function that acts as a
baseline with respect to the discounted return, defined as the expected return
$J({\bm{\omega}})$ from a chosen state sampled from trajectory $n$ at time
$i$, $\mathbf{s}_{n,i}$, if a policy $\pi_{\bm{\omega}}$ is followed from that
point onwards, i.e.

$\displaystyle
V^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i})\coloneqq\mathbb{E}_{\pi_{{\bm{\omega}}}}[G_{t}|S_{t}=\mathbf{s}_{n,i}].$
(23)

In eq. 22, it is common [212, 183, 72, 40] to approximate the discounted
return $\sum_{i=t}^{T}\gamma^{i-t}r(\mathbf{s}_{n,i},\mathbf{a}_{n,i})$, the
sum of rewards obtained from a particular, realised trajectory $n$, with

$\displaystyle
Q^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i},\mathbf{a}_{n,i})\coloneqq\mathbb{E}_{\pi_{{\bm{\omega}}}}[G_{t}|S_{t}=\mathbf{s}_{n,i},A_{t}=\mathbf{a}_{n,i}].$
(24)

This is the $Q$-function (or action value function) for the policy under
consideration, quantifying the value of performing an action in a certain
state, after which the policy is followed until the end of the episode. With
this substitution, one defines the advantage
$\text{Adv}^{\pi_{{\bm{\omega}}}}$, specifying how much better it is to take
action an action $\mathbf{a}_{n,i}$ as opposed to an average action161616This
can be seen by writing down explicitly the relationship between $V$ and $Q$,
in our case  $\displaystyle
V^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i})=\mathbb{E}_{\mathbf{a}\sim\pi_{{\bm{\omega}}}}\left[Q^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i},\mathbf{a}_{n,i})\right]$
(25)

$\displaystyle\text{Adv}^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i},\mathbf{a}_{n,i})\coloneqq
Q^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i},\mathbf{a}_{n,i})-V^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i})$
(26)

that is approximated using a _critic_ neural network trained to estimate only
the state value function from the reward (because the $Q$-function can be
rewritten as the sum of the reward at the current state and the expected state
value function at the next, i.e.
$Q^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i},\mathbf{a}_{n,i})=r(\mathbf{s}_{n,i},\mathbf{a}_{n,i})+\mathbb{E}_{\mathbf{s}_{n,i+1}\sim
P({\mathbf{s}_{n,i+1}|\mathbf{s}_{n,i},\mathbf{a}_{n,i}})}\left[V^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i+1})\right]$)
[212, 183, 72, 40]. The _actor_ part is represented instead by the policy
$\pi_{\bm{\omega}}$, parameterised by a policy neural network that outputs the
most suitable action given a certain state. A learning step then involves
sampling a batch of trajectories from the buffer, using them to evaluate
$\nabla_{\bm{\omega}}\hat{J}({\bm{\omega}})$, i.e. the gradient of the
estimated objective with respect to the policy parameters, and updating these
parameters, ${\bm{\omega}}$, using the gradient to derive a better policy,
conducive to expected cumulative reward maximisation.

Importantly, the learning problem becomes off-policy if the sampled
trajectories used to compute the gradients are not collected by the current
policy (as implemented by the policy network at the current time step) but by
a different one (implemented, for instance, by an old parameters configuration
of the policy network). In practical situations, this is often the case
because the replay buffer does not store only the most recent trajectory,
collected by the current policy, but also past trajectories. Therefore,
sampling a batch of them from the replay buffer turn the learning problem into
an off-policy one. Similarly, in an imitation learning scenario, policy
optimisation is also by default off-policy because the gradients are computed
using sampled trajectories that come from an expert demonstrator and not from
the policy currently followed by the agent (see section 6.2.2 for a more
detailed overview on imitation learning). In an off-policy setting, estimating
the gradient of the objective considered above, $\hat{J}({\bm{\omega}})$, is
problematic because the parameters of the current policy could be updated
based on actions and/or reward information (e.g. value functions) that in
reality characterise a different/earlier actor. In other words, the gradient
information in $\hat{J}({\bm{\omega}})$ might be inaccurate for updating the
current policy. Corrections can be applied to the gradient depending on the
exact RL method considered, going from a variety of importance sampling
techniques (for policy gradient methods, see section 5.3) to the use of
appropriate value functions, i.e. using the actions of the current actor (in
actor critic methods).

###  5.3 Combining different sources of causal information in RL

Following the idea of a buffer containing stored trajectories representing
experience to update a value function or policy, integration can be
interpreted as the ability of an agent to combine different kinds of
experience into its own decision-making process, appropriately weighted based
on context, task demands, origin, resources, etc. In principle, these kinds of
experience can include single-source causal information, e.g. when an agent
uses experience acquired at different points in time or in different
environments/tasks, such as in multi-task or meta-RL (see [15]). However, in
this context we focus on integration of different sources (see section 3.2),
with the goal of highlighting synergistic forms of causal understanding that
truly take advantage of the amalgamation of different causal perspectives.
This allows us thus to delineate an operational, computational account of the
notional idea of integration presented in [208].

In computational terms, the question of how best to integrate and use
information coming from different sources is a foundational aspect of offline
RL, where the replay buffer can store trajectory data collected by any policy
in a variety of virtual environments, more or less realistic, or from real-
world tasks. For example, recent datasets for offline RL tend to include
trajectories from experts (e.g. hand-designed controllers or human
demonstrators), from other RL agents trained online in a certain domain, from
the same agent operating in the same environment but performing slightly
different tasks (multi-task, past experience), from unsupervised (i.e. reward-
free) exploratory policies [54, 73, 271, 281].

The challenge of designing an offline RL algorithm is precisely that of
exploiting the collected data in such a way that the learned policy can be
safely applied to a given environment. This means that the learning algorithm
has to acquire and integrate causal information from various (PO)MDPs (those
in the training set) in such a way that the most appropriate actions for new
downstream tasks/environments can be extrapolated successfully from past
experience and generalised into unfamiliar contexts. A vanilla approach
consists of using importance sampling, originally tailored for dealing with
off-policy data (see previous section) [108, 164, 169, 120, 106, 212].

In this context, importance sampling corresponds to the introduction of
importance weights, ratios (computed over a trajectory) between the current
policy to be optimised, $\pi_{{\bm{\omega}}}$, and a behavioural policy,
$\pi_{{\bm{\beta}}}$, used to collect the transitions sampled from the replay
buffer

$\displaystyle{\bm{w}_{T}^{i}}=\prod_{t=1}^{T}\frac{\pi_{{\bm{\omega}}}(\mathbf{a}_{i,t}|\mathbf{s}_{i,t})}{\pi_{{\bm{\beta}}}(\mathbf{a}_{i,t}|\mathbf{s}_{i,t})}$
(27)

These weights are then used in eq. 22, obtaining:

$\displaystyle\nabla_{\bm{\omega}}\hat{J}({\bm{\omega}})=\frac{1}{N}\sum_{n=1}^{N}{\bm{w}_{T}^{i}}\sum_{t=1}^{T}\nabla_{{\bm{\omega}}}\log\pi_{{\bm{\omega}}}(\mathbf{a}_{n,t}|\mathbf{s}_{n,t})\biggl{(}\sum_{i=t}^{T}\gamma^{i-t}r(\mathbf{s}_{n,i},\mathbf{a}_{n,i})-V^{\pi_{{\bm{\omega}}}}(\mathbf{s}_{n,i})\biggr{)}$
(28)

Weights ${\bm{w}_{T}^{i}}$ adjust gradient information coming from trajectory
collected _off the current policy_ (i.e. using the behavioural policy)
according to how much the two policies are in agreement. Crucially, this
approach works under the assumptions that the marginal state probabilities
with respect to the current and old policy networks’ parameters, say
$p_{{\bm{\omega}}^{\text{\,cur}}}(\mathbf{s})$ and
$p_{{\bm{\omega}}^{\text{\,old}}}(\mathbf{s})$, are sufficiently close to each
other, and that the dynamics of the respective (PO)MDPs are the same. In the
off-policy case, these assumptions hold true because during learning the
replay buffer is regularly updated with newly experienced trajectories and
cleared from the older ones, and because the (PO)MDP dynamics are usually
assumed to be fixed. In the offline setting, this is however not guaranteed,
see [130] for a recent review of some of these and other open challenges, and
the next section for a more complete overview of relevant works in causal RL
on integration, explicitness and sources across causal reinforcement learning
and animal cognition.

##  6 Bringing together causality in natural and artificial agents

Recent work in (deep) reinforcement learning, in the area now called causal
reinforcement learning, can help us shed light on ways to translate algorithms
from machine learning into a more systematic study of causal learning agents.
Using this line of work, we thus review a series of algorithmic
implementations and models from causal RL and place them on a spectrum of
increasingly high disentanglement, providing a comparative analysis with
empirical and conceptual works in the animal cognition literature, see fig. 7.
This will in turn suggest a more concrete connection with the explicitness
dimension of [208]’s framework, paving then the way for an understanding of
causal information from different sources and possible strategies to integrate
them sensibly.

###  6.1 Explicitness of causal representations

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$i$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$Z_{1}$$Z_{2}$$Z_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=Generative
process$C$(structurally exact)(structurally
approximate)$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1}$$Z_{2}$$Z_{3}$$Z_{4}$$X^{\prime}$$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1,2}$$Z_{3}$$Z_{4}$$X^{\prime}$FactorsCodesConfoundersObservationsCausal
mechanisms:from factorsto other factorsLegendfrom confoudersto factorsfrom
factors/codesto observationsfrom codesto other codesGenerative modelWeak
disentanglementGenerative
process$S_{1}$$S_{2}$$S_{3}$$S_{4}$$C$$X$StaticDynamicFactorsCodesConfoundersObservationsCausal
mechanisms:from factorsto other
factorsLegend$Z_{1}$$Z_{2}$$Z_{3}$$Z_{4}$$C$$X$$S_{1}^{t}$$S_{2}^{t}$$S_{3}^{t}$$C^{t}$$X^{t}$$S_{1}^{t+1}$$S_{2}^{t+1}$$S_{3}^{t+1}$$C^{t+1}$$X^{t+1}$$Z_{1}^{t}$$Z_{2}^{t}$$Z_{3}^{t}$$C^{t}$$X^{t}$$Z_{1}^{t+1}$$Z_{2}^{t+1}$$Z_{3}^{t+1}$$C^{t+1}$$X^{t+1}$from
confoudersto factorsfrom factors/codesto observationsfrom codesto other
codesGenerative modelStrong
disentanglement[185][82][50][244][221][167][222][150][275][97][41][253][68][156][189][219,
216, 214, 215][140][188,
186][104][105][217][134][24][173][277][132][123][251][103][224][267][230][3][205][125][21][274][90][79,
78][198][111][254][151][76][212][160][51][154][241, 239][148][243][170,
171][214]Artificial AgentsNatural Agentscausal insightStrong
DisentanglementWeak Disentanglementinstrumental learningappreciation of causal
unobservablesappreciation of causal featuresconfounders(sparse) causal
dynamics“dense” learninginvariant aspectsExplicitnessHighLow

Figure 7: The explicitness spectrum.

####  6.1.1 Weak disentanglement

At the lower end of the explicitness spectrum (see fig. 7), we find agents of
traditional (non-causal) deep RL setups that are successful at solving a
variety of narrow tasks by engaging in forms of dense learning, meaning that
they often appear to learn at least some of the dependencies between their
actions and desired outcomes/rewards [212, 154, 76, 198], and in some cases
they are augmented with more sophisticated forms of planning, curiosity-based
exploration and the ability to achieve a variety of goals in high-dimensional
environments [51, 160, 79, 111, 254, 78, 151]. Nonetheless, the web of
dependencies learned by these agents are usually dense because, as
dependencies that are associative in nature, they include spurious features
and/or relationships. In other words, dense learning in these agents goes hand
in hand with a lack of causal information processing. These agents are in
several ways akin to animal subjects engaging in instrumental learning [243,
241, 239, 214, 171, 170, 148] (see section 2), except for the amount of data
samples used during training.

To have a better understanding of algorithms and empirical results higher in
our explicitness scale, and their relation to weak and strong disentanglement,
it is then useful to look at more specific features of causal representations.
In classical RL, particularly when the problem is presented as a POMDP, a
representation can be understood in two different ways:

  * •

in _model-free RL_ , these are representations of factors (i.e. codes) given
as inputs to a policy, $\pi(\mathbf{a}|\mathbf{s})$, i.e. the state
representations (implemented as vectors of state variables) fed to the policy
network to produce an action, while

  * •

in _model-based RL_ , the term representation points at a model of the
transition dynamics (involving, in turn, the state representations)
$P({\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}})$.

Based on this distinction, we suggest that there are two different ways to
understand what a causal representation involves in causal reinforcement
learning: 1) in causal model-free RL, a causal representation describes a
particular encoding, often a compression, of the observations into latent
states (with a causal flavour) while 2) in causal model-based RL, a causal
representation models both the latent states and the causal dynamics of the
environment. Here we suggest that the first kind of causal representation has
a lower degree of explicitness than the second one, since it fails to capture
the causal dynamics of the environment. We thus view it as a possible example
of _weak disentanglement_. The latter instead is defined precisely in terms of
its ability to map causal dynamics and is thus an example of _strong
disentanglement_ (section 5.1).

In causal model-free RL, in particular for a partially observable setting, an
agent is said to learn an explicit causal representation if it can map high-
dimensional observations to state representations that are disentangled,
uncovering codes that represent some parts of the causal structure (the causal
factors) of the data-generating process, but not including a complete
disentangled representation (see section 5.1). Thus, an agent can be said to
exploit this (partial) causal information if such information facilitates
policy learning or has a positive impact on policy execution when the causal
representation is fed to the policy network.

More specifically, the benefits of this kind of causal representation would
follow from capturing invariant aspects of an environment, represented by the
causal codes, making learning and using a policy more robust to changes in
secondary, irrelevant variables (i.e. improving on generalisation)171717The
connection between invariant aspects of an environment and causality has been
established, for instance in [161, 166, 207]. However, there is ongoing
disagreement on how traditional machine learning methods should be adjusted to
capture invariance in data (with some theoretical guarantees) [10, 33, 109,
175]. Similarly, it remains unclear how the RL framework could be impacted by
such theoretical advancements in the long term. See also works cited in the
main text.. There are many works on invariant representation learning for RL
[277, 19, 204, 210, 276, 144, 145], but it is not entirely clear whether all
these invariant representations qualify as disentangled representations in the
weak sense discussed here.

Conversely, as long as causal codes can be considered invariant aspects of the
environment, disentangled representations in a RL agent are necessarily some
kind of invariant representations. Empirically, the extent to which this is
key to credit assignment, i.e. the ability of an agent to determine which
actions (and/or states) contributed the most to successful performance in a
certain domain, remains however to be proved. At present, there is some
evidence indicating that learning a causal representation, one achieving weak
disentanglement, provides several benefits in RL, e.g. a better exploration of
the state space and more robust learning [3, 125, 90, 267, 224, 205, 274, 21,
230, 103].

At a comparable level of explicitness, in the animal cognition literature we
find subjects that can successfully solve trap-tube tasks, water-displacement
tasks, and/or tasks with similar setups where an attention to objects’
physical and functional properties is essential. Recognising these objects’
invariants and their causal role for the purpose of solving the given task is
suggestive of an appreciation of causal features constituting factors of the
generative process, by disambiguating them, at least in part, from non-causal
ones [134, 219, 216, 105, 140, 186, 188, 215, 214]. For instance, there is
empirical evidence that crows can drop stones into baited, water-filled tubes
according to stones’ width and water levels. Lower water levels and wide tubes
hinder in fact water displacement with the available stones, which is
necessary to reach the reward [140], see section 2.3.

####  6.1.2 Strong disentanglement

In causal model-based reinforcement learning the agent is specifically trained
to learn a _world model_ , a model of the dynamics of the environment, then
used for planning and decision-making (i.e. selecting the next action). In
this context, causality-inspired approaches involve revealing and exploiting
more causal aspects of the (modelled) environmental dynamics, regarded as
crucial for having more capable learning agents that, for instance, do not
fall prey to spurious correlations like agents with less explicit models
might.

To achieve this, we have attempts to handle confounders, hidden common causes
that can have an impact on factors and their state transitions (see section
4.2), by deconfounding the dynamics of a POMDP. Practically, this entails
modelling state-transitions as affected by confounders, whose presence is
either assumed from the start [132, 24, 277, 123], or can emerge from
initially unaccounted parts of the dynamics/predictive model through a process
of decomposition of observations into confounding and relevant state
information [173, 251]. This leads to more explicit models because the effects
of confounding factors are isolated to obtain a more robust understanding of
how events in the environment unfold.

More in detail, we can look at [132] as an example of the first kind (known
confounders), based on object-centric learning (using graph-neural networks)
combined with a model of the transition dynamics that is assumed to be
confounded by time-invariant hidden variables, e.g. the object’s masses,
friction coefficients, etc. The goal of the agent here is to solve a POMDP,
but this requires learning a generative model that is deconfounded, estimating
the confounding variables for each object (using tools from do-calculus [161,
162]), which in turn can be used to generate accurate observation trajectories
had the initial conditions been different (e.g. the objects’ position).
Effectively, this causal world model enables a kind of future counterfactual
planning that starts with the question of what would have happened under
alternative initial conditions, i.e. given an intervention that changes the
starting states. On the other hand, for the second group (unknown confounders)
we can consider [173] where agents with partial models, i.e. models learnt
using past actions and the initial agent’s state as opposed to the full
trajectory of past observations, are shown to be less robust to policy
changes. These partial models are in fact confounded by past observations,
which are not used to train the model but do anyway influence the policy
picked by the agent, but can be adjusted for such confounders by using once
again techniques from do-calculus.

In the animal cognition literature, understanding the influence of potential
confounders can be linked to an appreciation of causal unobservables, such as
in crows adjusting their actions depending on changes in experimental
variables that are not visible to them [217, 104]. In one study, crows were
tested on task consisting of extracting some food from a box, placed on a
table and in front of a curtain. From behind the curtain, a human could
operate a wooden stick that through a hole in the curtain could come close to
the food box, therefore causing trouble for the crows trying to reach the
food. The presence/absence of the human thus confounds whether it is “safe” to
go and retrieve the food from the box (because in principle a stick’s movement
does not create danger, unless it is intentionally used to poke through a
hole, for example by a human experiment), therefore it would be useful to be
able to reason about what is behind the curtain. The evidence reported by
[217] suggests that crows can attribute the movement of the stick to a hidden
agent behind the curtain and act accordingly, e.g. being more cautious when
they do not observe anyone leaving the experiment’s room (because the stick
could move again). Similarly, in a context where a food dispenser is activated
by means of placing objects on it and where an object’s weight confounds the
food release (only heavy enough objects activate the dispenser), crows can
learn to infer the weights of the objects from their movements in a breeze and
pick the appropriate ones to get the food from the dispenser [104]. In both
studies, the animal subjects were able to adjust their behaviour by paying
attention to the reward dynamics, i.e. to whether narrow or wide tube were to
be preferred (according to the respective water level), or to whether light or
heavy objects were activating the dispenser.

Beyond confounding factors, model-based reinforcement learning can be improved
by observing that key causal relationships in the environment, relevant for
solving a particular problem, do not involve all state variables and
transitions among them. That is, the causal dependencies among variables in
the environment that an agent can have an effect on, for the purpose of
reaching a certain goal state, form a causal structure that is _sparse_ , in
the sense that it only captures some deeper facts about a whole class of
problems (or environments) for a particular agent. For example, an agent might
be capable of accessing a certain area of a building by means of a detailed
world model that accurately predicts what happens when a red button located
next to a glass door is pressed, while standing on a floor with hexagonal
tiles. This detailed model keeps track of all possible dependencies among the
colours of buttons, the material of adjacent doors, and the geometric shape of
the floor tiles on which the agent stands (and it might predict with some
confidence that only when a combination of those features is encountered, then
access to a certain area will be granted). The problem is that most of those
dependencies are likely just spurious correlations, hiding the most
fundamental (causal) fact that a button next to a door in general tends to
open that door when pressed.

Thus, instead of learning indiscriminately every conditional dependence
relation (causal or not) among state variables at adjacent time steps, an
agent should strive to learn a causal transition model that identifies the
causal relationships that matters for the class of tasks at hand. Precisely,
focusing on the (sparse) causal dynamics present in a given environment means
to identify the subset of latent state variables, or causal factors $S$, that
are likely to form generalisable causal relationships, exploitable not only
for the task at hand but for similar tasks as well [97, 253, 275, 189, 68, 41,
156]. Therefore, in contrast to modelling dense dynamics, leveraging causal
sparsity is more computationally efficient and can help an agent to avoid
learning spurious correlations over time.

One influential approach hinges upon notions of state abstraction [230] (see
also [131] for more background). A state abstraction can be regarded as a
compact (latent) representation that is invariant to task-irrelevant
information (i.e. only information relevant to a specific problem is encoded),
and is technically defined as a (probabilistic) bisimulation. A bisimulation
is a structure-preserving equivalence relation of states of a (PO)MDP,
$B\subseteq S\times S$, describing equivalence classes of states $S/B$, for
all actions in the action space $A$, with transition dynamics leading from
states with the same reward $R$ to the same equivalence classes of states,
i.e. for $\mathbf{s}_{1},\mathbf{s}_{2}\in B\text{ and }\forall i,a\in S/B,A$
the following conditions apply [274, 275, 276, 230]

$\displaystyle P({i\mid\mathbf{s}_{1},a})$
$\displaystyle=P({i\mid\mathbf{s}_{2},a})$ $\displaystyle r(\mathbf{s}_{1},a)$
$\displaystyle=r(\mathbf{s}_{2},a)$ (29)

see also the notion of “causal states” in [191], roughly probabilistic
bisimulations for stationary stochastic processes, without rewards and
actions.

This line of work can lead to a higher level of explicitness, via strong(er)
disentanglement, as exemplified for instance by [253], showing empirically
that their architecture can learn codes identifying causal factors as well as
the causal mechanisms between them (across times steps). This in particular
reveals causal transition dynamics for each state, i.e. determining whether a
causal relationship between a state at time $t$ and its successor at time
$t+1$ is present or not, and is relevant for an agent solving a particular
problem181818To be precise, in [253] the state abstraction is learned on the
states of a MDP, so the challenge of deriving a disentangled latent
representation (from high-dimensional observations) is bypassed. In other
words, weak disentanglement is taken for granted..

In [208], a high-level of explicitness is connected with adaptive behaviour
supported by a flexible use of causal information, which enables an animal to
re-use acquired knowledge or past behaviours (with the appropriate changes, if
necessary) to reach new goals, the same goals but in a slightly different
context, and/or solve tasks never encountered before. Therefore, at the top
level of the explicitness spectrum, we find causal insight, a term we use to
refer broadly to this type of generalisation abilities, chiefly involving a
deeper realisation of what a problems/scenario entails, based on causal
knowledge. These can be often seen in transfer learning and
innovative/insightful problem solving, e.g. in the floating-peanut task [82,
222, 50, 185, 150, 167, 244, 221], in which the solution to a task is
allegedly reached via an adaptive restructuring of one’s experience [225,
121]. The main idea here is that non-human animal appear to be capable of
cognitive feats thanks to highly explicit causal models of both state
variables and transition dynamics, thought there is some disagreement in the
experimental literature on the extent to which this effectively happens [215,
197, 136].

###  6.2 Sources of causal information

ColourShapeBlueRedRoundSaggingFactorsObservationsCodes$0$$1$$2$$0$$1$$2$$3$GeneratingEncodingBalls$3$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\mathsf{id}_{S}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$=$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$i$$m$$\dots$$S_{1}$$S_{2}$$S_{n}$$Z_{1}$$Z_{2}$$Z_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$m_{1,1}$$m_{2,2}$$m_{n,n}$$\dots$$\dots$$\dots$$Z_{1}$$Z_{2}$$Z_{n}$$S_{1}$$S_{2}$$S_{n}$$i_{1,1}$$i_{2,2}$$i_{n,n}$$\dots$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$$\dots$$S_{1}$$S_{2}$$S_{n}$=Generative
process$C$(structurally exact)(structurally
approximate)$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1}$$Z_{2}$$Z_{3}$$Z_{4}$$X^{\prime}$$S_{1}$$S_{2}$$S_{3}$$S_{4}$$X$$Z_{1,2}$$Z_{3}$$Z_{4}$$X^{\prime}$FactorsCodesConfoundersObservationsCausal
mechanisms:from factorsto other factorsLegendfrom confoudersto factorsfrom
factors/codesto observationsfrom codesto other codesGenerative modelWeak
disentanglementGenerative
process$S_{1}$$S_{2}$$S_{3}$$S_{4}$$C$$X$StaticDynamicFactorsCodesConfoundersObservationsCausal
mechanisms:from factorsto other
factorsLegend$Z_{1}$$Z_{2}$$Z_{3}$$Z_{4}$$C$$X$$S_{1}^{t}$$S_{2}^{t}$$S_{3}^{t}$$C^{t}$$X^{t}$$S_{1}^{t+1}$$S_{2}^{t+1}$$S_{3}^{t+1}$$C^{t+1}$$X^{t+1}$$Z_{1}^{t}$$Z_{2}^{t}$$Z_{3}^{t}$$C^{t}$$X^{t}$$Z_{1}^{t+1}$$Z_{2}^{t+1}$$Z_{3}^{t+1}$$C^{t+1}$$X^{t+1}$from
confoudersto factorsfrom factors/codesto observationsfrom codesto other
codesGenerative modelStrong
disentanglement[185][82][50][244][221][167][222][150][275][97][41][253][68][156][189][219,
216, 214, 215][140][188,
186][104][105][217][134][24][173][277][132][123][251][103][224][267][230][3][205][125][21][274][90][79,
78][198][111][254][151][76][212][160][51][154][241, 239][148][243][170,
171][214]Artificial AgentsNatural Agentscausal insightStrong
DisentanglementWeak Disentanglementinstrumental learningappreciation of causal
unobservablesappreciation of causal featuresconfounders(sparse) causal
dynamics“dense” learninginvariant
aspectsExplicitnessHighLow[171][222][167][170][27][96][223][41][277][176][123][92][143][133][251][122][130][61][282][266][1][55][223][95][94][104][244][221][217][212][101][184][77][153][205][103][76][151][198][24][230][78][7][275][188,
186][148][216][134][241,
239][243][219][215][217][82][105][150][185][140]Artificial AgentsNatural
Agentsegocentric causal informationsocial causal informationnatural causal
informationonline learningonline + offpolicy learningimitation learningoffline
RLinverse RLtrap-tube tasktool-tube taskfloating-reward taskimitation vs.
emulation tasks"ghost"-condition tasksobservational causal tasksSources

Figure 8: Sources of causal information.

####  6.2.1 Egocentric causal information

Successfully learning from online interactions in an environment implies
appreciating, to some extent, the relevance of certain action-outcome
contingencies for reward maximisation or reaching a certain goal. Learning
from online interaction amounts to instrumental learning, which chiefly
involves egocentric causal information (see fig. 8) and has been extensively
studied in animals (see section 3.2), particularly with a variety of tasks
including trap-tube tasks [243, 241, 239, 134, 216, 186, 188, 148] (see
section 2.2), various tool-use tasks [219, 215, 217] and floating-reward tasks
[105, 140, 150, 82, 185] (among several others).

Similarly, most RL agents are designed to be egocentric causal learners, with
varying abilities to latch onto causal information provided by their own
experience, which is ultimately shaped by the provided reward signal. These
abilities come for instance from strategies to boost online learning via
particular training/optimisation techniques (e.g. uncertainty-based or
curiosity-driven exploration) [212, 76, 184, 153, 101, 77, 103, 205], or from
methods to maximise the benefits of online + off-policy learning [24, 198,
230, 275, 151, 7, 78].

With the exception of some of the works cited in section 3.1, most of these
approaches have not adopted a causal terminology. Furthermore, the agents in
question do not process feedback from the environment as causal information,
e.g. by paying attention to key causal relationships with techniques from
causal machine learning. Nonetheless, we refer to them as egocentric causal
learners because they have the ability to process (at least partially) the
consequences of their actions.

####  6.2.2 Social causal information

As already mentioned in section 5.2, artificial agents can be designed to
learn to solve a task through imitation learning, e.g. by relying on
demonstrations of the expected behaviour for the given task. The imitation
learning pipeline can be implemented in various ways, tailored to the specific
domain of application (for a recent review of the main techniques, see [98]).
In RL, the general idea is to allow a learning agent to have access to the
experience of an expert, i.e. trajectories of optimal interactions for solving
the task at hand, which are conveniently pre-processed in the same
representational format of information in the replay buffer, so that they can
guide the learning process towards a policy that achieves similar rewards
[176, 92, 133]. While broadly successful, imitation learning approaches do not
necessarily entail the processing of social causal information in a comparable
way to natural agents. Imitation learning in and by itself does not in fact
prevent a learning agent from simply exploiting correlations between state
variables and actions present in the dataset of expert’s demonstrations to
learn an optimal policy for a certain task. In the presence of distributional
shift, which arises every time trajectory information used for training comes
from a policy different from the one currently used by the learning agent,
agents that learn by imitation, but without causal knowledge, are usually
prone to causal confusion or misidentification (e.g. of what prompted the
expert to act in certain ways) [41]. If a correlation ceases to exist,
performing the same action in response to a certain state could in fact turn
out to be inappropriate in most cases. This knowledge deficit has been
highlighted and studied in depth by a few recent causal RL works, making a
first important step towards artificial agents trained via imitation that are
better equipped to deal with confounders and spurious correlations [41, 277,
123, 143], making them more “aware” of the causal structure of the problem
under consideration.

The emergence of offline RL has marked another milestone in approaches to