# An introduction to reinforcement learning for neuroscience

Kristopher T. Jensen Sainsbury Wellcome Centre, University College London
Computational and Biological Learning Lab, University of Cambridge

<EMAIL_ADDRESS>)

###### Abstract

Reinforcement learning has a rich history in neuroscience, from early work on
dopamine as a reward prediction error signal for temporal difference learning
(Schultz et al., , 1997) to recent work suggesting that dopamine could
implement a form of ‘distributional reinforcement learning’ popularized in
deep learning (Dabney et al., , 2020). Throughout this literature, there has
been a tight link between theoretical advances in reinforcement learning and
neuroscientific experiments and findings. As a result, the theories describing
our experimental data have become increasingly complex and difficult to
navigate. In this review, we cover the basic theory underlying classical work
in reinforcement learning and build up to an introductory overview of methods
used in modern deep reinforcement learning that have found applications in
systems neuroscience. We start with an overview of the reinforcement learning
problem and classical temporal difference algorithms, followed by a discussion
of ‘model-free’ and ‘model-based’ reinforcement learning together with methods
such as DYNA and successor representations that fall in between these two
categories. Throughout these sections, we highlight the close parallels
between the machine learning methods and related work in both experimental and
theoretical neuroscience. We then provide an introduction to deep
reinforcement learning with examples of how these methods have been used to
model different learning phenomena in the systems neuroscience literature,
such as meta-reinforcement learning (Wang et al., , 2018) and distributional
reinforcement learning (Dabney et al., , 2020). Code that implements the
methods discussed in this work and generates the figures is also provided.

##  1 Introduction

Humans and other animals learn from their experiences. Sometimes, this takes
the form of explicit demonstration, as is often the case in our educational
system. However, we often have to infer a good course of action on the basis
of trial and error together with feedback received from the world around us –
sometimes implicit and sometimes explicit. This is well illustrated by the
classical case study of Pavlov’s dogs, who learned to associate a so-called
‘conditioned stimulus’ (CS; e.g. the ringing of a bell) with the availability
of food shortly after. Following a brief period of learning, the dogs would
start to salivate in response to the CS in advance of any food actually being
served. This suggests that the dogs had learned to associate the CS with the
availability of ‘reward’ in the form of food, and that they produced an
appropriate physiological response to take advantage of this food
availability. Importantly, this occurred without any explicit instruction or
description of the sequence of events preceding food being served. Instead,
the dogs learned from their actual experience with their environment and the
presence of a salient, rewarding stimulus.

Such passive stimulus-response predictions are also called ‘Pavlovian
learning’ and have been commonly used in neuroscience to study learning from
external rewards (Niv, , 2009). This forms a specific instantiation of the
concept of ‘reinforcement learning’, which is a general term for settings
where an agent has to learn on the basis of reward signals from the
environment rather than explicit demonstration, as is the case in ‘supervised
learning’. Importantly, the past decades have shown that principles of
reinforcement learning can be used to explain not just behaviour, but also
neural activity in biological circuits (Niv, , 2009). An explicit neural basis
of RL was initially demonstrated in foundational work by Schultz et al.,
(1997), which showed that the firing rates of dopaminergic neurons in the
Ventral Tegmental Area (VTA) reflect the difference between expected and
actual ‘value’ when animals received a juice reward following a CS consisting
of a lever-press in response to a small light turning on. This provided a
potential neural substrate of the classical ‘temporal difference’ learning
algorithm, which has since been expanded to a wealth of evidence for
reinforcement learning in neural dynamics (Niv, , 2009; Dabney et al., ,
2020). However, these classical algorithms are generally restricted to simple
problem settings, while humans and other animals are capable of solving
complex high-dimensional problems involving extended planning and complex
motor control. The field of ‘deep reinforcement learning’ has recently emerged
to tackle such problems in a machine learning setting, which has led to
impressive results across a range of tasks (Mnih et al., , 2013; Schrittwieser
et al., , 2020; Wurman et al., , 2022; Vinyals et al., , 2019). Intriguingly
recent research has demonstrated that these deep RL algorithms also have
parallels in both behaviour and neural dynamics (Botvinick et al., , 2020;
Wang et al., , 2018; Dabney et al., , 2020; Jensen et al., , 2023), suggesting
that the field of neuroscience can continue to learn from advances in
reinforcement learning.

In this review, we provide an overview of the reinforcement learning problem
and popular algorithms, with a particular focus on parallels and uses of these
algorithms in neuroscience. This overview starts from classical tabular TD
learning and Q-learning algorithms, which have guided neuroscientific research
for decades. We then consider the important distinction between model-based
and model-free reinforcement learning, as well as methods that fall somewhere
in the gray area between these extremes, and discuss their neural correlates.
Finally, we generalize the tabular methods to the non-linear function
approximation setting and the resulting deep RL methods, which have
revolutionized machine learning in recent years. We do this with a focus on
methods that have had a strong influence on neuroscience to give the reader a
better idea of the mathematical and computational background of recent
neuroscientific findings. These include the ‘meta-reinforcement learning’
model of PFC by Wang et al.,  (2018) and the ‘distributional reinforcement
learning’ model of VTA dopaminergic neurons by Dabney et al.,  (2020) in
particular. We hope this introduction to RL for neuroscience will be useful
both for those who are interested in the theory underlying reinforcement
learning in neuroscience and for those who want an overview of how the
neuroscientific literature builds on principles from reinforcement learning
theory. Throughout the paper, the focus will be on an intuitive understanding
of the relevant RL methods, and explicit derivations are included only where
we consider them conducive to such understanding. We refer to Sutton and
Barto,  (2018) for a more in-depth treatment of the underlying theory.

##  2 Problem setting

Here we provide a short introduction to the reinforcement learning problem in
a discrete state and action space with a finite time horizon. For a more
general treatment, we refer to Sutton and Barto,  (2018). In the discrete
problem setting, the environment consists of states $s\in\mathcal{S}$, and the
agent can take actions $a\in\mathcal{A}$. The environment is characterized by
transition and reward probabilities $p(s_{t+1},r_{t}|s_{t},a_{t})$, where
$r_{t}$ is the reward at time $t$. We will use $r(s_{t},a_{t})$ to denote
either the reward when it depends deterministically on the state and action,
or its expectation otherwise. We will further make the _Markov assumption_
that the next state only depends on the current state and action,
$p(s_{t+1},r_{t}|s_{t},a_{t},s_{t-1},a_{t-1},...,s_{0},a_{0})=p(s_{t+1},r_{t}|s_{t},a_{t})$.

Figure 1:  The reinforcement learning problem and cliffworld environment. (A)
Illustration of the reinforcement learning problem. An agent (the chick) has
to interact with the world to maximize its lifetime reward. This involves a
balance between exploring potentially interesting states (e.g. learning to
fly) while also exploiting states known to yield high reward (e.g. sitting in
the nest and eating food brought back by its parents). At any given point in
time, the chick is in some state $s_{t}$ from which it can take an action
$a_{t}$, with the probability of different actions determined by the ‘policy’
$\pi(a|s_{t})$, which is controlled by the agent. $a_{t}$ then leads to a
change in the environment according the non-controllable environment dynamics
$s_{t+1},a_{t}\sim p(s_{t+1},r_{t}|s_{t},a_{t})$. Here, $r_{t}$ is the
empirical ‘reward’ received by the agent, and its objective is to collect as
much cumulative reward as possible. Often, reinforcement learning problems are
divided into ‘episodes’, with the agent learning over the course of multiple
repeated exposures to the environment. This could for example consist of the
chick learning over the course of multiple days when to wake up in
anticipation of its parents bringing back food. (B) The ‘cliffworld’
environment, which will be used to demonstrate the performance and behaviour
of a range of reinforcement learning algorithms in this work. The agent starts
in the lower left corner (location [0, 0]), and the episode finishes when it
encounters either the ‘cliff’ (dark blue) or the goal (yellow; location
[9,0]). If the agent walks off the cliff, it receives a reward of -100. If it
finds the goal, it receives a reward of +50. In any other state, it receives a
reward of -1. Such negative rewards for ‘neutral’ actions are commonly used to
encourage the agent to achieve its goal as fast as possible. The arrows
indicate the ‘optimal’ policy, which takes the agent to the goal via the
shortest possible route that avoids the cliffs.

We can now define a _trajectory_ $\tau=\\{s_{t},a_{t},r_{t}\\}_{t=0}^{T}$,
where

$p(\tau)=p(s_{0})\prod_{t=0}^{T}p(s_{t+1},r_{t}|s_{t},a_{t})p(a_{t}|s_{t}).$
(1)

$p(a_{t}|s_{t})$ is the probability of taking action $a_{t}$ in state $s_{t}$,
which is usually controlled by the agent and denoted a _policy_
$\pi(a_{t}|s_{t})$ (Figure 1A). The objective of the agent is to maximize the
expected total discounted reward

$\displaystyle J(\pi)$ $\displaystyle=\mathbb{E}_{\tau\sim
p(\tau)}\left[R_{\tau}\right]=\mathbb{E}_{\tau\sim
p(\tau)}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}|\tau\right],$ (2)

where $R_{\tau}:=\sum_{t=0}^{T}\gamma^{t}r_{t}|\tau$ and we have written
$J(\pi)$ since the policy uniquely specifies $J$ in a stationary environment.
In Equation 2, $\gamma$ is a ‘discount factor’, which stipulates that we
should care more about immediate rewards than rewards far in the future. We
can provide three interpretations for this discount factor. One is that agents
intrinsically care more about immediate reward than distant reward. A second
is that there is a finite probability $(1-\gamma)$ of the current ‘episode’ or
environment terminating or changing at each timestep, in which case we should
weight putative future reward by the probability that we are still engaged in
the task at that time. The third view is that $\gamma$ simply provides a tool
for reducing the variance of our learning methods, especially in temporally
extended tasks. This third view is most compatible with the fact that
_evaluation_ of our RL agents at the end of training is generally done without
any discounting.

Since $J(\pi$) depends on the policy of the agent, it is possible to search in
the space of policies for one that maximizes $J$, which is the topic of
reinforcement learning. It is often assumed that the experience $\\{\tau\\}$
is generated by the agent acting according to its policy, and the resulting
experience is then used to update the policy in a way that increases $J(\pi)$.
However, ‘off-policy’ and ‘offline’ reinforcement learning methods also exist,
where the agent learns on the basis of experience generated by a policy
different from $\pi$ (Levine et al., , 2020; Section 9.3).

##  3 Temporal difference learning

In classical temporal difference learning, we want to learn a _value function_
for a given state $s$ and policy $\pi$, which specifies the expected future
reward when following $\pi$ starting from $s$:

$V_{\pi}(s)=\mathbb{E}_{a\sim\pi}\left[\sum_{t^{\prime}\geq
t}\gamma^{t^{\prime}-t}r_{t^{\prime}}|s_{t}=s\right].$ (3)

Here, $\mathbb{E}_{a\sim\pi}[\cdot]$ indicates an expectation taken over
trajectories $\tau$ resulting from the agent following the policy $\pi$. For
the true value function, we can expand this as a self-consistency equation

$\displaystyle V_{\pi}(s)$
$\displaystyle=r(s)+\sum_{s^{\prime}}p_{\pi}(s_{t+1}=s^{\prime}|s_{t}=s)\mathbb{E}_{a\sim\pi}\left[\sum_{t^{\prime}=t+1}\gamma^{t^{\prime}-t}r_{t^{\prime}}|s_{t+1}=s^{\prime}\right]$
(4)
$\displaystyle=r(s)+\gamma\sum_{s^{\prime}}p_{\pi}(s_{t+1}=s^{\prime}|s_{t}=s)V_{\pi}(s^{\prime}),$
(5)

where
$p_{\pi}(s_{t+1}=s^{\prime}|s_{t}=s)=\sum_{a}\pi(a|s)p(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$
is the probability of transitioning from $s$ to $s^{\prime}$ under $\pi$.
Importantly, Equation 5 would not hold if $V_{\pi}(s)$ was not the true value
function (Sutton and Barto, , 2018). When learning an approximate value
function $V(s_{t})$, we can therefore use this bootstrapped self-consistency
relation as an objective function:

$\mathcal{L}\propto\left(V(s_{t})-(r(s_{t})+\gamma\mathbb{E}_{\pi}V(s_{t+1}))\right)^{2},$
(6)

Gradient descent w.r.t $V(s_{t})$ gives us an update rule

$\displaystyle\Delta V(s_{t})$
$\displaystyle\propto-\frac{\partial\mathcal{L}}{\partial V(s_{t})}$ (7)
$\displaystyle\propto-V(s_{t})+r(s_{t})+\gamma\mathbb{E}_{\pi}\left[V(s_{t+1})\right]$
(8) $\displaystyle\approx-V(s_{t})+r_{t}+\gamma V(s_{t+1}).$ (9)

The last line approximates the expectation over next states with a single
sample corresponding to the state and reward actually experienced. As more and
more experience is collected and many small gradient steps are taken according
Equation 9, these single-sample estimates will average out to the expectation
in Equation 8 (Figure 2A). Variants of this algorithm can also learn about
multiple past states at once using the notion of _eligibility traces_ (Sutton
and Barto, , 2018). However, Equation 9 is the canonical temporal difference
learning rule, and it leads to learning dynamics where the temporal difference
error
$\delta:=-V(s_{t})+r(s_{t})+\gamma\mathbb{E}_{\pi}\left[V(s_{t+1})\right]$ is
progressively propagated from the rewarding state to prior states that predict
the upcoming reward. This forms the basis of the ‘temporal difference’ signal
proposed to be represented in dopaminergic VTA neurons (Schultz et al., ,
1997). At the behavioural level, such theories of dopamine as a reinforcing
signal are consistent with experiments showing that stimulation of dopamine
neurons can be a strong driver of learning in instrumental conditioning tasks,
where animals have to take explicit actions to achieve rewarding outcomes
(Olds and Milner, , 1954; Tsai et al., , 2009). A wealth of other research
also suggests a close correspondence between midbrain dopaminergic neural
activity and temporal difference learning, and we refer to Niv,  (2009) and
others for a more comprehensive overview.

Figure 2:  Temporal difference learning. (A) Value functions aquired through
temporal difference learning (Equation 9) while acting according to either a
random (top) or an optimal (bottom) policy. These simulations were performed
with a random start state in the cliffworld environment to ensure full
coverage of the space. Dark blue indicates negative expected reward (-100) and
yellow indicates positive expected reward (+50). For these simulations, we
used a learning rate of $\delta=0.05$ and no temporal discounting
($\gamma=1$). Under the random policy, states near the cliff have low value
even though they are close to the goal, since the agent will often fall off
the cliff from there. Under the optimal policy, all states have high expected
reward, since the agent will always find the goal. States nearer the goal have
higher value than those further away, although this is hard to distinguish on
this color scale. (B) Empirical reward as a function of episode number for a
TD-learning agent that acts according to Equation 11 while updating its value
estimates according to Equation 9. For this agent, action selection assumes
access to a ‘one-step’ world model in order to evaluate the consequence of
each putative action. We see that the agent gradually converges to an optimal
policy. Parameters for the agent are as in (A). (C) Value function learned by
a greedy TD agent as in (B), plotted either early (top) or late (bottom) in
training. Early in training, the agent has learned that the cliff is bad but
doesn’t know where the goal is or how to get there. Late in training, the
agent has learned a value function that locally resembles the optimal value
function from (A), while it has not learned the value of distant states that
are rarely or never visited. This is a potential shortcoming of ‘greedy’
agents that can easily converge to a sub-optimal local maximum in more
complicated environments. For this analysis, we used a high learning rate of
$\delta=0.5$ to make the early TD updates larger and therefore more visible.

In Equation 9, we simply learned a function for predicting reward without
using this as the basis of taking good actions. Indeed, in classical Pavlovian
conditioning, there are only states which happen in a predetermined order with
no intervention or control on the part of the agent. However, having learned a
value function, it is simple to use it for action selection if we can estimate
$p(s_{t+1}|s_{t},a_{t})$. In this case, the expected reward associated with
action $a$ can be written as

$\mathbb{E}\left[\sum_{t^{\prime}=t}\gamma^{t^{\prime}-t}r_{t^{\prime}}|a_{t}=a\right]=r(a_{t},s_{t})+\gamma\sum_{s_{t+1}}p(s_{t+1}|s_{t},a_{t})V(s_{t+1}).$
(10)

We can then perform action selection by choosing the action with the highest
expected reward,

$a^{*}(s)=\text{argmax}_{a}\left(r(s,a)+\gamma\sum_{s_{t+1}}p(s_{t+1}|s_{t},a_{t})V(s_{t+1})\right).$
(11)

Updating the value function according to LABEL:{eq:TD-learning} while acting
in the environment according to LABEL:{eq:value_action_selection} leads to an
agent that gradually learns to take better actions as it learns a better value
function (Figure 2B-C).

##  4 Q-learning

Figure 3:  Q-learning. (A) Empirical reward as a function of episode number
for Q-learners with different levels of stochasticity in their policy
($\epsilon\in\\{0,0.1,0.2\\}$; legend). For these simulations, we used a
learning rate of $\delta=0.05$ for all agents and no temporal discounting
($\gamma=1$). The agent with $\epsilon=0$ converges to an optimal policy,
similar to the TD agent in Figure 2A. However, convergence is in this case
slower despite using the same learning rate, because the Q-learner has to
learn about each action independently, while the TD agent used its one-step
world model to aggregate learning across actions reaching the same state. In
this cliffworld environment, increasing epsilon leads to worse performance
since it increases the probability of falling off the cliff. Additionally,
there is no risk of getting stuck in a local minimum since there is only one
rewarding state, which decreases the value of exploration. Lines and shading
indicate mean and standard error across 10 simulations. (B) As in (A), now for
a non-cliffworld grid environment with two goals: one with a reward of +20 at
location (0, 4), and one with a reward of +50 at location (5,0). In this case,
having non-zero epsilon can increase the probability of discovering the ‘high
reward’ goal rather than getting stuck with a locally optimal policy of moving
to the ‘low reward’ goal. In these simulations, we used a learning rate of
$\delta=1$, since this effect is less robust with lower learning rates that
lead to more exploration of the environment across all agents. (C) Policy
learned by a Q-learning (top) or a SARSA (bottom) agent with $\epsilon=0.2$.
Colours indicate the maximum value of any action in a state from blue (-100)
to yellow (+50), and arrows indicate which action has the highest value. The
Q-learning agent learns to move right above the cliff, because this is the
optimal thing to do under the assumption that subsequent actions are also
optimal. This is because it is an ‘off-policy’ algorithm that does not take
into account the actual policy of the agent. In contrast, the SARSA agent
learns to move a ‘safe distance’ away from the cliff, since it is an ‘on-
policy’ algorithm that takes into account the finite probability of the agent
choosing to move off the cliff from upcoming states. Q-learning agents are
also frequently trained using a stochastic $\epsilon$-greedy policy and then
evaluated with the greedy policy corresponding to $\epsilon=0$, or they can be
trained while ‘annealing’ $\epsilon$ from some finite value to $0$ over many
episodes.

An alternative approach for learning to select good actions (the so-called
‘control problem’) when we don’t know the transition structure of the world is
to learn the value of taking action $a$ in state $s$ instead of only learning
the value of the state. This gives rise to ‘Q-learning’ methods, where we
learn the _state-action values_

$Q_{\pi}(s,a)=\mathbb{E}_{a\sim\pi}\left[\sum_{t^{\prime}=t}\gamma^{t^{\prime}-t}r_{t^{\prime}}|s_{t}=s,a_{t}=a\right].$
(12)

Given these state-action values, it is simple to perform action selection by
taking the action with the highest expected reward in a given state,

$a^{*}(s)=\text{argmax}_{a}Q(s,a).$ (13)

To learn the $Q$-values necessary for such action selection, we start by
expanding Equation 12,

$Q_{\pi}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\sum_{s_{t+1}}p(s_{t+1}|s_{t},a_{t})\sum_{a_{t+1}}\pi(a_{t+1}|s_{t+1})Q_{\pi}(s_{t+1},a_{t+1}).$
(14)

For the greedy policy $\pi^{g}(a|s):=\delta(a,a^{*}(s))$, this gives rise to a
self-consistency expression for the state-action values:

$Q_{\pi^{g}}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim
p(s_{t+1}|s_{t},a_{t})}\left[\text{max}_{a_{t+1}}Q_{\pi^{g}}(s_{t+1},a_{t+1})\right].$
(15)

Importantly, this self-consistency expression only holds when the Q-values
have converged to the true expected rewards, and the associated greedy policy
is therefore optimal (Sutton and Barto, , 2018). We can now use Equation 15 as
a loss function by defining

$\mathcal{L}\propto\left(Q(s_{t},a_{t})-(r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim
p(s_{t+1}|s_{t},a_{t})}\left[\text{max}_{a_{t+1}}Q(s_{t+1},a_{t+1})\right])\right)^{2},$
(16)

Gradient descent w.r.t $Q(s_{t},a_{t})$ gives us an update rule

$\displaystyle\Delta Q(s_{t},a_{t})$
$\displaystyle\propto-\frac{\partial\mathcal{L}}{\partial Q(s_{t},a_{t})}$
(17)
$\displaystyle\propto-Q(s_{t},a_{t})+r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim
p(s_{t+1}|s_{t},a_{t})}\left[\text{max}_{a_{t+1}}Q(s_{t+1},a_{t+1})\right]$
(18)
$\displaystyle\approx-Q(s_{t},a_{t})+r_{t}+\gamma\text{max}_{a_{t+1}}Q(s_{t+1},a_{t+1}).$
(19)

This is the so-called Q-learning update rule (Watkins, , 1989; Figure 3A),
where we have again estimated the expectation over next states with the single
sample actually seen by the agent.

Q-learning is guaranteed to converge to the optimal policy in the limit of
infinitesimal learning rates and infinite sampling of the state-action space
(Watkins and Dayan, , 1992; Sutton and Barto, , 2018). However, following the
greedy policy $a^{*}(s)=\text{argmax}_{a}Q(s,a)$ before convergence of the Q
values can lead to undersampling of the state-action space and poor
performance. It is therefore common to either use an ‘$\epsilon$-greedy’
policy, $\pi(a|s)=\epsilon/|\mathcal{A}|+(1-\epsilon)\delta(a,a^{*}(s))$, or a
softmax-policy, $\pi(a|s)\propto\text{exp}(\beta Q(a,s))$, to collect the
experience used to update the Q values (Figure 3B). These approaches give rise
to so-called ‘off-policy’ algorithms, since the policy used in the learning
update (the greedy policy) is different from the policy used for action
selection (the stochastic policy). An on-policy version of Q-learning known as
‘SARSA’ (state-action-reward-state-action) is also commonly used, where the
Q-learning update uses the Q-value corresponding to the action $a_{t+1}$
sampled at the next timestep instead of the greedy action (Figure 3C):

$\displaystyle\Delta Q(s_{t},a_{t})\propto-Q(s_{t},a_{t})+r_{t}+\gamma
Q(s_{t+1},a_{t+1}).$ (20)

This will converge to the true Q values for a given policy $\pi$, similar to
how the TD learning rule in Equation 9 is guaranteed to converge to the true
value function for a given policy, again under assumptions of infinite
sampling of the space. In ‘instrumental conditioning’ settings, where animals
have to choose between multiple actions with different values, studies have
also found evidence for midbrain dopamine neurons encoding either quantities
resembling the prediction error used for Q learning (Roesch et al., , 2007;
Niv, , 2009) or SARSA (Morris et al., , 2006; Niv, , 2009).

##  5 Model-free and model-based reinforcement learning

In the previous section, we developed a so-called ‘model-free’ reinforcement
learning algorithm. This involves learning a stimulus-response pattern that
says ‘when in state $s$, take action $a$’. Such algorithms do not require much
computation at decision time, where there is no need to plan far into the
future. However, it can require a lot of experience with the environment to
learn these model-free policies, and they can be very inflexible in changing
environments.

On the other hand, ‘model-based’ reinforcement learning uses a model of the
world to simulate the consequences of different actions at decision time. This
can be much more data efficient, since learning a world model is often simpler
than learning a full policy (Figure 4A). However, model-based decision making
can also be computationally intensive at decision-time in complicated state
spaces (Figure 4B). Such models have recently exhibited impressive performance
across a range of RL settings with large state spaces, including atari, chess,
shogi, and Go (Silver et al., , 2018; Schrittwieser et al., , 2020; Deisenroth
and Rasmussen, , 2011).

In model-based RL, experience is used to learn a transition-and-reward
function $p(s_{t+1},r_{t}|s_{t},a_{t})$ from past experience. Once this model
has been learned, it can be used for planning at decision time. This can be
done for example by expanding the Q-value relation from Equation 15:

$\displaystyle Q(s_{t},a_{t})$
$\displaystyle=r_{t}+\gamma\mathbb{E}_{p(s_{t+1}|s_{t},a_{t})}\left[\text{argmax}_{a_{t+1}}Q(s_{t+1},a_{t+1})\right]$
(21)
$\displaystyle=r_{t}+\gamma\mathbb{E}_{p(s_{t+1}|s_{t},a_{t})}\left[\text{argmax}_{a_{t+1}}\left[r(a_{t+1},s_{t+1})+\gamma\mathbb{E}_{p(s_{t+2}|s_{t+1},a_{t+1})}\left[\text{argmax}_{a_{t+2}}Q(s_{t+2},a_{t+2})\right]\right]\right]$
(22) $\displaystyle=\ldots$ (23)

If the environment is determinstic, $p(s_{t+1}|s_{t},a_{t})$ is a delta
function, and otherwise the next-state expectations may need to be
approximated with multiple samples. Unfortunately, the optimization over all
possible action sequences in Equation 21 is in general an exponentially large
search problem in the planning depth, which makes it infeasible for any
reasonably sized problem setting. It is therefore common to either use ‘depth-
first search’ with limited breadth, or ‘breadth-first search’ with limited
depth. In breadth-first search, we consider all possible actions at each level
of the search tree but terminate the search at a finite depth, instead using
cached ‘model-free’ state-values to estimate the reward-to-go from the
termination of the search. This kind of ‘plan-until-habit’ algorithm has also
been proposed as a model of human behaviour (Keramati et al., , 2016). In
depth-first search, we instead sample a series of paths from $s_{t}$ all the
way to termination (or some upper bound), using a heuristic to prioritize
actions expected to be good, and then pick a path associated with high
expected reward. This is what is implemented in most Monte Carlo tree search
(MCTS) methods, such as AlphaZero (Silver et al., , 2018), which uses a
hardcoded transition function, and MuZero (Schrittwieser et al., , 2020),
which uses a learned transition function. In both of these approaches, neural
networks are used (i) to estimate the ‘reward-to-go’ if the maximum search
depth is reached before termination, and (ii) to select which actions to
evaluate during the search process. Methods for learning such value functions
and policies in neural networks are covered in more detail in Section 7.

Figure 4:  Model-based reinforcement learning. (A) Learning curves for model-
based (MB) and model-free (MF) RL agents. In this case, the MB agent used a
depth-first search to compute an optimal path at each decision point. If a
shorter path to a given state had already been discovered, the search was not
continued. The reward and transition functions of the agent were initialized
to zero and the empty set respectively, and they were gradually populated as
the agent explored the environment across episodes. The MF agent was a simple
Q-learning agent with $\epsilon=0$ and $\delta=1$. The MB agent requires
substantially less experience than the MF agent for a given level of
performance. (B) Wallclock time needed to run 100 episodes of cliffworld with
either the MB or MF agents from (A), as a function of the length of the
environment. Although (A) shows that the MB agent requires substantially less
experience to learn a good policy, the wallclock time per episode was much
larger than that required by the MF agent. This illustrates an important
balance between model-based and model-free reinforcement learning, where MF
methods usually require more experience but MB methods require more compute
for a given level of experience. In domains such as robotics, where collecting
data is very time consuming, MB methods can still be a more efficient way to
reach a high level of performance. (C) Learning curve for an agent using the
successor representation (SR) together with learning curves for the model-
based agent in (A) and the greedy TD-agent from Figure 2. At episode 40
(vertical black line), the goal was moved from location (9, 0) to location (0,
4), and location (9, 0) was instead given a reward of -5. The MB and SR agents
had their reward functions updated to reflect this change and were immediable
able to adapt their policies, while the TD agent had no such mechanism for
robustness to changing reward functions. It is worth noting that an SR agent
cannot always adapt to a new reward function if the newly rewarded states have
low probablity under the old policy, since the successor matrix itself is a
function of the policy. Reward curves were convolved with a Gaussian kernel
($\sigma=3$ episodes), which is why performance appears to decrease slightly
before episode 40 for the TD agent. In this simulation, the TD and SR agents
were assumed to have access to a 1-step world model at initialization, while
the MB agent had to learn the transition structure from experience. This is
why the MB agent does not exhibit faster initial learning than the SR and TD
agents. (D) Learning curve for a standard Q-learning agent (blue) or DYNA
agents that perform different numbers of Q value updates for each step of
physical action (legend). In this case, the DYNA updates simply used cached
experience rather than data from a learned world model. We see that DYNA
agents are able to make better use of limited experience, although this comes
at the cost of increased compute (proportional to the number of updates).

A shortcoming of these MCTS-based approaches from a neuroscientific
perspective is that they often perform a large number of node expansions at
each decision point. For example, MuZero and AlphaZero perform 800 rollouts to
a leaf node for each decision that they make. While this can be done very
quickly on a computer, it seems unrealistic as a model of human decision
making. In some settings, it is possible to use more efficient algorithms once
a model has been learned, such as Djikstra’s algorithm or $A^{*}$-search in
shortest-path problems (Hart et al., , 1968). However, it is in general
necessary to somehow trade off the temporal opportunity cost of planning with
the improvement in policy and expected reward (Botvinick and Cohen, , 2014;
Agrawal et al., , 2022). This has been a popular research area in cognitive
science, where a wealth of literature on ‘resource-rational’ decision making
has emerged in recent years (Griffiths et al., , 2019; Callaway et al., ,
2022). However, this literature has often considered optimal behaviour in
simple tasks with less focus on the learning process and neural mechanisms
that might implement the necessary computations. Bridging this gap, recent
work has suggested that frontal cortex might initially store a ‘model-free’
policy in its network state and gradually update this with model-based
information from the hippocampal formation in cases where the policy
improvement is expected to make up for the temporal opportunity cost (Jensen
et al., , 2023). In this work, the authors found substantial policy
improvements from only a few rollouts (Vul et al., , 2014), and the ‘reaction
times’ of the agent exhibited similar task-related structure to human
behaviour.

While several model-based and model-free reinforcement learning methods have
thus been developed and used to model animal learning and behaviour, it
remains an open question when and whether different approaches are used. A
different line of research has explicitly investigated the trade-offs between
model-based and model-free RL in biological agents (Daw et al., , 2005; Geerts
et al., , 2020; Lengyel and Dayan, , 2007), where the choice between the two
approaches is thought to be guided by some notion of optimality on the basis
of available resources and uncertainty about the environment. A popular
paradigm for such studies attempting to distinguish between model-free and
model-based behaviour is the so-called ‘two-step’ task developed by Daw and
colleagues (Daw et al., , 2011; Momennejad et al., , 2017; Wang et al., ,
2018; although note Akam et al., , 2015).

It has generally been accepted that humans can use both model-free and model-
based decision making, with the dorsolateral striatum being particularly
important for model-free reinforcement learning (Yin et al., , 2004, 2005),
and the dorsomedial striatum, prefrontal cortex, and hippocampal formation
being important for model-based decision making (Vikbladh et al., , 2019;
Geerts et al., , 2020; Miller et al., , 2017; Niv, , 2009; Killcross and
Coutureau, , 2003). This has interesting parallels to recent work in the motor
learning literature, where the basal ganglia were found to be sufficient for
‘habitual’ motor sequences even in the absence of motor cortex, while motor
cortex was necessary for more flexible motor behaviours that are likely to
require a high-level ‘schema’ of the task structure (Mizes et al., 2023b, ;
Mizes et al., 2023a, ).

##  6 The successor representation

As we saw in the previous section, an important distinction can be made
between model-free reinforcement learning methods, which cache stimulus-
response mappings based on prior experience, and model-based reinforcement
learning methods, which compute a policy by simulating possible futures using
a world model at decision-time. However, Dayan,  (1993) introduced a new
formulation of the reinforcement learning reward function known as the
‘successor representation’ (SR), which combines some of the features of model-
free and model-based reinforcement learning. In particular, the SR allows for
flexible adaptation to changing reward functions without having to perform
expensive simulations at decision time.

This formulation rewrites the expected reward starting from state $s$ as

$\displaystyle V_{\pi}(s)$
$\displaystyle=\mathbb{E}_{\pi}\left[\sum_{t=0}\gamma^{t}r_{t}|s_{0}=s\right]$
(24)
$\displaystyle=\sum_{t=0}\gamma^{t}\sum_{s^{\prime}}p_{\pi}(s_{t}=s^{\prime}|s_{0}=s)r(s^{\prime})$
(25)
$\displaystyle=\sum_{s^{\prime}}r(s^{\prime})\sum_{t}\gamma^{t}p_{\pi}(s_{t}=s^{\prime}|s_{0}=s)$
(26) $\displaystyle=\bm{r}^{T}\bm{m}^{\pi}_{s}.$ (27)

Here, $\bm{r}$ is a vector of the reward associated with each state, and
$\bm{m}^{\pi}_{s}$ is a vector of the expected discounted future occupancy of
state $s^{\prime}$ given that the agent starts in state $s$ and follows the
policy $\pi$:

$M^{\pi}_{ss^{\prime}}=\sum_{t=0}\gamma^{t}p_{\pi}(s_{t}=s^{\prime}|s_{0}=s).$
(28)

The full matrix $\bm{M}^{\pi}$, constructed from stacking the
$\bm{m}^{\pi}_{s}$ corresponding to all states $s$, is denoted the ‘successor
matrix’, and constructing it allows us to write down a vector of expected
rewards from any state $s$ as

$\bm{v}_{\pi}=\bm{M}^{\pi}\bm{r}.$ (29)

Here, we have retained the superscript $\pi$ to indicate that the successor
matrix depends on the policy of the agent, which affects the expected
occupancy of different states. Since this allows us to compute the value of
each state, we can perform action selection using Equation 11.

The flexibility of the successor representation arises when the reward
structure of the environment changes, $\bm{r}\rightarrow\bm{r}^{\prime}$. We
can now compute the expected reward associated with each state under the new
reward function and old policy,

$\bm{v}_{\pi}^{\prime}=\bm{M}^{\pi}\bm{r}^{\prime}.$ (30)

Of course this new value function will lead to us eventually changing our
policy and having to update $\bm{M}$, but it provides a better starting point
than using the wrong policy _and_ value function as in standard temporal-
difference learning (Figure 4C). Additionally, it has the appealing feature
that the successor matrix can be learned by simple temporal-difference
learning when transitioning from $s_{t}$, analogous to Equation 9:

$\Delta M_{s_{t}s^{\prime}}\propto-
M_{s_{t}s^{\prime}}+\delta(s_{t},s^{\prime})+\gamma M_{s_{t+1}s^{\prime}}.$
(31)

Here, ${s^{\prime}}$ is the set of all other states, $s_{t+1}$ is the next
state actually observed, and the learning rule says that transitioning from
$s_{t}$ to $s_{t+1}$ means that (i) we have just been in state $s_{t}$, and
(ii) we should increase the expected occupancy of all states commonly reached
from $s_{t+1}$ (including $s_{t+1}$ itself). Alternatively, if the transition
matrix $\bm{T}$ is known, where
$T_{ss^{\prime}}=p_{\pi}(s_{t+1}=s^{\prime}|s_{t}=s)$, the successor matrix
can be computed as the geometric series
$\bm{M}=\bm{I}+\gamma\bm{T}+\gamma^{2}\bm{T}^{2}+\ldots=(\bm{I}-\gamma\bm{T})^{-1}$.

The SR has also been proposed as a model of how humans and other animals learn
and generalize (Momennejad et al., , 2017; Stachenfeld et al., , 2017; Geerts
et al., , 2020; Gershman, , 2018). For example, humans have been shown to be
able to adapt more readily to changes in the reward function than changes in
the transition function in a simple sequential binary decision-making task
(Momennejad et al., , 2017), consistent with the SR facilitating rapid
adaptation to changes in $\bm{r}$ but only slow learning of $\bm{M}$.
Additionally, hippocampal ‘place cells’ have been proposed to encode a
predictive map, with each cell corresponding to a column of $\bm{M}$
(Stachenfeld et al., , 2017). In this model, the firing of a place cell in a
given location $s$ reflects the expected future occupancy of its ‘preferred’
location $s^{\prime}$ conditioned on currently being at $s$. Stachenfeld et
al.,  (2017) showed that the SR model explains a range of findings in the
hippocampal literature, including asymmetric place fields on a one-dimensional
track (Mehta et al., , 2000) and local remapping of place fields near a
barrier (Alvernhe et al., , 2011).

While the SR is perhaps the most prominent model in systems neuroscience that
combines features of model-free and model-based RL, it is not the only one.
Another interesting approach that has found parallels in the neuroscience
literature is the ‘DYNA’ architecture of Sutton,  (1991). In this framework, a
model of the world is learned from experience and used to train a model-free
policy offline by bootstrapping imagined experience sampled from the model.
This allows for more data-efficient learning of model-free policies at the
cost of additional compute during ‘rest’ but without necessarily needing more
compute at decision time, in contrast to the model-based approaches discussed
previously (Figure 4D). The model used to simulate data for offline training
can either be an explicit learned world model, or it can simply be a memory
buffer of past experiences in the form of
$(s_{t},a_{t},r_{t},s_{t+1},a_{t+1})$ tuples. Such experience replay has
proven crucial to the success of modern deep reinforcement learning agents by
allowing for higher data efficiency and reducing the instability arising from
the autocorrelated nature of online experience (Mnih et al., , 2013; Schaul et
al., , 2015). A prominent theory in neuroscience suggests that hippocampal
replays could be implementing such a DYNA-like algorithm by generating
imagined experience used to train the model-free RL systems of the brain
(Mattar and Daw, , 2018). This theory is supported by the finding that the
patterns of replay exhibited by rodents in various navigation tasks are
consistent with the optimal replays of a Q-learning agent with DYNA.

##  7 Deep reinforcement learning

So far, we have considered the setting of small state- and action-spaces, for
which we can simply learn a tabular policy. However, in many realistic
settings, the state space is large enough that we cannot enumerate all
possible states and actions. In these cases, we instead have to rely on
_function approximation_ (Sutton and Barto, , 2018). This approach makes the
assumption that similar states will also be associated with similar state-
action value functions and should therefore have similar policies. By making
this assumption, we can generalize to unseen states based on our previous
experience, provided that our function approximator is sufficiently good.
Recent years have seen incredible progress in this setting by using deep or
recurrent neural networks as powerful function approximators for reinforcement
learning – the domain of ‘deep reinforcement learning’. This approach has seen
an increase in interest not just in the machine learning literature, but
recently also as a model of neural dynamics and behaviour in humans and other
animals (Wang et al., , 2018; Jensen et al., , 2023; Makino, , 2023; Merel et
al., , 2019; Banino et al., , 2018). For a more comprehensive overview of the
links between deep RL and neuroscience, we refer the interested reader to
Botvinick et al.,  (2020). Modern approaches in (model-free) deep RL can
largely be divided into two categories: policy gradient methods and deep Q
learning. In policy gradient methods, we directly train the network to output
a policy, whereas deep Q networks (DQNs) output a set of state-action values
that can be used for action selection.

#### Policy gradient methods

The conceptually simplest approach for deep reinforcement learning relies on
the so-called policy gradient methods (Sutton and Barto, , 2018). In this
approach, we train a neural network with parameters $\theta$ to take as input
the (observable) state of the environment and directly output a policy
$\pi_{\theta}$. Our goal is then to find the setting of $\theta$ that
maximizes the expected reward in Equation 2. A conceptually simple way to
achieve this would be to define $R_{\tau}:=\sum_{t=0}^{T}\gamma^{t}r_{t}$ and
use gradient descent with gradients given by

$\displaystyle\nabla_{\theta}J(\theta)$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{\tau\sim\pi_{\theta}}\left[R_{\tau}\right]$
(32) $\displaystyle=\sum_{\tau}R_{\tau}\nabla_{\theta}p_{\theta}(\tau).$ (33)

Here, $\tau\sim\pi_{\theta}$ indicates trajectories sampled from the
distribution over trajectories induced by policy $\pi_{\theta}$, and
$J(\theta)$ indicates the expectation of $R_{\tau}$ under $\pi_{\theta}$ (c.f.
Equation 2). However, evaluating Equation 33 requires us to know how the
environment will respond to our actions in order to compute the necessary
derivatives, which in general may not be the case (although some model-based
deep RL approaches train a differentiable predictive model of the environment
to do this; Clavera et al., , 2020). Instead, we use the ‘log-derivative
trick’, which takes advantage of the linearity of the expectation and the
identity $\nabla_{\theta}\log
f(\theta)=f(\theta)^{-1}\nabla_{\theta}f(\theta)$ to write

$\displaystyle\nabla_{\theta}J(\theta)$
$\displaystyle=\sum_{\tau}R_{\tau}\nabla_{\theta}p_{\theta}(\tau)$ (34)
$\displaystyle=\sum_{\tau}R_{\tau}p_{\theta}(\tau)\nabla_{\theta}\log
p_{\theta}(\tau)$ (35)
$\displaystyle=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[R_{\tau}\nabla_{\theta}\log
p_{\theta}(\tau)\right],$ (36)

Since the environment does not depend on $\theta$, we can simplify the
calculation of $\nabla_{\theta}\log p_{\theta}(\tau)$:

$\displaystyle\nabla_{\theta}\log p_{\theta}(\tau)$
$\displaystyle=\nabla_{\theta}\left[\log p(s_{0})+\sum_{t=0}^{T}\log
p(s_{t+1}|s_{t},a_{t})+\log\pi_{\theta}(a_{t}|s_{t})\right]$ (37)
$\displaystyle=\sum_{t=0}^{T}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t}).$
(38)

Inserting Equation 37 in Equation 34 and taking a Monte Carlo estimate of the
expectation, we arrive at the REINFORCE algorithm (Williams, , 1992):

$\displaystyle\nabla_{\theta}J(\theta)$
$\displaystyle=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[R_{\tau}\sum_{t=0}^{T}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\right]$
(39)
$\displaystyle\approx\frac{1}{N}\sum_{\tau\sim\pi_{\theta}}\left(\sum_{t=0}^{T}\gamma^{t}r_{t}\right)\left(\sum_{t=0}^{T}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\right).$
(40)

Intuitively, Equation 40 says that we should preferentially upregulate the
probability of trajectories with high reward, which will lead to
downregulation of trajectories with lower reward due to the normalization of
the policy. Importantly, Equation 40 no longer requires us to differentiate
through the environment – only the policy.

While the REINFORCE algorithm is unbiased, it also has high variance, which
can make learning slow and unstable. It is therefore common to introduce
modifications, which can help reduce the variance. The first of these comes
from noting that an action taken at time $t$ cannot affect the reward received
at times $t^{\prime}<t$. This allows us to define
$R_{t}:=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r_{t^{\prime}}$ and
rewrite our REINFORCE update as

$\nabla_{\theta}J(\theta)\approx\frac{1}{N}\sum_{\tau\sim\pi_{\theta}}\sum_{t=0}^{T}R_{t}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t}).$
(41)

This is the formulation most commonly used, but it is worth noting that it is
actually _not_ the same as Equation 40, which would require us to use
$R_{t}=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-0}r_{t^{\prime}}$. As briefly
discussed in Section 2, this is because the discount factor $\gamma$ is
generally used as a variance reduction method rather than an indication that
we intrinsically care less about rewards later in the task, although Equation
41 is also consistent with an interpretation of $(1-\gamma)$ as a termination
probability.

It is also straightforward to show that for a baseline $B(s_{t})$ that does
not depend on $a_{t}$,

$\displaystyle\mathbb{E}_{s_{t},a_{t}\sim
p_{\theta}(s_{t},a_{t})}\left[B(s_{t})\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\right]$
$\displaystyle=\int_{s_{t}}B(s_{t})p(s_{t})\left[\nabla_{\theta}\int_{a_{t}}\pi_{\theta}(a_{t}|s_{t})da_{t}\right]ds_{t}$
(42)
$\displaystyle=\int_{s_{t}}B(s_{t})p(s_{t})\left[\nabla_{\theta}1\right]ds_{t}=0.$
(43)

A corollary of this result is that we can subtract such a baseline from our
empirical reward and still have an unbiased estimator while potentially
reducing its variance. A common choice here is the expected future reward

$V(s_{t})=\mathbb{E}\left[\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}=t}r_{t^{\prime}}|s_{t}\right]$
(44)

This gives rise to the so-called ‘actor-critic’ algorithm

$\nabla_{\theta}J(\theta)\approx\frac{1}{N}\sum_{\tau\sim\pi_{\theta}}\sum_{t=0}^{T}(R_{t}-V(s_{t}))\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t}).$
(45)

Intuitively, Equation 45 says that we should upregulate the probability of
actions that lead to higher-than-expected reward and downregulate the
probability of actions that lead to lower-than-expected reward. Interestingly,
many studies in the neuroscience literature have suggested that the brain
could be using something akin to an actor-critic algorithm, with dorsal
striatum implementing the ‘actor’ and ventral striatum the ‘critic’ (Takahashi
et al., , 2008; Sutton and Barto, , 2018).

Finally, it is common to reduce the variance of our gradient estimator through
an approach known as ‘bootstrapping’. To implement this, we note that

$\mathbb{E}\left[R_{t}\right]=\mathbb{E}[r_{t}+\gamma V(s_{t+1})].$ (46)

This is useful because $r_{t}+\gamma V(s_{t+1})$ has lower variance than
$R_{t}$, so using this bootstrapped estimate of the reward-to-go can reduce
the noise in our gradient estimate when using finite sample sizes. We
therefore use $r_{t}+\gamma V(s_{t+1})$ to define the ‘advantage function’
$A_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})$, which gives rise to the ‘advantage
actor-critic’ (A2C) algorithm:

$\nabla_{\theta}J(\theta)\approx\frac{1}{N}\sum_{\tau\sim\pi_{\theta}}\sum_{t=0}^{T}(r_{t}+\gamma
V(s_{t+1})-V(s_{t}))\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t}).$ (47)

While we have considered two extreme cases of a full Monte Carlo estimate of
$R_{t}$ and a ‘one-step’ bootstrap, the sum in $R_{t}$ can be truncated to any
order with $R_{t^{\prime}}$ replaced by $V(s_{t^{\prime}})$. In theory, this
estimator remains unbiased if our value function is correct. However, in
practice the estimate of $V(s_{t^{\prime}})$ learned by our critic will be
inexact, which introduces a bias to our parameter updates. The bootstrapping
procedure outlined above thus leads to a tradeoff between the bias and
variance of our parameter updates, and the optimal tradeoff will depend on the
problem setting, network architecture, and training paradigm being used.

For these actor-critic algorithms, it is common to parameterize both the
policy $\pi_{\theta}(a_{t}|s_{t})$ and the value function $V_{\phi}(s_{t})$
with neural networks – often with a subset of shared parameters between
$\theta$ and $\phi$. This leaves the problem of updating the parameters. In
order to use out-of-the-box automatic differentiation software, we do this by
defining an auxiliary utility (i.e. negative loss)

$\tilde{J}(\theta)=\frac{1}{N}\sum_{\tau\sim\pi_{\theta}}\sum_{t=0}^{T}(R_{t}-V(s_{t}))\log\pi_{\theta}(a_{t}|s_{t}),$
(48)

where $R_{t}$ and $V(s_{t})$ are treated as constant w.r.t. $\theta$. In
Equation 48 and the following equations, we can use either the empirical
reward-to-go $R_{t}=\sum_{t^{\prime}\geq
t}\gamma^{t^{\prime}-t}r_{t^{\prime}}$, or we can use the bootstrapped
estimate $R_{t}\approx r_{t}+\gamma V(s_{t+1})$. While $\tilde{J}(\theta)$ has
no intrinsic interpretation, it is chosen such that

$\nabla_{\theta}\tilde{J}(\theta)=\nabla_{\theta}J(\theta).$ (49)

The gradient of the value function loss (i.e. negative utility) is given by

$\nabla_{\phi}\mathcal{L}_{V}=\nabla_{\phi}\sum_{t}\frac{1}{2}(R_{t}-V_{\phi}(s_{t}))^{2}=\sum_{t}-(R_{t}-V_{\phi}(s_{t}))\nabla_{\phi}V_{\phi}(s_{t}).$
(50)

When $\pi_{\theta}$ and $V_{\phi}$ share parameters, we also need to balance
the gradients w.r.t. $\theta$ and $\phi$ in our update step. We do this by
defining $\delta_{t}:=R_{t}-V(s_{t})$ (or $\delta_{t}:=r_{t}+\gamma
V(s_{t+1})-V(s_{t})$), introducing a hyperparameter $\beta_{V}$, and
maximizing the joint utility

$\mathcal{L}=\frac{1}{N}\sum_{\tau\sim\pi_{\theta}}\sum_{t=0}^{T}(\log\pi_{\theta}(a_{t}|s_{t})+\beta_{V}V_{\phi}(s_{t}))\delta_{t}.$
(51)

Importantly, we do not propagate gradients w.r.t. $\phi$ or $\theta$ through
the computation of $\delta_{t}$. Finally, it is also common to add an
auxiliary entropy loss of the form
$\mathcal{L}_{E}=\beta_{E}\frac{1}{N}\sum_{\tau\sim\pi_{\theta}}\sum_{t=0}^{T}\sum_{a}\pi_{\theta}(a_{t}|s_{t})\log\pi_{\theta}(a_{t}|s_{t})$
to encourage exploration.

Figure 5:  Meta reinforcement learning. The results in this figure reproduce
some of the preliminary analyses in Figure 1 of Wang et al.,  (2018). (A) We
trained a recurrent meta-reinforcement learning agent in a two-armed bandit
task, where the reward probabilities of each arm were sampled independently
from $\mathcal{U}(0,1)$ at the beginning of each episode and remained fixed
throughout the episode. A recurrent neural network was trained across many
episodes with _different reward probabilities_ using an actor-critic
algorithm. The input to the agent consisted of the previous action, the
pervious reward, and the time-within-trial. The average reward per episode is
plotted against the episode number, showing that the agent gradually learns to
adapt within each episode to the particular instantiation of the bandit task.
Importantly, the parameters of the network are fixed within an episode,
meaning that this adaptation occurs through the recurrent dynamics. Dashed
horizontal lines indicate the reward of an agent selecting random actions and
an ‘oracle’ agent that always chooses the best arm. (B) Heatmap showing
example behaviour of the agent in episodes with different reward probabilities
for the first arm, $p(r|a=1)$. For these plots, the probability of reward from
the second arm was set to $p(r|a=2)=1-p(r|a=1)$. Across episodes, the agent
experiments with different actions and eventually converges on taking the
optimal action. For episodes with more symmetric reward probabilities (near
the middle), it takes longer for the agent to identify the optimal action.
This balance between exploration and exploitation is mediated by the recurrent
network dynamics, which are learned over many episodes using deep
reinforcement learning. (C) We averaged the hidden state of the RNN over 100
episodes for different reward probabilities, ranging from high reward for
action 1 (blue) to low reward for action 1 (green), and performed PCA on the
resulting matrix of average hidden states for different reward probabilities.
This figure shows the 2-dimensional embedding of neural activity in the meta-
RL agent, where we see that the agent converges to different regions of state
space for different reward probabilities. Crosses indicate the hidden state at
the beginning of each episode.

While these policy gradient methods may seem far removed from neuroscience,
they have had several applications in the field in recent years. For example,
Li et al.,  (2022) recently demonstrated that an off-policy variant known as
the ‘soft actor-critic’ algorithm (Haarnoja et al., , 2018) can be used to
train an agent that shapes the behaviour of C. elegans via an optogenetic
action space. Networks trained with policy gradient algorithms have also
recently attracted significant interest as models of learning and neural
dynamics in the biological brain (Wang et al., , 2018; Jensen et al., , 2023;
Merel et al., , 2019). Of particular interest, Wang et al.,  (2018) suggested
that frontal cortex can be well described as a _recurrent_ deep RL agent,
where the RNN parameters are configured by learning from rewards over long
periods of time from many tasks that have a shared underlying structure.
Importantly, this ‘slow’ model-free learning process gives rise to an agent
that can rapidly learn from experience with _fixed parameters_ when exposed to
a new task from the same distribution. This is achieved by the agent learning
to effectively implement a fast RL-like algorithm in the _dynamics_ of the
network (Figure 5). This process, whereby an agent trained slowly on a large
distribution of tasks can rapidly adapt to a new task, is known as ‘meta-
reinforcement learning’ and is a popular area of research in machine learning
(Finn et al., , 2017; Ritter et al., , 2018; Duan et al., , 2016; Wang et al.,
, 2016). Wang et al.,  (2018) showed that such as meta-RL model can explain a
range of neuroscientific findings, including

  * •

Dynamic adaptation of the effective learning rate of an agent to the
volatility of the environment (Behrens et al., , 2007).

  * •

The emergence of ‘model-based’ behaviour in the ‘two-step’ task commonly used
to distinguish between model-free and model-based RL (Miller et al., , 2017;
Daw et al., , 2011).

  * •

The ability of animals to get progressively faster at learning when exposed to
multiple tasks with a consistent abstract task structure (Harlow, , 1949).

Recently, Jensen et al.,  (2023) also extended the work of Wang et al.,
(2018) to allow the meta-RL agent to learn not just from physical experience,
but also from imagined experience using a learned model of the environment.
This resulted in an agent that computes an intial model-free policy, which can
be adaptively improved with model-based computations when the resulting policy
improvement makes up for the temporal opportunity cost of the model-based
computation. Interestingly, the response times of the resulting agent mirrored
those of human participants in a maze navigation task, where the agent
similarly spent more time ‘thinking’ early in a trial and far from the goal.

#### Deep Q-learning

While the policy gradient methods discussed above are useful, they also have
limitations. One is that most policy gradient methods are fundamentally ‘on-
policy’, which means that the data used for learning must be sampled from the
agent itself. This is necessary for the log derivative trick to work. Another
is that the gradient can have fairly high variance despite all of our variance
reduction efforts. We can begin to tackle both of these shortcomings using a
method known as ‘deep Q-learning’. To do so, we first note that our actor-
critic algorithm above required us to estimate the value function
$V_{\phi}(s_{t})$. However, as we saw in Equation 11, such value functions can
directly be used for action selection without having to fit a separate policy
network. We also noted previously that this requires simulating the results of
the actions, which can be inconvenient. As in the tabular setting, we can
circumvent this problem by directly fitting state-action values (‘Q-values’),
now using function approximation. This gives rise to the family of ‘deep
Q-learning’ methods, which closely mirror the tabular Q-learning considered
previously, but now with function approximation.

The general recipe involves defining a Q function $Q_{\theta}(a,s)$, where the
parameters $\theta$ of the deep network defining our agent are learned as
follows:

  * •

Collect experience $(s_{t},a_{t},r_{t},s_{t+1})$.

  * •

Define a loss
$\mathcal{L}=0.5[Q_{\theta}(s_{t},a_{t})-(r_{t}+\gamma\text{max}_{a}Q_{\theta}(s_{t+1},a))]^{2}$.

  * •

Update the network parameters
$\Delta\theta\propto-\frac{\partial\mathcal{L}}{\partial\theta}$.

The above steps can either be run ‘online’ using the experience of the agent
as it is being generated, or it could occur ‘in batch’, whereby a collection
of experiences is first generated, followed by optimization of the Q network.
The ‘max’ in the target value ensures that we continuously improve our policy.
When acting according to our policy, we simply pick the action predicted to
have the highest value, usually using some variant of $\epsilon$-greedy or
softmax to introduce stochasticity and variability into our trajectories.

On the surface, the above looks like a straightforward generalization of
tabular Q-learning, and it may seem surprising that deep Q-learning did not
see significant use or success until the foundational work of Mnih et al.,
(2013). However, a major difficulty arises from the autocorrelation of the
states observed by the agent, which leads to the network ‘chasing’ a target
function $(r_{t}+\gamma\text{max}_{a}Q_{\theta}(s_{t+1},a))$ that changes with
the environment. A critical advance that overcame the resulting instability
was the use of an experience replay buffer, whereby the experience generated
by the agent is added to a global replay buffer $\mathcal{B}$. One or more
_random_ experiences are then sampled from the buffer and used to update the
network parameters – reminiscent of the ‘DYNA’ architecture described
previously. An additional algorithmic instability arises from the fact that
$\mathcal{L}$ includes the term $\text{max}_{a}Q_{\theta}(s_{t+1},a)$, which
we cannot take gradients through despite its dependence on $\theta$. This
means that we are trying to optimize our parameters for a continuously moving
target, which destabilitizes the optimization process. To combat this, it is
common to use a ‘target network’ $Q_{\theta^{\prime}}(s_{t+1},a)$ that is
fixed for a period of time, usually after being set to a copy of our ‘student
network’. Together, these two approaches give rise to the ‘deep Q network’
(DQN) developed by Mnih et al.,  (2013), which is trained as follows:

  * •

Collect experience $(s_{t},a_{t},r_{t},s_{t+1})$ and add to $\mathcal{B}$
[optionally many iterations].

  * •

Sample a _new_ experience
$(s^{\prime}_{t},a^{\prime}_{t},r^{\prime}_{t},s^{\prime}_{t+1})\sim\mathcal{B}$
[optionally a full batch].

  * •

Define a loss
$\mathcal{L}=0.5[Q_{\theta}(s^{\prime}_{t},a^{\prime}_{t})-(r^{\prime}_{t}+\gamma\text{max}_{a}Q_{\theta^{\prime}}(s^{\prime}_{t+1},a))]^{2}$
[optionally averaged over the full batch]. Note that we are training the
student network with parameters $\theta$ while using a target network with
parameters $\theta^{\prime}$ inside the max.

  * •

Update the network parameters
$\Delta\theta\propto-\frac{\partial\mathcal{L}}{\partial\theta}$.

  * •

Once we have repeated the above a sufficient number of times, set our target
network to the student network, $\theta^{\prime}\leftarrow\theta$.

As indicated above, this procedure usually involves averaging over a full
batch of experience when computing gradients. Experience can also be collected
in batch, and usually ‘stale’ experiences are periodically removed from
$\mathcal{B}$. It is worth noting that this algorithm is effectively off-
policy, since most experience in $\mathcal{B}$ was collected by a policy
defined by an old set of parameters $\theta$ – and the data in $\mathcal{B}$
can in fact be generated completely independently of the agent we are
training. Finally, even though the above is more stable than naive deep
Q-learning, an additional instability arises from the fact that
$Q_{max}(s^{\prime}_{t+1})=\text{max}_{a}Q_{\theta^{\prime}}(s^{\prime}_{t+1},a)$
uses the same Q values both to estimate which action is best and what the
value of that action is, which leads to a positively biased estimate. This can
be mitigated by using _separate_ Q networks to select the best action and
evaluate its value, $Q_{max}(s^{\prime}_{t+1})\leftarrow
Q_{\theta^{\prime}}(s^{\prime}_{t+1},\text{argmax}_{a}(Q_{\theta}(s^{\prime}_{t+1},a)))$,
in an approach known as ‘double Q-learning’ (Van Hasselt et al., , 2016).
Commonly, this is done by simply using the student network for action
selection and the target network to evaluate its expected value.

While DQNs have shown impressive performance across a range of machine
learning settings (Mnih et al., , 2013; Lillicrap et al., , 2015; Schaul et
al., , 2015; Kalashnikov et al., , 2018), they have seen less interest from
the neuroscience community. This is perhaps paradoxical given the prevalence
of tabular Q-learning in the theoretical neuroscience literature. An
interesting exception to this is recent work by Makino,  (2023), which shows
parallels between the values learned by a DQN and neural representations in
mammalian cortex during a compositional behavioural task. Additionally, the
importance of experience replay in DQNs (Mnih et al., , 2013; Schaul et al., ,
2015) has close parallels to the proposal that hippocampal replay constitutes
a form of experience replay (Mattar and Daw, , 2018) – although the latter was
only analyzed in a tabular setting.

## Distributional reinforcement learning

In Equation 3 and Equation 12, we defined the _expected_ reward for a given
state or state-action pair. The methods considered so far have only used such
expectations as a learning signal. However, recent work has suggested that
performance could be improved by learning the full _distribution_ of
cumulative future rewards for a given state-action pair (Bellemare et al., ,
2017, 2023; Dabney et al., , 2018), a single sample of which we denote $Z$:

$Z(s,a)\sim
p\left(\sum_{t^{\prime}>=t}\gamma^{t^{\prime}-t}r_{t^{\prime}}|s_{t}=s,a_{t}=a\right)$
(52)

The stochasticity of $Z$ can both be due to stochasticity in environment
dynamics and reward, and it can be due to stochasticity in the policy of the
agent itself. Clearly, the expectation of samples of $Z$ equal the
corresponding Q value:

$\mathbb{E}_{p(Z)}\left[Z(s,a)\right]=Q(s,a).$ (53)

Instead of only estimating this expectation, we now want to learn the full
distribution of returns under our policy $\pi$, $p_{\pi}(Z(s,a))$. One use of
learning this distribution is to develop methods that are risk averse
(Morimura et al., , 2010, 2012) or explicitly take into account uncertainty
(Dearden et al., , 1998). However, recent work has suggested that such a
distributional approach can also lead to improved performance in terms of
expected rewards by improving representation learning in the deep RL setting
(Bellemare et al., , 2017; Dabney et al., , 2018; Rowland et al., , 2019;
Bellemare et al., , 2023). Some intuition for this can be had from noting that
traditional deep RL approaches will only learn to distinguish states that have
different expected value, while distributional RL will learn to distinguish
any states that have different cumulative return distributions.

To learn $p_{\pi}(Z(s,a))$, one option is to define a set of ‘atoms’ that tile
the (expected) support of the distribution and then discretize the
distribution by assigning the full probability mass to these discrete
locations. An approximate $p_{\pi}(Z(s,a))$ can then be iteratively improved
using temporal differences by propagating each atom through the Bellman
equation (Equation 15), projecting the density back onto the discretized
locations, and training an RL agent to predict the corresponding
probabilities. This is the approach originally employed by Bellemare et al.,
(2017). However, it turns out to be more convenient from a neuroscientific
perspective to represent a different set of sufficient statistics for
$p_{\pi}(Z(s,a))$ (Dabney et al., , 2020; Lowet et al., , 2020). We do this by
defining the $\tau\textsuperscript{th}$ _expectile_ of $p_{\pi}(Z(s,a))$,
$\epsilon_{\tau}$, which is defined for a random variable $Z$ as the solution
to the equation

$\tau\mathbb{E}[\text{max}(0,Z-\epsilon_{\tau})]=(1-\tau)\mathbb{E}[\text{max}(0,\epsilon_{\tau}-Z)].$
(54)

This is a generalization of the mean, which is recovered for $\tau=0.5$, and
is similar to how the quantile generalizes the notion of a median. A
distribution is uniquely specified by its expectiles, and we can therefore
represent $p_{\pi}(Z(s,a))$ in terms of $\\{\epsilon_{\tau}\\}$. Translating
this to an RL algorithm involves training a network (or tabular values) to
predict a set of expectiles for a given state (and action). The parameters of
the agent are then updated by propagating the distribution implied by the
predicted expectiles through the Bellman equation and minimizing the
difference between the initial and propagated distributions.

In the tabular value learning case (c.f. Section 3), this turns out to be
particularly simple do via the use of a modified TD-update rule (c.f. Equation
9; Lowet et al., , 2020; Figure 6A). In particular, we consider a set of units
$\\{V_{i}(s)\\}$, each with a target expectile
$\tau_{i}:=\frac{\alpha_{i}^{+}}{\alpha_{i}^{+}+\alpha_{i}^{-}}$ of the return
distribution $p_{\pi}(Z(s))$ (where
$Z(s)=\mathbb{E}_{a\sim\pi(a|s)}\left[Z(a,s)\right]$). These expectiles can be
learned by sampling experience from the environment under the policy and
defining a TD error for each unit and state transition as

$\displaystyle\delta_{i}:=r_{t}+\gamma\tilde{Z}_{j}(s_{t+1})-V_{i}(s_{t}).$
(55)

Here, $\tilde{Z}_{j}(s_{t+1})$ is a _random sample_ from the learned
approximate distribution of cumulative future returns from state $s_{t+1}$,
$p_{\\{V_{i}\\}}(\tilde{Z}(s_{t+1}))$ (Lowet et al., , 2020; Dabney et al., ,
2020). We then apply the following update rule to all units:

$\Delta
V_{i}(s_{t})=\alpha_{i}^{+}\text{max}(\delta_{i},0)+\alpha_{i}^{-}\text{min}(\delta_{i},0).$
(56)

In other words, we apply the TD update rule to each unit with learning rate
$\alpha_{i}^{+}$ for positive TD errors and learning rate $\alpha_{i}^{-}$ for
negative TD errors. When running this algorithm to convergence, $V_{i}(s)$
will approach the $\tau_{i}\textsuperscript{th}$ expectile
($\epsilon_{\tau_{i}}$) of $p_{\pi}(Z(s))$. In the deep RL setting, we would
additionally use the chain rule to multiply $\Delta V_{i}(s_{t})$ from
Equation 56 by $\partial V_{i}(s_{t})/\partial\theta$ and sum over $i$ to
learn a model with parameters $\theta$ that can predict the full set of
expectiles. Another variant of this algorithm updates each unit with
$\alpha_{i}\text{sign}(\delta_{i})$ instead of $\alpha_{i}(\delta_{i})$, which
leads to learning of the $\tau_{i}\textsuperscript{th}$ _quartiles_ of the
distribution instead of the expectiles. We refer to Bellemare et al.,  (2017);
Dabney et al.,  (2018); Rowland et al.,  (2019); Bellemare et al.,  (2023);
and Dabney et al.,  (2020) for additional mathematical details and extensions
to the control setting.

Figure 6:  Distributional reinforcement learning. The results in this figure
reproduce some of the key simulations of Dabney et al.,  (2020). These
simulations were all run on a simple value estimation task with a single state
and no actions. (A) Distribution of rewards (black), plotted together with the
values $V_{i}$ converged to across 20 units all learning through standard TD
learning (blue), or through distributional RL with different
$\tau_{i}=\frac{\alpha_{i}^{+}}{\alpha_{i}^{+}+\alpha_{i}^{-}}$ (green). The
TD units all converge to the mean reward, while the expectile units end up
tiling the distribution. (B) For both the TD units and distributional units
from (A), we plot the temporal difference updates performed in response to
different rewards from the reward distribution. These updates have been
proposed to be represented in the firing rates of dopaminergic VTA neurons
relative to baseline (Schultz et al., , 1997; Dabney et al., , 2020). The TD
units show a constant linear scaling across positive and negative rewards,
while the distributional units show an asymmetric scaling of firing rate with
reward magnitude above and below their reversal point (black horizontal line).
The ratio of slopes above and below the reversal point scales positively with
the value of the reversal point. These features of dopaminergic VTA neurons
were used by Dabney et al.,  (2020) to argue that the brain implements a form
of distributional RL. (C) True reward distribution (black) reproduced from
(A), now plotted together with the reward distribution
$p_{\\{V_{i}\\}}(\tilde{Z})$ implied by the distributional units from (A) and
(B) at different stages of learning (light to dark green). These imputed
distributions were computed by _assuming_ that the value $V_{i}$ learned by
unit $i$ corresponds to expectile
$\tau_{i}=\frac{\alpha_{i}^{+}}{\alpha_{i}^{+}+\alpha_{i}^{-}}$ of the reward
distribution. We then impute the distribution implied by these expectiles
under the assumption that it consists of a set of $N$ delta functions in the
case of $N$ expectiles (Rowland et al., , 2019). Finally, we convolved the
resulting delta functions with a Gaussian kernel ($\sigma=0.1$) for
visualization. This whole process was repeated using $\\{V_{i}\\}$ at
different stages of learning. The units were all initialized at $V_{i}=0.5$,
so the initial distribution is a delta function at $r=0.5$. At the end of
learning, the population faithfully represents the true reward distribution,
capturing key features including bimodality and the relative magnitude of the
two modes. Dabney et al.,  (2020) used a similar approach to infer the
distribution implicitly encoded by dopaminergic VTA neurons at the end of
animal training and found a close match to the true reward distribution.

Intriguingly, recent work in neuroscience has suggested that a similar
algorithm could potentially underlie value learning in biological neural
circuits (Dabney et al., , 2020; Lowet et al., , 2020). In particular, Dabney
et al.,  (2020) recorded the activity of dopaminergic VTA neurons during a
task with stochastic rewards and showed that the neurons appeared to represent
the full distribution of possible outcomes using an expectile representation
as described above. More concretely, they showed that:

  * •

The VTA neurons exhibited a range of different ‘reversal points’ – defined as
the reward magnitude at which the firing rate of a neuron did not change from
its baseline firing rate. This is consistent with a distributional RL theory,
where the changes in neural firing rates from baseline correspond to the
expectile TD updates considered above. In this case, the reversal point of a
neuron $i$ should be $V_{i}\approx\epsilon_{\tau_{i}}$ (Figure 6B).

  * •

Neurons had different slopes describing the relationship between expected
reward and firing rate in the regimes where expected reward was above and
below the reversal point ($V_{i}$) of each neuron. This is consistent with the
algorithm described in Equation 56 (Figure 6B).

  * •

When independently fitting a slope to the data above ($\alpha_{i}^{+}$) and
below ($\alpha_{i}^{-}$) the reversal point of neuron $i$, the reversal point
of the neuron was positively correlated with
$\tau_{i}=\frac{\alpha_{i}^{+}}{\alpha_{i}^{-}+\alpha_{i}^{+}}$. This is
consistent with the expectile distributional RL setting, where the reversal
point is $V_{i}\approx\epsilon_{\tau_{i}}$ (Figure 6B).

  * •

When ‘imputing’ the distribution (approximated as a set of delta functions)
implied by the VTA neurons when interpreted as expectiles (Figure 6C), the
resulting fit was remarkably similar to the true distribution of rewards in
the experiment.

These findings generalize the canonical RPE view of Schultz et al.,  (1997),
which can be seen as the averaged version of the theory put forward by Dabney
et al.,  (2020). The expectile regression algorithm investigated by Dabney et
al.,  (2020) relies on non-local TD updates and non-linear ‘imputation’ of the
return distribution $p_{\\{V_{i}\\}}(\tilde{Z}(s))$ induced by the learned
expectiles $\\{V_{i}(s)\\}$. However, recent work has suggested more
biologically plausible distributional RL updates that are also consistent with
the biological data (Tano et al., , 2020).

##  8 Discussion

In this work, we have provided a mathematical overview of some of the many
reinforcement learning methods that are commonly used in systems and
computational neuroscience. We have also highlighted a range of explicit
parallels between these methods and findings in the neuroscience and cognitive
science literature to illustrate the utility of reinforcement learning as a
framework for understanding biological learning and decision making. This has
ranged from classical work on reward prediction errors (Schultz et al., ,
1997) to recent findings on the plausibility of distributional reinforcement
learning in biological circuits (Dabney et al., , 2020).

While RL has thus already had a profound influence on systems neuroscience,
several open questions remain. In particular, much of the work in neuroscience
has focused on simple stimulus-response or binary decision making tasks. This
is a far reach from ethologically relevant problems that involve processing
complex multimodal stimuli, decision making with long-lasting consequences,
and complex high-dimensional motor control. Some recent work building on deep
RL has started to bridge this gap. For example, Banino et al.,  (2018) showed
the emergence of grid cells in agents navigating complex environments, Merel
et al.,  (2019) showed similarities between an RL agent trained on a suite of
complex motor tasks and rodent motor representations, and Jensen et al.,
(2023) showed parallels between a recurrent meta-RL agent and human behaviour
in a navigation task requiring temporally extended thinking. However, much
work remains to be done to extend our neuroscientific findings to
ethologically relevant settings, both on the experimental side and on the
computational side.

A related challenge will be to combine different components of existing models
to capture the generalist nature of biological circuits. This is in contrast
to past work, which has often focused on a single neural circuit or function,
such as motor control or navigation. Such a generalist approach will involve
explicit modeling of the roles of different brain regions, and more
importantly it will require us to capture how they interact with one another
during learning and decision making. Clearly, such models will need to be
constrained by experimental data, both at the level of behaviour and at the
level of neural activity. This is becoming increasingly feasible with recent
advances in recording technologies, both for high-resolution behavioural
tracking (Mathis et al., , 2018; Dunn et al., , 2021) and for simultaneous and
long-term recording of neural activity (Steinmetz et al., , 2021; Pachitariu
et al., , 2016; Dhawale et al., , 2017).

Finally, most work on reinforcement learning in a neuroscientific context has
considered short-term decision making tasks, where planning and decision
making in primitive state and action spaces is feasible. This is in stark
contrast to most human decision making, which occurs over extended timescales
and often involves hierarchies of decisions. For example, we may decide to
pursue an undergraduate degree at Cambridge University, which then requires us
to (i) write an application, (ii) prepare for an interview, and (iii) arrange
our travel. Each of these processes in turn require us to plan increasingly
low-level decisions, such as booking a flight or deciding which bus to take to
the airport. This is the topic of hierarchical reinforcement learning, which
has already been highlighted as a potentially useful model of human behaviour
(Eckstein and Collins, , 2020; Botvinick, , 2008; Botvinick et al., , 2009)
and is becoming an increasingly important area of research in machine learning
(Pateria et al., , 2021).

As is the case for hierarchical RL, this review has unfortunately not been
able to cover in detail all topics in reinforcement learning that are of
interest to neuroscientists. A brief overview of this and additional topics of
interest is therefore provided in Section 9, which we hope can also serve as a
useful pointer to the broader reinforcement learning literature and some of
the active research topics being pursued by the community.

## References

  * Agrawal et al., (2022) Agrawal, M., Mattar, M. G., Cohen, J. D., and Daw, N. D. (2022).  The temporal dynamics of opportunity costs: A normative account of cognitive fatigue and boredom.  Psychological review, 129(3):564.
  * Akam et al., (2015) Akam, T., Costa, R., and Dayan, P. (2015).  Simple plans or sophisticated habits? state, transition and learning interactions in the two-step task.  PLoS computational biology, 11(12):e1004648.
  * Alvernhe et al., (2011) Alvernhe, A., Save, E., and Poucet, B. (2011).  Local remapping of place cell firing in the tolman detour task.  European Journal of Neuroscience, 33(9):1696–1705.
  * Banino et al., (2018) Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., Pritzel, A., Chadwick, M. J., Degris, T., Modayil, J., et al. (2018).  Vector-based navigation using grid-like representations in artificial agents.  Nature, 557(7705):429–433.
  * Barreto et al., (2017) Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. (2017).  Successor features for transfer in reinforcement learning.  Advances in neural information processing systems, 30.
  * Behrens et al., (2007) Behrens, T. E., Woolrich, M. W., Walton, M. E., and Rushworth, M. F. (2007).  Learning the value of information in an uncertain world.  Nature neuroscience, 10(9):1214–1221.
  * Bellemare et al., (2017) Bellemare, M. G., Dabney, W., and Munos, R. (2017).  A distributional perspective on reinforcement learning.  In International conference on machine learning, pages 449–458. PMLR.
  * Bellemare et al., (2023) Bellemare, M. G., Dabney, W., and Rowland, M. (2023).  Distributional reinforcement learning.  MIT Press.
  * Blanco-Pozo et al., (2021) Blanco-Pozo, M., Akam, T., and Walton, M. (2021).  Dopamine reports reward prediction errors, but does not update policy, during inference-guided choice.  bioRxiv.
  * Botvinick and Toussaint, (2012) Botvinick, M. and Toussaint, M. (2012).  Planning as inference.  Trends in cognitive sciences, 16(10):485–488.
  * Botvinick et al., (2020) Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., and Kurth-Nelson, Z. (2020).  Deep reinforcement learning and its neuroscientific implications.  Neuron, 107(4):603–616.
  * Botvinick, (2008) Botvinick, M. M. (2008).  Hierarchical models of behavior and prefrontal function.  Trends in cognitive sciences, 12(5):201–208.
  * Botvinick and Cohen, (2014) Botvinick, M. M. and Cohen, J. D. (2014).  The computational and neural basis of cognitive control: charted territory and new frontiers.  Cognitive science, 38(6):1249–1285.
  * Botvinick et al., (2009) Botvinick, M. M., Niv, Y., and Barto, A. G. (2009).  Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective.  cognition, 113(3):262–280.
  * Callaway et al., (2022) Callaway, F., van Opheusden, B., Gul, S., Das, P., Krueger, P. M., Griffiths, T. L., and Lieder, F. (2022).  Rational use of cognitive resources in human planning.  Nature Human Behaviour, 6(8):1112–1125.
  * Clavera et al., (2020) Clavera, I., Fu, V., and Abbeel, P. (2020).  Model-augmented actor-critic: Backpropagating through paths.  arXiv preprint arXiv:2005.08068.
  * Dabney et al., (2020) Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., and Botvinick, M. (2020).  A distributional code for value in dopamine-based reinforcement learning.  Nature, 577(7792):671–675.
  * Dabney et al., (2018) Dabney, W., Rowland, M., Bellemare, M., and Munos, R. (2018).  Distributional reinforcement learning with quantile regression.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
  * Daw et al., (2011) Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., and Dolan, R. J. (2011).  Model-based influences on humans’ choices and striatal prediction errors.  Neuron, 69(6):1204–1215.
  * Daw et al., (2005) Daw, N. D., Niv, Y., and Dayan, P. (2005).  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control.  Nature neuroscience, 8(12):1704–1711.
  * Dayan, (1993) Dayan, P. (1993).  Improving generalization for temporal difference learning: The successor representation.  Neural computation, 5(4):613–624.
  * Dearden et al., (1998) Dearden, R., Friedman, N., and Russell, S. (1998).  Bayesian q-learning.  Aaai/iaai, 1998:761–768.
  * Deisenroth and Rasmussen, (2011) Deisenroth, M. and Rasmussen, C. E. (2011).  Pilco: A model-based and data-efficient approach to policy search.  In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465–472.
  * Dhawale et al., (2017) Dhawale, A. K., Poddar, R., Wolff, S. B., Normand, V. A., Kopelowitz, E., and Ölveczky, B. P. (2017).  Automated long-term recording and analysis of neural activity in behaving animals.  Elife, 6:e27702.
  * Duan et al., (2016) Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P. (2016).  RL2: Fast reinforcement learning via slow reinforcement learning.  arXiv preprint arXiv:1611.02779.
  * Dunn et al., (2021) Dunn, T. W., Marshall, J. D., Severson, K. S., Aldarondo, D. E., Hildebrand, D. G., Chettih, S. N., Wang, W. L., Gellis, A. J., Carlson, D. E., Aronov, D., et al. (2021).  Geometric deep learning enables 3d kinematic profiling across species and environments.  Nature methods, 18(5):564–573.
  * Eckstein and Collins, (2020) Eckstein, M. K. and Collins, A. G. (2020).  Computational evidence for hierarchically structured reinforcement learning in humans.  Proceedings of the National Academy of Sciences, 117(47):29381–29389.
  * Finn et al., (2017) Finn, C., Abbeel, P., and Levine, S. (2017).  Model-agnostic meta-learning for fast adaptation of deep networks.  In International conference on machine learning, pages 1126–1135. PMLR.
  * Geerts et al., (2020) Geerts, J. P., Chersi, F., Stachenfeld, K. L., and Burgess, N. (2020).  A general model of hippocampal and dorsal striatal learning and decision making.  Proceedings of the National Academy of Sciences, 117(49):31427–31437.
  * Gershman, (2018) Gershman, S. J. (2018).  The successor representation: its computational logic and neural substrates.  Journal of Neuroscience, 38(33):7193–7200.
  * Griffiths et al., (2019) Griffiths, T. L., Callaway, F., Chang, M. B., Grant, E., Krueger, P. M., and Lieder, F. (2019).  Doing more with less: meta-reasoning and meta-learning in humans and machines.  Current Opinion in Behavioral Sciences, 29:24–30.
  * Gronauer and Diepold, (2022) Gronauer, S. and Diepold, K. (2022).  Multi-agent deep reinforcement learning: a survey.  Artificial Intelligence Review, pages 1–49.
  * Haarnoja et al., (2018) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018).  Soft actor-critic algorithms and applications.  arXiv preprint arXiv:1812.05905.
  * Harlow, (1949) Harlow, H. F. (1949).  The formation of learning sets.  Psychological review, 56(1):51.
  * Hart et al., (1968) Hart, P. E., Nilsson, N. J., and Raphael, B. (1968).  A formal basis for the heuristic determination of minimum cost paths.  IEEE transactions on Systems Science and Cybernetics, 4(2):100–107.
  * Jaderberg et al., (2016) Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. (2016).  Reinforcement learning with unsupervised auxiliary tasks.  arXiv preprint arXiv:1611.05397.
  * Jensen et al., (2023) Jensen, K. T., Hennequin, G., and Mattar, M. G. (2023).  A recurrent network model of planning explains hippocampal replay and human behavior.  bioRxiv, pages 2023–01.
  * Kalashnikov et al., (2018) Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018).  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  arXiv preprint arXiv:1806.10293.
  * Keramati et al., (2016) Keramati, M., Smittenaar, P., Dolan, R. J., and Dayan, P. (2016).  Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum.  Proceedings of the National Academy of Sciences, 113(45):12868–12873.
  * Killcross and Coutureau, (2003) Killcross, S. and Coutureau, E. (2003).  Coordination of actions and habits in the medial prefrontal cortex of rats.  Cerebral cortex, 13(4):400–408.
  * Lai and Gershman, (2021) Lai, L. and Gershman, S. J. (2021).  Policy compression: An information bottleneck in action selection.  In Psychology of Learning and Motivation, volume 74, pages 195–232. Elsevier.
  * Lengyel and Dayan, (2007) Lengyel, M. and Dayan, P. (2007).  Hippocampal contributions to control: the third way.  Advances in neural information processing systems, 20.
  * Levine, (2018) Levine, S. (2018).  Reinforcement learning and control as probabilistic inference: Tutorial and review.  arXiv preprint arXiv:1805.00909.
  * Levine et al., (2020) Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020).  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  arXiv preprint arXiv:2005.01643.
  * Li et al., (2022) Li, C., Kreiman, G., and Ramanathan, S. (2022).  Integrating artificial and biological neural networks to improve animal task performance using deep reinforcement learning.  bioRxiv, pages 2022–09.
  * Lillicrap et al., (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015).  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971.
  * Loukola et al., (2017) Loukola, O. J., Solvi, C., Coscos, L., and Chittka, L. (2017).  Bumblebees show cognitive flexibility by improving on an observed complex behavior.  Science, 355(6327):833–836.
  * Lowet et al., (2020) Lowet, A. S., Zheng, Q., Matias, S., Drugowitsch, J., and Uchida, N. (2020).  Distributional reinforcement learning in the brain.  Trends in neurosciences, 43(12):980–997.
  * Makino, (2023) Makino, H. (2023).  Arithmetic value representation for hierarchical behavior composition.  Nature Neuroscience, 26(1):140–149.
  * Mathis et al., (2018) Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., and Bethge, M. (2018).  Deeplabcut: markerless pose estimation of user-defined body parts with deep learning.  Nature neuroscience, 21(9):1281–1289.
  * Mattar and Daw, (2018) Mattar, M. G. and Daw, N. D. (2018).  Prioritized memory access explains planning and hippocampal replay.  Nature neuroscience, 21(11):1609–1617.
  * Mehta et al., (2000) Mehta, M. R., Quirk, M. C., and Wilson, M. A. (2000).  Experience-dependent asymmetric shape of hippocampal receptive fields.  Neuron, 25(3):707–715.
  * Merel et al., (2019) Merel, J., Aldarondo, D., Marshall, J., Tassa, Y., Wayne, G., and Ölveczky, B. (2019).  Deep neuroethology of a virtual rodent.  arXiv preprint arXiv:1911.09451.
  * Miller et al., (2017) Miller, K. J., Botvinick, M. M., and Brody, C. D. (2017).  Dorsal hippocampus contributes to model-based planning.  Nature neuroscience, 20(9):1269–1276.
  * (55) Mizes, K. G., Lindsey, J., Escola, G. S., and Ölveczky, B. P. (2023a).  Dissociating the contributions of sensorimotor striatum to automatic and visually guided motor sequences.  Nature Neuroscience, pages 1–14.
  * (56) Mizes, K. G., Lindsey, J., Escola, G. S., and Olveczky, B. P. (2023b).  Motor cortex is required for flexible but not automatic motor sequences.  bioRxiv, pages 2023–09.
  * Mnih et al., (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013).  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602.
  * Momennejad et al., (2017) Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., and Gershman, S. J. (2017).  The successor representation in human reinforcement learning.  Nature human behaviour, 1(9):680–692.
  * Morimura et al., (2010) Morimura, T., Sugiyama, M., Kashima, H., Hachiya, H., and Tanaka, T. (2010).  Nonparametric return distribution approximation for reinforcement learning.  In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 799–806.
  * Morimura et al., (2012) Morimura, T., Sugiyama, M., Kashima, H., Hachiya, H., and Tanaka, T. (2012).  Parametric return density estimation for reinforcement learning.  arXiv preprint arXiv:1203.3497.
  * Morris et al., (2006) Morris, G., Nevet, A., Arkadir, D., Vaadia, E., and Bergman, H. (2006).  Midbrain dopamine neurons encode decisions for future action.  Nature neuroscience, 9(8):1057–1063.
  * Niv, (2009) Niv, Y. (2009).  Reinforcement learning in the brain.  Journal of Mathematical Psychology, 53(3):139–154.
  * Nowé et al., (2012) Nowé, A., Vrancx, P., and De Hauwere, Y.-M. (2012).  Game theory and multi-agent reinforcement learning.  Reinforcement Learning: State-of-the-Art, pages 441–470.
  * Olds and Milner, (1954) Olds, J. and Milner, P. (1954).  Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain.  Journal of comparative and physiological psychology, 47(6):419.
  * Pachitariu et al., (2016) Pachitariu, M., Stringer, C., Schröder, S., Dipoppa, M., Rossi, L. F., Carandini, M., and Harris, K. D. (2016).  Suite2p: beyond 10,000 neurons with standard two-photon microscopy.  BioRxiv, page 061507.
  * Pan et al., (2017) Pan, Y., Cheng, C.-A., Saigol, K., Lee, K., Yan, X., Theodorou, E., and Boots, B. (2017).  Agile autonomous driving using end-to-end deep imitation learning.  arXiv preprint arXiv:1709.07174.
  * Pateria et al., (2021) Pateria, S., Subagdja, B., Tan, A.-h., and Quek, C. (2021).  Hierarchical reinforcement learning: A comprehensive survey.  ACM Computing Surveys (CSUR), 54(5):1–35.
  * Piray and Daw, (2021) Piray, P. and Daw, N. D. (2021).  Linear reinforcement learning in planning, grid fields, and cognitive control.  Nature communications, 12(1):1–20.
  * Rao and Ballard, (1999) Rao, R. P. and Ballard, D. H. (1999).  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.  Nature neuroscience, 2(1):79–87.
  * Ritter et al., (2018) Ritter, S., Wang, J., Kurth-Nelson, Z., Jayakumar, S., Blundell, C., Pascanu, R., and Botvinick, M. (2018).  Been there, done that: Meta-learning with episodic recall.  In International conference on machine learning, pages 4354–4363. PMLR.
  * Roesch et al., (2007) Roesch, M. R., Calu, D. J., and Schoenbaum, G. (2007).  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards.  Nature neuroscience, 10(12):1615–1624.
  * Rowland et al., (2019) Rowland, M., Dadashi, R., Kumar, S., Munos, R., Bellemare, M. G., and Dabney, W. (2019).  Statistics and samples in distributional reinforcement learning.  In International Conference on Machine Learning, pages 5528–5536. PMLR.
  * Schaul et al., (2015) Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015).  Prioritized experience replay.  arXiv preprint arXiv:1511.05952.
  * Schrittwieser et al., (2020) Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020).  Mastering atari, go, chess and shogi by planning with a learned model.  Nature, 588(7839):604–609.
  * Schultz et al., (1997) Schultz, W., Dayan, P., and Montague, P. R. (1997).  A neural substrate of prediction and reward.  Science, 275(5306):1593–1599.
  * Silver et al., (2018) Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018).  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144.
  * Solway and Botvinick, (2012) Solway, A. and Botvinick, M. M. (2012).  Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates.  Psychological review, 119(1):120.
  * Stachenfeld et al., (2017) Stachenfeld, K. L., Botvinick, M. M., and Gershman, S. J. (2017).  The hippocampus as a predictive map.  Nature neuroscience, 20(11):1643–1653.
  * Steinmetz et al., (2021) Steinmetz, N. A., Aydin, C., Lebedeva, A., Okun, M., Pachitariu, M., Bauza, M., Beau, M., Bhagat, J., Böhm, C., Broux, M., et al. (2021).  Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings.  Science, 372(6539):eabf4588.
  * Sutton, (1991) Sutton, R. S. (1991).  Dyna, an integrated architecture for learning, planning, and reacting.  ACM Sigart Bulletin, 2(4):160–163.
  * Sutton and Barto, (2018) Sutton, R. S. and Barto, A. G. (2018).  Reinforcement learning: An introduction.  MIT press.
  * Sutton et al., (1999) Sutton, R. S., Precup, D., and Singh, S. (1999).  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  Artificial intelligence, 112(1-2):181–211.
  * Takahashi et al., (2008) Takahashi, Y., Schoenbaum, G., and Niv, Y. (2008).  Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model.  Frontiers in neuroscience, 2:282.
  * Tano et al., (2020) Tano, P., Dayan, P., and Pouget, A. (2020).  A local temporal difference code for distributional reinforcement learning.  Advances in neural information processing systems, 33:13662–13673.
  * Todorov, (2006) Todorov, E. (2006).  Linearly-solvable markov decision problems.  Advances in neural information processing systems, 19.
  * Todorov, (2009) Todorov, E. (2009).  Efficient computation of optimal actions.  Proceedings of the national academy of sciences, 106(28):11478–11483.
  * Tsai et al., (2009) Tsai, H.-C., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., De Lecea, L., and Deisseroth, K. (2009).  Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning.  Science, 324(5930):1080–1084.
  * Van Hasselt et al., (2016) Van Hasselt, H., Guez, A., and Silver, D. (2016).  Deep reinforcement learning with double q-learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 30.
  * Vértes and Sahani, (2019) Vértes, E. and Sahani, M. (2019).  A neurally plausible model learns successor representations in partially observable environments.  Advances in Neural Information Processing Systems, 32.
  * Vikbladh et al., (2019) Vikbladh, O. M., Meager, M. R., King, J., Blackmon, K., Devinsky, O., Shohamy, D., Burgess, N., and Daw, N. D. (2019).  Hippocampal contributions to model-based planning and spatial memory.  Neuron, 102(3):683–693.
  * Vinyals et al., (2019) Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019).  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  Nature, 575(7782):350–354.
  * Vul et al., (2014) Vul, E., Goodman, N., Griffiths, T. L., and Tenenbaum, J. B. (2014).  One and done? optimal decisions from very few samples.  Cognitive science, 38(4):599–637.
  * Wang et al., (2018) Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., and Botvinick, M. (2018).  Prefrontal cortex as a meta-reinforcement learning system.  Nature neuroscience, 21(6):860–868.
  * Wang et al., (2016) Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. (2016).  Learning to reinforcement learn.  arXiv preprint arXiv:1611.05763.
  * Watkins and Dayan, (1992) Watkins, C. J. and Dayan, P. (1992).  Q-learning.  Machine learning, 8:279–292.
  * Watkins, (1989) Watkins, C. J. C. H. (1989).  Learning from delayed rewards.
  * Whittington et al., (2020) Whittington, J. C., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., and Behrens, T. E. (2020).  The Tolman-Eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation.  Cell, 183(5):1249–1263.
  * Williams, (1992) Williams, R. J. (1992).  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine learning, 8:229–256.
  * Wurman et al., (2022) Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlic, A., Eckert, F., Fuchs, F., et al. (2022).  Outracing champion Gran Turismo drivers with deep reinforcement learning.  Nature, 602(7896):223–228.
  * Yin et al., (2004) Yin, H. H., Knowlton, B. J., and Balleine, B. W. (2004).  Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning.  European journal of neuroscience, 19(1):181–189.
  * Yin et al., (2005) Yin, H. H., Ostlund, S. B., Knowlton, B. J., and Balleine, B. W. (2005).  The role of the dorsomedial striatum in instrumental conditioning.  European Journal of Neuroscience, 22(2):513–523.
  * Zintgraf et al., (2019) Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., and Whiteson, S. (2019).  VariBAD: A very good method for bayes-adaptive deep RL via meta-learning.  arXiv preprint arXiv:1910.08348.

##  9 Additional topics of interest

While we have tried to provide a fairly comprehensive overview of topics in
reinforcement learning of interest to neuroscience, there are naturally many
interesting areas that we have had to omit. Here we provide a brief
description of some of these together with pointers to relevant literature for
those who are interested in exploring them further.

###  9.1 Auxiliary losses

A key challenge in deep reinforcement learning is that of ‘representation
learning’ (Botvinick et al., , 2020). In particular, because rewards are
generally sparse, it can be difficult to learn the full processing pipeline
from raw inputs to a latent representation conducive to action selection on
the basis of rewards alone. It is therefore common in the machine learning
literature to introduce additional ‘auxiliary costs’ to the RL utility
function, which use a richer data source to drive learning in the network. A
common example is to include predictive losses that require the agent to
predict the next observation from the current state and action (Jaderberg et
al., , 2016; Zintgraf et al., , 2019). Such predictive losses have close
parallels in neuroscience, where it has been suggested that predictive
learning could result in many of the representations observed in biological
circuits (Rao and Ballard, , 1999; Stachenfeld et al., , 2017; Whittington et
al., , 2020; Blanco-Pozo et al., , 2021) and serve as a foundation for model-
based planning (Jensen et al., , 2023). This suggests a potentially important
interaction between self-supervised representation learning and reward-driven
reinforcement learning in biological circuits.

###  9.2 Hierarchical reinforcement learning

So far, we have considered a simple environment consisting of states and
actions, and all planning and decision making has taken place in the space of
action primitives. However, when planning over longer horizons, it can be
necessary to break down the overall policy into a series of sub-goals, sub-
policies, or ‘skills’ (Sutton et al., , 1999; Pateria et al., , 2021). This is
the topic of hierarhical reinforcement learning (HRL) and ‘options’, where an
agent learns a high-level policy over policies that can themselves be
specified in terms of primitive actions or even lower-level policies. Such HRL
has been found to explain features of human behaviour (Eckstein and Collins, ,
2020; Botvinick, , 2008; Botvinick et al., , 2009) and remains an area of
substantial interest in the neuroscience literature.

###  9.3 Off-policy & offline reinforcement learning

In most of the work considered in this paper, the experience used to train the
RL agents has been sampled from the policy of the agent itself. Indeed this is
required for the gradients to be unbiased in the policy-gradient setting.
However, an area of substantial interest is that of offline reinforcement
learning, where the agent is trained from scratch on the basis of pre-
collected experience (Levine et al., , 2020). This is particularly important
in cases where online data collection is expensive or too risky but large-
scale datasets exist, such as in many healthcare settings. Off-policy
reinforcement learning is the related problem of learning from a combination
of online data and pre-generated data, possibly from a ‘stale’ version of the
current agent. The off-policy setting is especially relevant to biology, where
data collection is expensive and we therefore wish to make maximum use of
existing data. This can e.g. be achieved through experience replay, which can
be prioritized (instead of sampled at random) to maximize future reward and
minimize temporal opportunity costs (Mattar and Daw, , 2018; Agrawal et al., ,
2022; Schaul et al., , 2015).

###  9.4 Imitation learning

Related to the problem of offline reinforcement learning is that of _imitation
learning_ , where we also learn from pre-collected data. However, in contrast
to offline RL where we make no assumption about the quality of the policy used
to collect the data, imitation learning assumes that the data has been
collected by an ‘expert’ we wish to imitate (Levine et al., , 2020). This is
useful in cases where a large amount of expert data is available, such as the
case of autonomous driving (Pan et al., , 2017). Imitation learning is clearly
important during early development in biological organisms, where we learn
from observing the individuals around us. Indeed, such imitation learning is a
hallmark not just of humans but has also been demonstrated in organisms as
‘simple’ as the bumblebee (Loukola et al., , 2017).

###  9.5 Linear reinforcement learning

As we have seen in most of this tutorial, reinforcement learning is generally
difficult and requires iterative algorithms that often scale poorly with the
problem size. However, there are settings where we can simplify the problem to
the point where it becomes analytically tractable in an approach known as
‘linear reinforcement learning’ (Todorov, , 2006, 2009). This is similar to
the SR approach, where we saw that the value function reduces to a linear
function of the reward-per-state. Similar to how the SR matrix can be seen as
describing the dynamics of some ‘base policy’, we also define a base policy in
linear RL and compute a ‘control cost’ as the KL divergence between transition
dynamics with and without our controller:

$\mathcal{L}_{ctrl}(s)=KL\left[u(s^{\prime}|s)||p(s^{\prime}|s)\right],$ (57)

where $p(s^{\prime}|s)$ are the prior transition dynamics and
$u(s^{\prime}|s)$ are the controlled transition dynamics marginalized over the
policy. For $\mathcal{L}_{ctrl}(s)$ to be well-defined, we require
$u(s^{\prime}|s)=0$ whenever $p(s^{\prime}|s)=0$, which prevents impossible
transitions even under our flexible controller. When subtracting this loss
from the RL objective, the resulting utility turns out to be convex in $u$ and
can therefore be solved efficiently for the controller, which implicitly
specifies the policy. This approach has recently been used as an explicit
model of biological decision making (Piray and Daw, , 2021). It also has close
parallels to learning and planning as inference (Levine, , 2018; Solway and
Botvinick, , 2012; Botvinick and Toussaint, , 2012) and to RL with information
bottlenecks (Lai and Gershman, , 2021). Both of these families of approaches
also involve reinforcement learning with a KL-regularized reward function, and
they have also been used as models of biological decision making.

###  9.6 Successor features

In Section 6, we saw that the successor representation can be used as the
basis of decision making that flexibly adapts to environments with changing
reward structures. However, we developed this framework only in the tabular
setting despite extending TD-learning and Q-learning to the ‘deep RL’ setting
with function approximation. This leaves open the question of whether a
similar generalization of the SR exists. This turns out to be the case and is
known as ‘successor features’ (SF; Barreto et al., , 2017), where the expected
future observation of a given feature of the environment is used in place of
the expected future state occupancy. Successor features have also been shown
to have a biologically plausible implementation that facilitates learning and
generalization in noisy and partially observable environments (Vértes and
Sahani, , 2019).

###  9.7 Multi-agent reinforcement learning

We have only considered the case of single agents interacting with a black-box
environment. However, in many cases, multiple agents are simultaneously
interacting with each other and the environment around them (Gronauer and
Diepold, , 2022). This means that, from the point of view of a single agent,
the other agents are part of its environment. In such settings, there are
interesting learning dynamics beyond the scope of the present tutorial, but
which are covered in detail by e.g. Gronauer and Diepold,  (2022), and which
are also of substantial interest in game theory (Nowé et al., , 2012). In some
cases, a whole group of agents may be working together to maximize a single
joint reward function – as is the case for members of a single sports team.
Interestingly, the learning of many individual neurons in the brain from a
single common reinforcing signal (such as dopamine) can be modelled as such a
multi-agent reinforcement learning problem (Sutton and Barto, , 2018). If the
‘agents’ (or neurons) are assumed to have Bernoulli-logistic outputs,
Williams,  (1992) shows that the independent learning of individual agents
from the global reward signal leads to the implementation of a policy gradient
algorithm at the population level (Sutton and Barto, , 2018).