Bayesian Modelling
of
Visual Discrimination Learning in Mice

Pouya Baniasadi, PhD

Department of Physiology, Development and Neuroscience

UNIVERSITY OF CAMBRIDGE

August 2020

This project report is written in partial fulfilment of the requirement for
the
Master of Philosophy in Basic and Translational Neuroscience

Supervised by
---
| Dr. Jasper Poort
---
|  | Prof. Máté Lengyel
---
| Selective Vision Laboratory
---
|  | Computational and Biological
---
Learning Laboratory
| Department of Psychology
---
|  | Department of Engineering
---

## Dedication

_For my parents Mahin and Ghasem, who taught me about pursuing dreams,
for their endless love, support and sacrifices_

## Declaration

This report describes work carried out at Cambridge University from Jan 2020
to Jul 2020 under the supervision of Dr Jasper Poort (Selective Vision
Laboratory at the Department of Psychology) and Prof. Máté Lengyel
(Computational and Biological Learning Lab at the Department of Engineering)
as a part of the MPhil program in Basic and Translational Neuroscience. I
confirm that the material in this report is not copied from any published
material, nor is it a paraphrase or abstract of any published material unless
it is identified as such and a full source reference is given. I confirm that,
other than where indicated above, this document is my own work.

Pouya Baniasadi

August 2020

## Abstract

The brain constantly turns large flows of sensory information into selective
representations of the environment. It, therefore, needs to learn to process
those sensory inputs that are most relevant for behaviour. It is not well
understood how learning changes neural circuits in visual and decision-making
brain areas to adjust and improve its visually guided decision-making. To
address this question, head-fixed mice were trained to move through virtual
reality environments and learn visual discrimination while neural activity was
recorded with two-photon calcium imaging. Previously, descriptive models of
neuronal activity were fitted to the data, which was used to compare the
activity of excitatory and different inhibitory cell types. However, the
previous models did not take the internal representations and learning
dynamics into account. Here, I present a framework to infer a model of
internal representations that are used to generate the behaviour during the
task. We model the learning process from untrained mice to trained mice within
the normative framework of the ideal Bayesian observer and provide a Markov
model for generating the movement and licking. The framework provides a space
of models where a range of hypotheses about the internal representations could
be compared for a given data set.

###### Contents

  1. Declaration
  2. Abstract
  3. 1 Introduction
    1. 1.1 Mathematical preliminaries
  4. 2 The experiment
    1. 2.1 Experimental setup
    2. 2.2 Behavioral data and observations
  5. 3 Behavioral model part 1: internal representations
    1. 3.1 Structure of spatial states
    2. 3.2 Space of models for spatial states
    3. 3.3 Bayesian learning model
      1. 3.3.1 Learning reward probability within a state
      2. 3.3.2 Learning state transitions
  6. 4 Behavioral model part 2: the generative model
    1. 4.1 Spatial state parameter $\tilde{\lambda_{k}}$: licking rate
    2. 4.2 Parameter $\tilde{\nu_{k}}$: target speed within the current spatial state
    3. 4.3 Generative model of licking and speed
    4. 4.4 Estimation of model parameters
  7. 5 Discussion
    1. 5.1 Limitations
    2. 5.2 Implications

##  Chapter 1 Introduction

Learning modifies neural representations of behaviourally relevant
information. While changes in response selectivity to behaviourally relevant
stimuli have been observed in many studies across different species (Yang &
Maunsell, 2004; Yan et al., 2014; Poort et al., 2015). There has been growing
evidence that different cell types, classified using molecular and cellular
properties (Kepecs & Fishell, 2014), have specific roles in learning (Khan et
al., 2018; Fishell & Kepecs, 2019). However, the nature of these changes and
how they relate to sensory coding is not well understood (Yap & Greenberg,
2018).

Probabilistic models of behavioural learning are an important approach to link
the changes in neural representations to internal representation of the
environment and decision-making (Fiser et al., 2010; Berkes et al., 2011;
Heeger, 2017). Given the non-deterministic nature of events in the real world,
human and animal learning must involve at least some internal representations
of the uncertainties in the environment (Barlow et al., 1961). There has been
an extensive body of research on how the nervous system represents uncertainly
about the environment (Pouget et al., 2003; Beck et al., 2008; Fiser et al.,
2010; Kriegeskorte & Douglas, 2018).

Bayesian learning theory provides a normative framework of learning that
represents uncertainty in probabilistic outcomes(Bishop, 2006). In particular,
the ideal observer analysis uses the Bayesian learning theory for achieving
optimal learning performance in a given task (Geisler, 2003, 2011). Learning
can be conceptualised as the incorporation of sensory information to update
and improve performance on a given task. the ideal observer performs at the
theoretical limits of information processing to update their beliefs. However,
it is important to note that optimality in this context refers to the optimal
incorporation of information, which is not equivalent to achieving the optimal
solution in all trials. While the nervous system may or may not have
representations similar to an ideal observer, the ideal observer analysis
provides a systematic framework to formulate hypotheses about the internal
representations and learning dynamics (Maloney & Mamassian, 2009; Orbán et
al., 2008).

In this thesis, we describe a Bayesian learning model using the framework of
ideal observer learning. Our goal is to develop a model of internal
representations of reward and space that are used for learning and adjusting
behaviour in the visual discrimination task. This model will allow us in
future work to relate the neuronal activity measurements (Poort et al., 2015;
Khan et al., 2018) to the internal representations that guide behaviour. We
continue this chapter with a brief overview of the basic mathematical ideas
used to develop the model. Then, in Chapter 2, we explain the experimental
setup and describe the behavioural data. A space of models (for the structure
of Markov models) is introduced in Chapter 3 which defines the internal
representations of reward and state transitions. Then, a Bayesian model of
learning reward probabilities and state transitions is described that uses the
ideal observer framework. In Chapter 4, we introduce a generative Markov model
that uses internal representations to generate behaviour. We also discuss the
use of maximum likelihood estimation to estimate the model parameters.
Finally, in Chapter 5, we discuss the potential applications and limitations
of the model and set out a path for the continuation of the research.

###  1.1 Mathematical preliminaries

In this section, I briefly introduce the concepts that provide the
mathematical foundation of the Behavioral model.

##### _Markov chain model_

A system has the _Markov property_ if the predictions about future events only
require the knowledge of the system’s present state. In other words, given the
present state of the system, future events are conditionally independent of
past events. A _Markov chain_ is a stochastic model of a sequence of events
with the Markov property.

Let $S={s_{1},s_{2},...,s_{r}}$ be a set of _states_ for a Markov chain. The
process starts in one of these states and moves sequentially from one state to
another. Each move is called a _step_. Let $X_{n}$ be the current step. We
denote by $p_{ij}=P(X_{n+1}=s_{j}|X_{n}=s_{i})$, the _transition probability_
of visiting state $s_{j}$ after visiting $s_{i}$. Note that by Markov
property, given $X_{n}$, $X_{n+1}$ is conditionally independent of the past
states. A transition from $s_{i}$ to $s_{j}$ can be represented as a directed
edge $(s_{i},s_{j})$ with a corresponding transition probability $p_{ij}$. The
sum of transition probabilities of the outgoing edges from a state should add
up to 1. Figure 1.1 illustrates a Markov chain with 4 states and transition
probabilities.

Figure 1.1: Movement in a corridor simulated in the VR environment.

Let $T=[p_{ij}]$ be the transition probability matrix for the Markov chain and
let $\bm{u}$ be the probability vector which represents the starting
distribution (i.e., $X_{k}\sim\bf{u}$). Then the probability that the chain is
in state $s_{i}$ after $m$ steps is the $i$-th entry in the vector
$\bm{u}(m)\coloneqq\bm{u}\ T^{m}$. That is,

$\displaystyle P(X_{k+m}|X_{k})=\bm{u}^{(i)}(m),\ \ \ \ \ \ \ \mbox{ where
}\bm{u}(m)=\bm{u}T^{m}.$ (1.1)

##### _Bayesian learning_

The probability of an event $A$ is denoted by $P(A)$. Consider another event
$B$ and its corresponding probability $P(B)$. The conditional probability
$P(A|B)$ is the conditional probability of $A$ given $B$. Bayes Theorem states
that

$P(A|B)=\frac{P(B|A)~{}P(A)}{P(B)}$

Consider a system that generates data and a space of possible models for
describing the behaviour of the system. The probability distribution over the
space of models $P(Model)$ represents the _prior_ knowledge about the system.
Suppose that a set of data $D$ is observed from the system. Then
$P(D~{}|~{}Model)$ is called the _likelihood_ and $P(Data)$ is called the
_model evidence_ or _marginal likelihood_. The _posterior distribution_ over
the models $P(Model~{}|~{}D)$ represents our beliefs about the system after
observing the data $D$. Bayes rule provides a principled way of updating our
beliefs about the system after observing data. Formally,

$P(Model~{}|~{}Data)=\frac{P(Data~{}|~{}Model)~{}P(Model)}{P(Data)}.$ (1.2)

##### _Dirichlet distribution learning of categorical probability values_

Consider a random variable which can take on $K$ possible categories. The
_categorical distribution_ is a discrete probability distribution for the
random variable, where the probability of each category is separately
specified. The categorical distribution is a generalisation of the Bernoulli
distribution for a discrete variable with more than two outcomes, such as the
probability of outcomes for a 6-sided die. It is also a special case of the
_multinomial distribution_ where the number of trials in one.

If the probabilities of each outcome for a categorical distribution are
unknown, using Bayesian learning, we can update prior probability
distributions of probability values. The _Dirichlet distribution_ is a
conjugate before the multinomial (and categorical) distribution, meaning
starting with a Dirichlet prior and multinomial likelihood, the resulting
posterior is also a Dirichlet distribution. The probability mass function for
the Dirichlet distribution $Dir_{K}(\bm{\alpha})$ with $K$ categories is

$f(\bm{p}|\bm{\alpha})=\frac{1}{B(\bm{\alpha})}\prod_{i=1}^{K}p_{i}^{\alpha^{(i)}-1},$
(1.3)

where $\bm{\alpha}=(\alpha^{(1)},\ldots,\alpha^{(K)})$ is the vector of
parameters. Furthermore,

${B(\bm{\alpha})}=\prod_{i=1}^{K}\frac{\Gamma(\alpha^{(i)})}{\Gamma\big{(}\sum^{K}_{i=1}\alpha^{(i)}\big{)}}$

where for positive real number $n$,

$\Gamma(n)=\int_{0}^{\infty}{x^{n-1}e^{-x}dx}.$

For integer values, $\Gamma(n)=n!\ $.

To learn probabilities for a categorical distribution, given a prior
distribution $Dir_{K}(\bm{\alpha})$ over the probability vector
$\bm{p}=(p_{1},\ldots,p_{K})$, and data $\bm{c}=\\{c_{1},\ldots,c_{k}\\}$
representing the number of observation Dirichlet category, the posterior
distribution is

$\displaystyle P(\bm{p}|\bm{x})$ $\displaystyle=(\bm{p}|\bm{x}+\bm{\alpha})$
$\displaystyle\bm{p}|\bm{x}$ $\displaystyle\sim Dir_{K}(\bm{x}+\bm{\alpha})$
(1.4)

Finally, the Beta distribution is a special case of the Dirichlet distribution
where the outcomes are binary (true or false). To distinguish this special
case, we may use the notation $Beta(\beta^{(1)},\beta^{(2)})\equiv
Dir_{2}(\bm{\alpha})$ where $\bm{\alpha}=\\{\beta^{(1)},\beta^{(2)}\\}$.

##  Chapter 2 The experiment

In this chapter, I describe the experimental setup in Khan et al. (2018) and
Poort et al. (2015) for which we have developed a behavioural model in the
later chapters. A summary of previous findings and a description of the
behavioural data accompanied by figures are also included.

###  2.1 Experimental setup

The experimental setup involves the placement of the mouse on a cylindrical
treadmill where its head is fixed to enable imaging of neural activity. The
mouse can move forward (and backward). In front of the mouse, a screen is
shown to the animal where visual feedback connected to the movement can
simulate the movement of the subject in an environment. By controlling the
setup of the space and visual stimulus while allowing imaging, the VR setup
has been extensively used for studying the visual cortex and hippocampus in
mice in recent years (Harvey et al., 2009; Dombeck et al., 2010; Khan et al.,
2018; Poort et al., 2015; Saleem et al., 2018). Figure 2.1 illustrates the VR
setup.

Figure 2.1: Movement in a corridor simulated in the VR environment.

##### _Specifics of the corridor space and reward administration_

We specifically consider the experimental setup described in Khan et al.
(2018); Poort et al. (2015). In these two studies, the activity of populations
of neurons in V1 was measured with two-photon calcium imaging Chen et al.
(2013) during a visual discrimination task in a virtual reality (VR)
environment. Head-fixed mice ran through a simulated corridor where different
types of visual stimuli were displayed on the walls. Three types of wall
patterns characterise the different corridors. In the _grey corridor_ a short
stretch of circle patterns followed by grey walls for a random distance,
before the pattern on the walls abruptly changes to one of the _grating
corridors_. The grating corridors either displayed _vertical gratings_
(illustrated in Figure 2.1) or _angled gratings_ for a fixed length (60 VR
length units), before the grey corridor. An illustration of the corridor space
is displayed in Figure 2.2.

Figure 2.2: Illustration of the corridor space.

A milk dispenser was placed in front of the mouse to administer rewards. Mice
received a reward for licking the dispenser in a reward zone starting halfway
in the vertical grating corridor and halfway for around 10 VR-length units. If
the mouse licked the dispenser in the reward zone, it would trigger the
opening of the reward valve and a drop of soy milk would appear at the
dispenser. No punishment was given for licking in the corridors with grey and
angled grating walls. All mice learnt to discriminate the two stimuli,
starting at the chance performance (behavioural $d^{\prime}$ close to zero)
and reaching the threshold criterion of $d^{\prime}>2.0$ within 5-9 days.

##### summary of previous findings

The motivation behind developing a behavioural model is to take advantage of
the behavioural data for the future analysis of experiments similar to Khan et
al. (2018). A summary of results in Khan et al. (2018) is as follows. After
learning the visual discrimination task, neurons showed increased stimulus
selectivity for the angled and vertical gratings. Interestingly, this effect
depended on the cell types. In particular, stimulus selectivity for
populations of pyramidal cells (PYR) along with parvalbumin (PV), somatostatin
(SOM), and vasoactive intestinal peptide-expressing (VIP) inhibitory
interneurons in layer 2/3 (L2/3) of the primary visual cortex (V1) were
compared. Selectivity was increased for PYR and PV cells. PV neurons became as
selective as the PYR cells, and showed changes in functional interactions,
particularly with PYR cells. On the other hand, SOM neurons became
decorrelated from the network and PYR–SOM coupling before learning predicted
selectivity increases in individual PYR cells. While SOM inhibition seemed to
gate changes in selectivity, PV cells provided strong stimulus selective
inhibition after learning. A multivariate autoregressive linear model (MVAR
model) fitted the activity of the neurons, and further supported the
statistical analysis results. However, the MVAR model arguably neglects
potentially important information in the behavioural data. Even though speed
is taken into account, its contribution to the behaviour of the MVAR model is
negligible. Accordingly, one of the primary motivations of the behavioural
model proposed in this report is potential improvements in the (MVAR model).
This is discussed in more detail in Chapter 5.

###  2.2 Behavioral data and observations

Behavioural data were collected during the experiment. The distance travelled
from the onset of the current corridor and the corridor type (determined by
the wall patterns) is continuously recorded. The observed variables of spatial
location and visual stimuli at each time are marked by a pair
$(x,y)\in(xLoc\times Cor)$, where $x$ is the distance travelled from the onset
of the current corridor pattern, and $y$ is the corridor pattern. Set
$xLoc=[0,max(x)]$ is an interval from $0$ to the maximal length of an interval
$max(x)$ and set $Cor=\\{grey,vertical,angled\\}$ is the set of corridor
types. The speed of the subject at each time is also recorded. A list of
licking times and valve opening times (indicating reward administration) is
also given by the data.

For the generative behavioural model in Chapter 4, we discretize the data into
time intervals of $\Delta\tau$ seconds, each identified by an index
$t\in\\{1,2,\ldots,N\\}$. The value of $\Delta\tau$ determines the time
resolution of the behavioural data. Since the imaging data of Khan et al.
(2018) is taken in $\frac{1}{8}$ second intervals, time resolutions lower than
$\frac{1}{8}$ seconds are not useful. Higher time resolutions may be desirable
because they will decrease the computational cost of the analysis, but the
cost of losing time resolution must be discussed. However, unless explicitly
discussed, we can assume $\Delta\tau=\frac{1}{8}$ for the data analysis. Table
2.1 describes the notation used to describe the data. Note that some of the
records are behavioural, while others specify the values that are observed by
the subject.

Table 2.1: Behavioral and observational records for $t\in\\{1,2\ldots,N\\}$.  Data | Type | Description
---|---|---
$x_{t}$ | Observation |  |  $x_{t}$ is the true value of distance from the onset of the current
---
corridor at time step $t$.
$y_{t}$ | Observation |  |  $y_{t}\in Cor=\\{grey,vertical,angled\\}$ is the true value of the
---
corridor type, which determines the visual stimuli at time
step $t$.
$o_{t}$ | Observation |  |  $o_{t}$ is a binary value for whether the reward valve has opened
---
during the time step.
$v_{t}$ | Behavior |  | Speed (average) at time step $t$
---
$l_{t}$ | Behavior | Number of licks at time step $t$

##### Instance of data visualisations

The Figures below are instances of behavioural data visualizations from the
experimental data. Figures 2.4 and 2.4 (Poort et al., 2015) illustrate the
licking behaviour at different positions in different corridors, and Figures
2.6 (Poort et al., 2015) and 2.6 (Khan et al., 2018) give a colour map of
speed at the different positions in the different corridors. For all Figures,
the horizontal axis represents the position concerning the onset of the
grating corridor 111note that for the grey corridor this is obtained by
shifting $x_{t}$ by the length of the grey corridor, and the vertical axis is
the trial index. Higher trial numbers are later. The black or red labels are
data labels the ls associated with the experimental sessions.

The following observations about the licking behaviour have influenced
parameter definitions and assumptions about prior beliefs of the animal in
Chapter 3. These observations are consistent among all subjects.

  * reward association prior: The mice do not know reward associations before the reward. However, the mice know that moving forward and licking the dispenser may lead to a reward. Initially, the licking behaviour is frequent to explore the space and discover reward associations. A uniformly random prior for reward probability may be appropriate.

  * Change of visual discrimination: The behaviour of the mice in the grating area and the grey area starts to diverge immediately, and the behaviour of the mouse in angled and vertical grating corridors seems to be similar at first; the differences of licking behaviour seem to be only after the reward is present in the vertical grating corridor. The dissociation of the award from the angled grating is realised substantially later than the dissociation of the reward from the grey area. It seems that at different points in the trial, the set of visually discriminated stimuli is different.

  * Location is also taken into account As the learning progresses, the licking concentrates close to the reward zone. It seems that the mice associate a spatial region, characterised by both visual stimuli and spatial positioning, with the reward area.

Figure 2.3: Lick locations for M27. See the figure descriptions below.

Figure 2.4: Lick locations for M31 in all trials. The horizontal axis
represents the location in a corridor, with 0 being set at the onset of a
grating corridor. Negative values are in the grey corridors and positive
values are in the grating corridors. The licking locations are marked by
coloured points. Red dots represent licking within 1 length unit before a
valve opening, and yellow indicates the licking after the opening of the
reward valve, in a grating corridor. All other lick locations are marked in
black. The trial number on the vertical axis shows the sequential order of the
trials in each plot. The right plot shows all trials, where each trial is
passing through one grey corridor followed by a grating corridor. The middle
and the left plots show a closer look at the vertical and angled grating
corridors. The red labels are labels for the experimental sessions.

The following observations about the speed have influenced our generative
model of speed in Chapter 4.3. These observations are consistent among all
subjects.

  * Reward association influences speed: the graphs suggest that the dissociation of reward in upcoming regions is associated with higher speed while anticipation of reward in upcoming regions is associated with reduction of speed.

  * Evidence for change in the internal model: while speed behaviour in the grey corridor diverges from the grating corridor quickly, the divergence of speed behaviour for angled grating and angled grating happen at a later point. This suggests that the mice initially correlate the grating areas with the reward, and then learn to differentiate between the grating areas to dissociate the angled grating with the reward.

  * Change of visual discrimination:: Similar to the licking behaviour, initially speed behaviour seems to discriminate between the angled and vertical gratings only after the reward is present in the vertical grating corridor. This suggests that the mice initially correlate the grating areas with reward, and then learn to discriminate between the vertical and angled grating areas.

Figure 2.5: Speed vs location for M31. See the figure descriptions below.

Figure 2.6:  Speed and licks vs location for M70. The horizontal axis
represents the location in the corridor, with 0 being set at the onset of a
grating corridor. Negative values are in the grey corridors and positive
values are in the grating corridors. The trial number on the vertical axis
shows the sequential order of the trials in each plot. The right plot shows
all trials, where each trial is passing through one grey corridor followed by
a grating corridor. The middle and the left plots show a closer look at the
vertical and angled grating corridors. The colour for each location of each
trial represents the speed of the animal at that point according to the colour
scale; warmer colours represent higher speeds and cooler colours represent
lower speeds. Note that for Figure 2.6, the speed is averaged over 5 unit
intervals due to virtual memory limits. The white points show the lock
locations for M70, and the small black star indicates a valve opening location
during a trial. The black labels are data labels associated with experimental
sessions.

##  Chapter 3 Behavioral model part 1: internal representations

The behavioural model presented here provides a framework for inferring an
internal model that can predict the animal’s behaviour at a given time. Before
getting into the specifics, consider a broad perspective on inferring a model
that generates the current behaviour by incorporating past experiences. Figure
3.1 is a graphical model of big-picture relation between the history of
animal’s observations $\mathcal{H}$, the internal model $\mathcal{M}$ that
incorporates experience into internal representations, and the observed
behaviour $\mathcal{B}$. This chapter discusses the relationship between the
history of observations and behaviorally relevant representations in the
internal model ($\mathcal{H}\to\mathcal{M}$ in the graphical model of Figure
3.1). I introduce a space of models where a range of hypotheses about the
internal model can be systematically examined. The internal representations
about reward and space are then used in the next chapter to construct a
generative model of behaviour ($\mathcal{M}\to\mathcal{B}$ in the graphical
model of Figure 3.1). Then using a systematic approach, an internal model is
inferred that best describes the data ($\mathcal{H}$ and $\mathcal{B}$).

Figure 3.1: Relation between history of experimental observations
$\mathcal{H}$, internal model $\mathcal{M}$, and behavior $\mathcal{B}$.
$\mathcal{H}$ and $\mathcal{B}$ are observed in the experimental data, but the
internal model $\mathcal{M}$ is unobserved.

By exploring and experiencing the environment, the brain uses experience to
update its beliefs (i.e., learning) about the environment using its internal
representations. In this learning model, the normative framework of Bayesian
ideal observer analysis (Geisler, 2003, 2011) is used to learn behaviorally
relevant internal representations. These include learning about the
probability of reward in different regions of the VR corridor, and
expectations about upcoming spatial regions when moving forward111The subject
can only move forward due to the experimental setup..

Model of spatial states in Section 3.1 describes how the space (VR corridor)
is divided into states corresponding to spatial segments, where the
representation of reward probability within a state only depends on the
information (history of reward outcomes) obtained at that state. The structure
of these states is a Markov chain. The space of models in Section 3.2
prescribes a range of Markov chain structures of spatial states within which a
model is selected. For given states of a model, the dynamics for learning
reward associations and state transitions are considered within the normative
framework of the Bayesian ideal observer model in Section 3.3.

###  3.1 Structure of spatial states

Animals’ observation of visual stimuli and spatial positioning is an
observation of the current $(x,y)\in\\{xLoc,Cor\\}$. Observations about reward
association at the current location $(x,y)$ may be relevant to reward
association at some other locations. It is therefore necessary to define
spatial regions where reward observations are relevant to the entire region
but explicitly irrelevant to other regions. To formalise this concept, the
objective of this section is to associate the segments of space with states
where the information about reward association is relevant to the current
state and no other state. A reasonable way to define such states is to group
areas that are spatially close by, visually similar, or both.

##### _Defining states associated with spatial segments_

Taking into account both spatial proximity and visual similarity, consider
sectioning $xLoc$ into a finite set of mutually exclusive spatial segments
each identified by a fixed $y$, and an interval $I_{x}$ for $x$ values. We
illustrate an example of spatial segmentation in Figure 3.2. Denote by $S$ a
set of states and associate each segment with only one state (note that
multiple segments may be associated with the same state). Then we say that
_the mouse is in state $s$_ if its position $(x,y)$ is inside a segment that
is associated with $s$. We associate all positions in all corridors with only
one state with the function $f:(xLoc\times Cor)\to S$. The mouse may map
locations onto states in multiple ways. By considering various ways to map
between locations and states, we can infer the mapping that best matches the
behavioural data (see 4.4).

Figure 3.2: An example of dividing the corridor space into mutually exclusive
spatial segments. Each segment is then associated with exactly one state.

##### _Spatial state transition event and structural properties_

Let $X_{k}$ be the random variable describing the $k$-th visited spatial
state, where a spatial state transition event (i.e., transition to the next
spatial step) happens when the subject _crosses the initial point of a segment
associated with a state_ 222Note that the time spent in each state is not
fixed in this Markov model.. Given the current position, the future positions
do not depend on the history of visited positions, so given $X_{k}$, state
$X_{k+1}$ is conditionally independent of $X_{n}$ for $n<k$. It follows that
the state structure as defined above satisfies the Markov property.

We assume that the spatial states are _fully observable_. In other words,
given a state structure, we assume that the subject always knows which state
is the current state. Observations of the animal may be noisy and inaccurate,
so assuming fully observable states is a simplification that may be contended
with in a more sophisticated future model. However, states are associated with
intervals of space rather than precise points in space, and they already
incorporate some approximation about the spatial awareness of the subject.

We assume that the mouse learns two things from the visual stimuli and licking
in state $s$. First, it learns the reward association in that state. Second,
it learns the transition from that state to other states. Let $r(s)$ be the
probability that licking in state $s$ leads to reward in state $s$. Also,
denote by $p_{(s,s^{\prime})}=P(X_{k+1}=s^{\prime}|X_{k}=s)$ the transition
probability of visiting any state $s^{\prime}$ after $s$. These parameters are
initially unknown to the mouse and should be learned. In Section 3.3, I
discuss a semi-normative model of learning for these parameters using the
ideal observer framework.

It is worth noting that the state transitions of the Markov chain are sparse.
To understand the sparsity of state transitions, first note that $x$ is a
positive real value, which ranges from $0$ to the maximal length of a corridor
with the same patterns, and $y$ is a discrete value with three possible
entries. From the onset of a corridor, until the onset of the next corridor,
the spatial location is a continuous function of time. Within the period
between two consecutive onsets, if a state transition happens, it can only be
to the state associated with the next interval of $x$, with the same $y$.
Moreover, when passing the onset of the next corridor, there is a discrete
change in the value of $y$, and $x=0$ at the onset of the new corridor. This
event can only be a state transition to the start of a new corridor (a state
that starts at $x=0$) so there are at most three such possible transitions. It
follows that the structure of states is a sparse Markov chain.

###  3.2 Space of models for spatial states

To define a space of models $\mathscr{M}$, we use two parameters for
identifying a model in the model space; one for the set of discriminated
patterns ($\mathcal{V}$), and one for the length of segments ($d$).

##### _Spatial model parameter $\mathcal{V}$: set of discriminated visual
stimuli_

Let $\mathcal{V}$ be the set of visual stimuli that are discriminated in the
spatial state model. The set of possible choices for $\mathcal{V}$ is
$\\{V_{1},V_{2},V_{3}\\}$ which are described below.

  * •

$V_{1}=\\{u\coloneqq\mbox{undifferentiated}\\}$, where the grey and grating
are not discriminated.

  * •

$V_{2}=\\{g\coloneqq\mbox{grey},va\coloneqq\mbox{angled or vertical
grating}\\}$, where the grey corridor is discriminated from the grating
corridors, but where angled and vertical grating corridors are not
discriminated.

  * •

$V_{3}=\\{g\coloneqq\mbox{grey},v\coloneqq\mbox{vertical},a\coloneqq\mbox{angled}\\}$,
where the grey corridor, the angled and vertical grating corridor are
discriminated.

While set $Cor$ contains the types of visual stimuli on the corridors, set
$\mathcal{V}$ refers to subjective visual discrimination (or classification)
between corridors by the mouse. Also note that the choices for set
$\mathcal{V}$ implicitly contain a mapping from $Cor$ to $\mathcal{V}$.

##### _Spatial model parameter $d$: length of states_

Denote by $d$ a value in the interval $(0,max(x)]$ for the length of spatial
segments. Value $d$ uniquely defines a sequence of intervals of $x$ values.
For example, the associated sequence of intervals to $d=30$ is
$\\{[0,30),[30,60),\ldots\\}$. Then state $s_{ij}$ is associated with the
$j$-th interval of $x$, which is $[(j-1)d~{},~{}jd)$, and $i\in C$ identifies
the visual stimuli. For example, for $\mathcal{V}=\\{g,p\\}$ and $d=30$, the
state $s_{p,2}$ refers to intervals of $x\in[30,60)$ for both the vertical and
angled grating corridors.

Figure 3.3: Nine instances of Markov chain models $M_{\mathcal{V},d}$ for
choices of $\mathcal{V}$ selected instances of $d$. For $d=x_{m}ax$, there is
only one state per and self transition event only occurs when the corridor
type changes. The length of the angled and vertically grating corridors is
exactly 60 (VR length units) in the experiment. So for $d=60$ and $d=20$,
there are exactly 1 and 3 states associated with the relevant element in
$\mathcal{V}$. Note that the figure illustrates only selected instances of the
model space $\mathscr{M}$.

##### _Model space_

Now it is possible to introduce a Markov model
$M_{\mathcal{V},d}\in\mathscr{M}$ with the set of states $S$ that are
associated with the spatial intervals induced by $\mathcal{V}$ and $d$. Since
the length of the a corridor is bounded by $max(x)$, model $M_{\mathcal{V},d}$
is a finite state Markov model. For example, $M_{V_{1},max(x)}$ and
$M_{V_{3},max(x)}$ have exactly one and three states, respectively. Figure 3.3
illustrates the states of Markov chain models $M_{\mathcal{V},D}$ for example
cases of $\mathcal{V}$ and $d$.

Parameters $\mathcal{V}$ and $d$ are free parameters that will be set during
the model selection, which will be further discussed in Section 4.4. The fit
for parameter $\mathcal{V}$, selected from $V_{1}$, $V_{2}$ or $V_{3}$, is
determined by which stimuli the animal discriminates. The true value for $d$
is the length of spatial segments where information about reward associations
and state transitions in the current segment is reasonably independent of
segments associated with other states. For the sake of simplicity, it is
assumed that $d$ is a fixed value, and it is the same across different visual
stimuli. However, relaxing this assumption is possible by having more free
parameters, for example, by introducing a free parameter of distance for each
element of $\mathcal{V}$. For example, suppose $V=V_{3}$. Then instead of a
free parameter $d$, we could use three parameters in
$D=\\{d_{g},d_{a},d_{v}\\}$ which contains one free parameter of distance for
every element of $\mathcal{V}$. In the initial implementation of the model,
one parameter $d$ is considered.

Table 3.1: Parameters for the model of spatial states

.  Parameter Type Description $\mathcal{V}$ Spatial model parameter Set of
discriminated visual stimuli on the corridors in the model
$M_{\mathcal{V},d}$; Possible options are $V_{1}=\\{u\\}$, $V_{2}=\\{g,p\\}$
and $V_{3}=\\{g,v,a\\}$. $d$ Spatial model parameter A constant length in
$(0,max(x)]$ for the length of the spatial for model $M_{\mathcal{V},d}$.

In summary, parameters $\mathcal{V}$ and $d$ for a model $M_{\mathcal{V},d}$
determine the structure of the states in the Markov chain, where for each
state the learning dynamics about reward association and state transitions is
only dependent on the observations in that state. The learning dynamics are
discussed in the next section.

###  3.3 Bayesian learning model

As first noted in Section 3.1, in any state $s$, the subject uses sensory
information to learn $r(s)$, the probability that licking in $s$ leads to the
administration of reward in $s$, or _reward probability of $s$_ for short.
Furthermore, state transition probability $p_{(s,s^{\prime})}$, which is the
probability of visiting state $s^{\prime}$ after visiting $s$, is also unknown
to the subject and it is learned. Here, we use the ideal observer framework
(Geisler, 2003) to develop a semi-normative model for learning both reward
associations and state transitions. In this section, the learning dynamics are
discussed for a given model $M\in\mathscr{M}$. Therefore, states $S$ and their
associated spatial intervals are unambiguous.

####  3.3.1 Learning reward probability within a state

Recall that reward is given to the subject immediately after the subject licks
the dispenser in the reward zone (see Section 2.1 for details of the
experimental setup). The reward is a fixed amount of milk administered via the
dispenser. We noticed that even in trained animals, licking started before the
reward zone (see example mice in Figures 2.4 and 2.4). This suggests that the
mouse associates an extended region with the reward delivery which starts
before the reward zone set by the experimenters.

##### _Reward outcome $\bm{R}_{k}$ of current spatial step $k$_

If the mouse licks the dispenser in state $s$, it collects some information
about the unknown parameter $r(s)$. If the subject does not lick the
dispenser, it obtains no information about $r(s)$. Let the random variable
$\bm{R}_{k}=(R^{(T)}_{k},R^{(F)}_{k})$ be the _reward outcome of spatial step
$k$_, where $R^{(T)}_{k}$ counts the number of positive outcomes, and
$R^{(F)}_{k}$ counts the number of negative outcomes in spatial step $k$. As a
consequence of the experimental setup, the amount of reward and the frequency
of licking in the experiment does not provide any additional information about
a reward region. Furthermore, spatial states are defined to be regions where
licking at different points within the region does not provide additional
information about the reward. Therefore, each visit to a state provides only
three possible reward outcomes:

  * •

$\bm{R}_{k}=(1,0)$ for subject licking the dispenser in spatial step $k$
followed by reward becoming available in spatial step $k$,

  * •

$\bm{R}_{k}=(0,1)$ for subject licking the dispenser in spatial step $k$
followed by no reward in spatial step $k$, and

  * •

$\bm{R}_{k}=(0,0)$ for subject not licking the dispenser in spatial step $k$.

##### _Normative model for updating internal reward representations
(Bayesian)_

Let us first discuss how an ideal observer updates its prior beliefs about
$r(s)$ after visiting state $s$ in spatial step $k$. The ideal observer
provides a theoretical upper limit of performance, given the collected data.
It is therefore a normative framework for updating the beliefs about reward
association. Let prior beliefs about $r(s)$ right before visiting spatial step
$k$ be a $Beta$ distribution

$\displaystyle Beta(\beta^{(1)}_{k}(s),\beta^{(2)}_{k}(s))$

over the interval $[0,1]$. The reward outcome
$\bm{R}_{k}=(R^{(T)}_{k},R^{(F)}_{k})$ is the data that is newly collected
about the reward. By Equation 1.1, the posterior is

$\displaystyle r(s)|\bm{R}_{k}\sim
Beta(R^{(T)}_{k}+\beta^{(1)}_{k}(s),R^{(F)}_{k}+\beta^{(2)}_{k}(s)).$

##### _Reward learning rate $\eta_{r}$_

The above is a theoretical bound on learning from observations in state $s$,
assuming a prior $Beta$ distribution over $[0,1]$ for the reward probability
$r(s)$. Some mice learn faster than others, and all of them will perform no
better than the ideal observer model above. To allow for individual
differences, and different learning rates, we introduce a model parameter
$\eta_{r}\in[0,1]$, which dials the amount of data required for the same
amount of learning as an ideal observer. The _update rule_ (i.e., posterior)
is

$\displaystyle r(s)|\bm{R}_{k}$ $\displaystyle\sim
Beta(\eta_{r}R^{(T)}_{k}+\beta^{(1)}_{k}(s),\eta_{r}R^{(F)}_{k}+\beta^{(2)}_{k}(s)).$

To keep track of learning parameters, let
$\mathbf{B}_{k}=\big{\\{}\bm{\beta}_{k}(s)\coloneqq\big{(}\beta^{(1)}_{k}(s),\beta^{(2)}_{k}(s)\big{)}:s\in
S\big{\\}}$ be the beta parameters for beliefs about reward probabilities of
all states in spatial step $k$. Note that after visiting state $s$ in spatial
step $k$,

$\displaystyle\bm{\beta}_{k+1}(s)=\eta_{r}\bm{R}_{k}+\bm{\beta}_{k}(s)$ for
$s=X_{k}$, and (3.1)
$\displaystyle\bm{\beta}_{k+1}(s^{\prime})=\bm{\beta}_{k}(s)$ for
$s^{\prime}\not=X_{k}$.

Note that $\eta_{r}$ is defined to have the same value across all states. If
$\eta_{r}=1$, the mice performs as well as the normative ideal observer, and
if $\eta_{r}=0$, the mouse never learns reward associations. For the values in
between $0$ and $1$, the mouse requires extra data points for updating its
beliefs to the same extent as an ideal observer model. The model parameter
$\eta_{r}$ can be interpreted as the data efficiency of learning. It could be
used to compare individual learning differences among subjects. Furthermore,
it is interesting to assess whether differences of $\eta_{r}$ in individuals
is predictive of comparative learning rates on other learning tasks. It also
provides a qualitative way to assess the model. For example, if the value is
unreasonably high, it may indicate a flaw in the state structure or an
incorrect choice of prior.

Table 3.2: Guide for variables (Var) and parameters (Par) relevant to internal
reward representations.

.  Var/Par Type Description $\bm{R}_{k}$ observed A binary pair representing
the reward outcome of step $k$, $(1,0)$ lick and reward within step $k$
$(0,1)$ lick but no reward within step $k$ $(0,0)$ no lick within step $k$
$\mathbf{B}(k)$ inferred List of
$\big{(}\beta^{(1)}_{k}(s),\beta^{(2)}_{k}(s)\big{)}$, for all $s\in S$, where
$Beta(\big{(}\beta^{(1)}_{k}(s),\beta^{(2)}_{k}(s)\big{)})$ represents the
beliefs about $r(s)$ at spatial step $k$. $\eta_{r}$ model parameter A
constant in the $[0,1]$ interval for learning rate of reward association.

##### _Implementation notes_

To simplify model implementation, we can derive the posterior distribution at
step $k$ by merely keeping a list record of the total count of positive and
negative reward outcomes in state $s$. In particular, at step $k$, for state
$s$, let $\bm{c}_{k}(s)=\Big{(}c_{k}^{(T)}(s),c_{k}^{(F)}(s)\Big{)}$ be the
total count of positive and negative outcomes in state $s$, from step $1$ up
to the start of step $k$. That is,

$\bm{c}_{k}(s)=\sum^{k}_{\begin{subarray}{c}n=1\\\
X_{n}=s\end{subarray}}\bm{R}_{k}.$

For current spatial state $k$, a list of numbers can store values of
$\bm{c}_{k}(s)$. Assuming a uniform prior at the start of the experiment, or
$\beta^{(1)}_{1}(s)=\beta^{(2)}_{1}(s)=1$, the prior probability distribution
of $r(s)$ at step $k$ is

$\displaystyle r(s)\sim
Beta(\eta_{r}c_{k}^{(T)}(s)+1,\eta_{r}c_{k}^{(F)}(s)+1),$

for which,

$\displaystyle\bm{\beta}_{k}(s)=\eta_{r}\bm{c}_{k}(s)+1.$ (3.2)

####  3.3.2 Learning state transitions

Learning dynamics for state transitions $p_{(s,s^{\prime})}$ is defined
similarly to the reward associations. Let $E$ be the set of transition edges
(directed edges), and let $Adj(s)=\\{s^{\prime}:(s,s^{\prime})\in E\\}$ be the
set of states which for $X_{k}=s$, outcome of $X_{k}$ is in $Adj(s)$.
Therefore, transition probabilities from $s$, $P(X_{k+1}|X_{k}=s)$ is a
distribution of outcomes over $Adj(s)$. Assuming fixed probability
transitions, $P(X_{k+1}|X_{k}=s)$ can be represented by a list of
probabilities $\bm{p}(s)\coloneqq\big{(}p_{(s,s^{\prime})}:s^{\prime}\in
Adj(s)\big{)}$. Note that if the subject is not familiar with the space, the
true distribution is unknown, and the subject learns about these probabilities
through experience.

##### _Normative model for updating internal transition representations
(Bayesian)_

Every time the subject leaves state $s$ and the next step is observed, one
observation is made about the outcome of $X_{k+1}$ given $X_{k}=s$. Because
the outcome is a multinomial random variable, where possible outcomes are
states in $Adj(s)$, we use a Dirichlet prior distribution to represent
uncertainties about $\bm{p}(s)$. Specifically, at spatial step $k$,

$\displaystyle\bm{p}(s)\sim Dir\big{(}\bm{\alpha}_{k}(s)\big{)}$

where the list of parameters $\bm{\alpha}_{k}(s)$ contains an element
corresponding to each possible outcome. In particular,

$\bm{\alpha}_{k}(s)=\big{(}\alpha_{k}(s,s^{\prime}):s^{\prime}\in
Adj(s)\big{)}.$

Suppose $X_{k}=s$ and consider an ideal observer whose prior beliefs about
$\bm{p}(s)$ at spatial step $k$ is described by $Dir(\bm{\alpha_{k}}(s))$.
Also suppose, the ideal observer visits the next state and makes the
observation $X_{k+1}=\breve{s}$. Then by Equation 1.1, the posterior
distribution is

$\displaystyle\bm{p}(s)|(X_{k+1}=\breve{s},X_{k}=s)\sim
Dir\big{(}\bm{\alpha}_{k+1}(s)\big{)}$

where any element $\alpha_{k}(s,s^{\prime})$ of $\bm{\alpha}_{k+1}(s)$ is
updated as follows:

$\displaystyle\alpha_{k+1}(s,s^{\prime})=1+\alpha_{k}(s,s^{\prime})$ for
$s^{\prime}=\breve{s}$, and
$\displaystyle\alpha_{k+1}(s,s^{\prime})=\alpha_{k}(s,s^{\prime})$ for
$s^{\prime}\not=\breve{s}$.

Furthermore, for any other state $s^{\prime\prime}\not=s$, it is obvious that
the beliefs are not updated, i.e.,
$\bm{\alpha}_{k+1}(s^{\prime\prime}\not=s)=\bm{\alpha}_{k}(s^{\prime\prime}\not=s)$.

##### _Reward learning rate $\eta_{p}$_

Similar to introducing a learning rate for learning reward association, we
introduce a $\eta_{p}\in[0,1]$ to account for data inefficiency compared to
the ideal observer. Denote by $\mathbf{A}_{k}$, the list of all learning
parameters of state transition probabilities
$\mathbf{A}_{k}=\big{(}\bm{\alpha}_{k}(s):s\in S\big{)}$. Now, the _update
rule_ (posterior distribution) is

$\displaystyle\bm{p}(s)|(X_{k+1},X_{k})\sim
Dir\big{(}\bm{\alpha}_{k+1}(s)\big{)}$

where any element $\alpha_{k}(s,s^{\prime})$ of a list of parameters in
$\mathbf{A}_{k}$ is updated as follows:

$\displaystyle\alpha_{k+1}(s,s^{\prime})=\eta_{p}+\alpha_{k}(s,s^{\prime})$
for $s=X_{k}$ and $s^{\prime}=X_{k+1}$ (3.3)
$\displaystyle\alpha_{k+1}(s,s^{\prime})=\alpha_{k}(s,s^{\prime})$ otherwise.

Table 3.3: Parameter guide for learning transition probabilities

.  Parameter(s) Type Description $(X_{k+1}|X_{k})$ observed Transition outcome
from a given state $X_{k}$ $\mathbf{A}_{k}$ inferred List of
$\bm{\alpha}_{k}(s)$, for all $s\in S$, where $Dir(\bm{\alpha}_{k}(s))$
represents beliefs about $\bm{p}(s)$ at step $k$ (list of state transition
probabilities from $s$ to adjacent states) $\eta_{p}$ free parameter A
constant in the $[0,1]$ interval for learning rate of transition probabilities

For an ideal observer, $\eta_{p}=1$. The lower the value of $\eta_{p}$ is, the
slower the learning becomes, because the subject would require more data for
similar updates in beliefs. If $\eta_{p}=0$, the subject never learns from
observing consecutive states. Note that the same parameter $\eta_{p}$ is used
for learning all transition probabilities.

##### _Implementation notes_

For prior beliefs about state transitions, a uniform prior would ensure that
the prior does not privilege any probability value over another probability
value. Then, for any entry $\alpha_{1}(s,s^{\prime})$ of
$\bm{\alpha}_{1}(s,s^{\prime})$, we assume that $\alpha_{1}(s,s^{\prime})=1$

So, at spatial step $k$, for entry $\alpha_{s^{\prime}}(s,k)$ of
$\bm{\alpha}(s,k)$,

$\displaystyle\alpha_{k}(s,s^{\prime})=\eta_{p}c_{(s,s^{\prime})}(k)+1.$ (3.4)

where $c_{(s,s^{\prime})}(k)$ is the total number of observed transitions from
$s$ to $s^{\prime}$ from step $1$ to step $k$. By keeping track of
$c_{(s,s^{\prime})}(k)$ in a matrix, any parameter in $\mathbf{A}(k)$ can be
calculated on demand using Equation 3.4 for the current state.

##  Chapter 4 Behavioral model part 2: the generative model

In the previous chapter, I discussed the internal representations of spatial
regions and reward probabilities within those regions. This chapter describes
a model that utilizes internal representations to generate behaviour. The
learning model for updating beliefs about reward probabilities and state
transitions utilized a normative model of Bayesian learning. In contrast, we
present a descriptive model of behaviour that does not explicitly enforce any
optimal decision-making criteria. Before making normative assumptions about
behaviour, it is important to have a descriptive framework for systematically
assessing assumptions about behaviour.

Recall that location, visual stimulus, licking and speed of the mouse are
recorded in the experimental data (see Chapter 2.2). To improve readability,
Table 4.1 includes notation used to represent the behavioural data.

Table 4.1: Behavioral and observational records for $t\in\\{1,2\ldots,N\\}$.  Data | Type | Description
---|---|---
$x_{t}$ | Observation |  |  $x_{t}$ is the true value of the distance from the onset of the current
---
corridor at time step $t$.
$y_{t}$ | Observation |  |  $y_{t}\in Cor=\\{grey,vertical,angled\\}$ is the true value of the
---
corridor type, which determines the visual stimuli at time
step $t$.
$o_{t}$ | Observation |  |  $o_{t}$ is a binary value for whether the reward valve has opened
---
during the time step.
$v_{t}$ | Behavior |  | Speed (average) at time step $t$
---
$l_{t}$ | Behavior | Number of licks at time step $t$

A spatial state transition event triggers updating internal representations of
reward probability and spatial transitions. During the period between two
transition events, the parameters associated with internal representations
(specified by elements of $\bm{B}_{k}$ and $\bm{A}_{k}$) are unchanged.
Assuming that the internal representations are guiding the behaviour, we
define behavioural parameters for speed and licking rate derived from internal
representations’ parameters. Figure 4.1 describes the conditional dependence
structure of parameters associated with a spatial state. In this model, the
internal representations are used to derive two parameters that guide the
licking and speed behaviour. These parameters are _target speed_
$\tilde{\nu_{k}}$, and _licking rate_ $\tilde{\lambda_{k}}$, and they are
discussed in detail in the Section 4.2 and Section 4.1 respectively.

Figure 4.1: Graphical model of updating internal representations at a given spatial step, the associated learning parameters (green), and the associated behavioural parameters (blue). The dotted squares indicate internal representations that are not observed in the data. Variables inside circles have stochastic outcomes given their parents, and variables inside squares have deterministic outcomes given their parents. State transitions trigger updating these variables for the new step $k+1$. Note that the model satisfies the Markov property. A description of the conditional dependencies is included in Table 4.2. Table 4.2: Description of updating internal representations of a given step using the graphical model of 4.1. Variables (_Var._) and their parents _(Par(.))_ are included in the first and second columns respectively. The third column _(Type)_ indicates whether the outcome of the variable given its parents is stochastic (_Stoch._) or deterministic (_Deter._) given its parents. The conditional dependence of the variable on its parents is described in the last column.  Var. | Par(.) | Type | Update description
---|---|---|---
$X_{k+1}$ |  | $X_{k}$
---
Stoch. |  | Stochastic outcome of the state immediately
---
following $X_{k}$.
$\bm{B}_{k+1}$ |  | $\bm{B}_{k},\bm{R}_{k},X_{k}$
---
Deter. |  | Updating reward probability distribution of the
---
previous state using Equation 3.1.
$\bm{A}_{k+1}$ |  | $\bm{A}_{k},X_{k+1},X_{k}$
---
Deter. |  | Updating the transition probability distribution
---
for the last transition using Equation 3.3.
distributions for reward, to $\bm{B}_{k+1}$
$r_{k}$ | $\bm{B}_{k},X_{k}$ | Deter. |  | Reward distribution of the current state
---
$\gamma_{k}(\rho)$ | $\bm{B}_{k},\bm{A}_{k},\bm{X}_{k}$ | Deter. |  | Discounted reward probability of present and
---
future states given by Equation 4.6, with the
discount factor $\rho$.
$\tilde{\nu}_{k}$ | $\gamma_{k}(\rho)$ | Deter. |  | Value of target speed in spatial step $k$ adjusted by
---
value of $\gamma_{k}(\rho)$.
$\tilde{\lambda}_{k}$ | $r_{k}$ | Deter. |  | Licking rate in step $k$ given by Equation 4.3.
---
$R_{k}$ | $\tilde{\lambda}_{k}$ | Stoch. |  | Reward outcome of spatial state $k$
---

###  4.1 Spatial state parameter $\tilde{\lambda_{k}}$: licking rate

Consider the relevance of the reward probability distribution for $r_{k}$ to
the licking behaviour. First, it is reasonable to consider the mouse
regulating its licking rate using its perception of expected reward
probability in the current state. The expected value of the reward probability
in the current state (in step $k$) is the expected value of
$Beta(\beta^{(1)}_{k}(s),\beta^{(2)}_{k}(s))$, which is

$\displaystyle\mu(r_{k})=\frac{\beta^{(1)}_{k}}{\beta^{(1)}_{k}\beta^{(2)}_{k}}.$
(4.1)

Second, independently from the expectation of reward, the degree of
uncertainty about the true probability of reward may also be relevant to
behaviour (Zhao & Warren, 2015), and in particular, the rate of licking in the
current state. More variance in the reward probability may mean that the
current state should be further explored by licking, to decrease the
uncertainty about reward values. The variance reward probability beliefs can
also be calculated from the $Beta()$ distribution.

$\displaystyle\sigma^{2}(r_{k})=\frac{\beta^{(1)}_{k}\beta^{(2)}_{k}}{(\beta^{(1)}_{k}+\beta^{(2)}_{k})^{2}~{}(\beta^{(1)}_{k}+\beta^{(2)}_{k}+1)}$
(4.2)

Let $L_{t}$ be a random variable for the number of licks at time step $t$. We
assume that the licking rate is generated by a Poisson distribution

$\displaystyle L_{t}\sim Pois(\tilde{\lambda_{k}})$

where for model parameters $\omega_{1}$, $\omega_{2}$ and $\omega_{3}$,

$\displaystyle\tilde{\lambda_{k}}=\omega_{1}\mu(r_{k})+\omega_{2}\sigma(r_{k})+\omega_{3},$
(4.3)

is the licking rate at a time step spent within the current spatial step. The
probability that $L_{t}=l_{t}$, for a number of licks $l_{t}$ is given by

$\displaystyle
P(L_{t}=l_{t})=\frac{\lambda_{k}^{l_{t}}e^{-\lambda_{k}}}{l_{t}!}$ (4.4)

Table 4.3: Parameters relevant to the licking behaviour.

.  Parameter Type Description $\tilde{\lambda_{k}}$ Spatial state parameter
Rate of the Poisson distribution generating the licking behavior within a time
step spent in spatial step $k$ $\omega_{1}$ Model parameter Weight of the
expected reward probability of the current reward distribution for calculating
the spatial state parameter $\tilde{\lambda_{k}}$ $\omega_{2}$ Model parameter
Weight of the standard deviation of the current reward distribution for
calculating the spatial state parameter $\tilde{\lambda_{k}}$ $\omega_{3}$
Model parameter base licking rate for calculating $\tilde{\lambda_{k}}$

###  4.2 Parameter $\tilde{\nu_{k}}$: target speed within the current spatial
state

We noticed that the mouse tends to speed up if it does not expect a reward in
upcoming states (for example, see Figures 2.6 and 2.6). We model this behavior
using a discounted measure of future rewards.

##### _Discounted future reward_

Expected average reward probability $m$ steps after the current state $s$ can
be formulated as follows

$\displaystyle\sum_{s^{\prime}\in
S}E(r[s^{\prime}])P(X_{k+m}=s^{\prime}|X_{k}=s)$ (4.5)

Value of $P(X_{k+m}|X_{k})$ can be estimated by the transition probability
matrix obtained by the expected value of transition probabilities and standard
Markov chain transition properties (Equation 1.1) (Häggström et al., 2002). To
estimate the values of the transition probability matrix, we use the expected
value of transition probability for $p_{(s,s^{\prime})}$, using parameters of
Dirichlet distributions for transition probabilities in $\bm{A_{k}}$;

$\displaystyle
E[p_{(s,s^{\prime})}]=\frac{\alpha_{k}(s,s^{\prime})}{{\sum}_{s^{\prime\prime}\in
Adj(s)}\alpha_{k}(s,s^{\prime\prime})}$

is the estimated probability value for $p_{(s,s^{\prime})}$ entry of the
transition probability matrix. To conclude the discussion for the calculation
of expression 4.5, note that
$E(r[s^{\prime}])=\beta^{(1)}_{k}(s^{\prime})/\big{(}\beta^{(1)}_{k}(s^{\prime})\beta^{(2)}_{k}(s^{\prime})\big{)}.$

Now, let us define the _discounted future reward $\gamma_{k}(\rho)$_ for a
fixed value of $\rho$ in the current step $k$ to be

$\displaystyle\gamma_{k}(\rho)\coloneqq\frac{{\sum}^{\infty}_{m=0}\rho^{m}{\sum}_{s^{\prime}\in
S}\Big{(}E[r(s^{\prime})]P(X_{k+m}|X_{k})\Big{)}}{\sum^{\infty}_{m=0}\rho^{m}}$
(4.6)

Note that $\gamma_{k}(\rho)$ is a normalised sum of discounted present and
future expected reward probability values. Similar to the value function in
reinforcement learning (Sutton & Barto, 2018), or the concept of discounted
cash flow in financial asset valuation (Damodaran, 2012), it incorporates all
future reward values by iteratively giving less weight to future rewards that
are further away.

When transitioning from one state to another, lower discounted future reward
$\gamma_{k}(\rho)$ is likely to indicate that the next reward is further away.
In this case, the mouse may choose to adjust its behavior (Kleinfeld et al.,
2006), by speeding up to pass the unrewarded regions more quickly. Since the
discounted value of future reward does not change as long as the mouse is in
the same spatial state, the desired speed at the current spatial step can be
modeled as a spatial state parameter. Let the _target speed_ $\tilde{\nu_{k}}$
for the current state be

$\displaystyle\tilde{\nu_{k}}:=v_{max}\Big{(}1-\gamma_{k}(\rho)\Big{)}$ (4.7)

where $v_{max}$ is a model parameter that puts an upper bound on the target
speed. A simple model of speed for time step $t$ is the following

$\displaystyle v_{t}\sim\mathcal{N}(\tilde{\nu_{k}},\sigma_{\nu}^{2}).$ (4.8)

However, physical constraints on the movement does not permit an instant jump
in speed when the spatial state changes. The alternative model of speed that
takes the physical constraints into considerations (by adding more
parameters), is

$\displaystyle v_{t+1}\sim\mathcal{N}(E[v_{t+1}],Var[v_{t+1}]),$ (4.9)

where,

$\displaystyle(E[v_{t+1}],Var[v_{t+1}])=\begin{cases}(v_{t}+\delta_{v}^{\scalebox{0.5}{+}},\sigma_{v}^{2})&\text{
for }v_{t}<\tilde{\nu_{k}}-\epsilon,\\\
(v_{t}+\delta_{v}^{\scalebox{0.8}{-}},\sigma_{v}^{2})&\text{ for
}v_{t}>\tilde{\nu_{k}}+\epsilon,\\\ (v_{t},\sigma_{v}^{2})&\text{ otherwise;
i.e., for }v_{t}\in[\tilde{\nu_{k}}-\epsilon,\tilde{\nu_{k}}+\epsilon].\\\
\end{cases}$ (4.10)

where the model parameters $\delta_{v}^{\scalebox{0.5}{+}}$ and
$\delta_{v}^{\scalebox{0.8}{-}}$ are constant values for acceleration and
deceleration, $\sigma_{v}^{2}$ is the variance of speed outcome in the next
time-step. Furthermore, model parameter $\epsilon$ determines the range where
non-random acceleration or deceleration is not enforced.

Table 4.4: Parameters relevant to the speed behavior.

.  Parameter Type Description $\rho$ Model parameter Discount rate of future
reward (Expression 4.6) $\tilde{\nu_{k}}$ Spatial state parameter Target speed
(Expression 4.7) $\sigma_{\tilde{\nu}}^{2}$ Model parameter Variance of speed
in the first model (Expression 4.8) $\sigma_{v}^{2}$ Model parameter Variance
of speed change Expression 4.9 (second model)
$\delta_{v}^{\scalebox{0.5}{+}},\delta_{v}^{\scalebox{0.8}{-}}$ Model
parameter Acceleration and deceleration rate (second model) $\epsilon$ Model
parameter Range of random only of speed change (second model)

###  4.3 Generative model of licking and speed

For given spatial states structure (by fixing parameters $\mathcal{V}$ and
$d$), there exists a function $f_{\mathcal{V},d}:(xLoc\times Cor)\to S$ that
associates each position to states. Then it is possible to determine time
steps associated with state transitions. In Chapter 3.1, we assumed that the
states are fully observable to the subject. Therefore, the subject knows the
value of $f_{\mathcal{V},d}$ at any current time step.

##### _Binary variable $K_{t}$: indicator of spatial state transition event_

For the current time step $t$, let $K_{t}$ be a binary variable such that

$\displaystyle K_{t+1}=\begin{cases}0,&\text{ for
}f_{\mathcal{V},d}(x_{t},y_{t})=f_{\mathcal{V},d}(x_{t+1},y_{t+1})\\\
1,&\text{ for
}f_{\mathcal{V},d}(x_{t},y_{t})\not=f_{\mathcal{V},d}(x_{t+1},y_{t+1}).\\\
\end{cases}$ (4.11)

That is to say, $K_{t}=1$ if $(x_{t},y_{t})$ and $(x_{t+1},y_{t+1})$ are not
in the same state, ans so a state transition has occurred. Note that a spatial
state transition triggers an update in the beliefs about the environment
(reward probability within states and state transitions). Then the internal
representations in the graphical model of Figure 4.1 are updated to the next
spatial step, and the behavioral parameters $\lambda_{k_{t+1}}$ and
$\tilde{nu}_{k_{t+1}}$ correspond to the new spatial step. For $K_{t}=0$, the
behavioral parameters $\tilde{\lambda}_{k_{t+1}}$ and $\tilde{nu}_{k_{t+1}}$
remain unchanged from the previous time-step.

Figure 4.2 is the graphical model for the generative model of behavior within
time steps. The model assumes that the spatial state associated with
$(x_{t},y_{t})$ is unambiguously determined by the subject (fully observable
spatial states). Therefore, the value of $K_{t+1}$, which indicates a state
transition, is also observed by the subject. Furthermore, $K_{t+1}$ can be
deterministically inferred from the experimental data using the Equation 4.11.
Hence, it is also observed in the behavioral data. If $K_{t+1}=1$, then the
graphical model of updating internal representations is used to find the new
behavioral parameters (indicated by green arrows). If $K_{t+1}=0$, the
behavioral parameters remain unchanged from the previous step. A description
of the relationships is included in Table 4.5.

Table 4.5: Description of relationships in the generative model of behavior in the graphical model of 4.2. Variables (_Var._) and their parents _(Par(.))_ are included in the first and second column respectively. Third column _(Type)_ indicates whether the outcome of the variable given its parents is stochastic (_Stoch._) or deterministic (_Deter._) given its parents. The conditional dependence of the variable on its parents is described in the last column.  Var. | Par(.) | Type | Update description
---|---|---|---
$K_{t}$ |  | $(x_{t},y_{t})$
---
$(x_{t+1},y_{t+1})$
Stoch. |  | Transition event indicator (Expression 4.11).
---
$\tilde{\nu}_{k_{t+1}}$ |  | $\tilde{\nu}_{k_{t}},K_{t}$
---
Deter. |  | For $K_{t}=0$, $\tilde{\nu}_{k_{t+1}}=\tilde{\nu}_{k_{t}}$. Otherwise, spatial state changes,
---
and graphical model 4.1 updates the value.
$\tilde{\lambda}_{k_{t+1}}$ |  | $\tilde{\lambda}_{k_{t}},K_{t}$
---
Deter. |  | For $K_{t}=0$, $\tilde{\lambda}_{k_{t+1}}=\tilde{\lambda}_{k_{t}}$. Otherwise, spatial state changes,
---
and graphical model 4.1 updates the value.).
$l_{t}$ | $\tilde{\lambda_{k}}$ | Stoch. |  | Poisson distributed value with rate $\tilde{\lambda_{k}}$ (Expression 4.3)
---
$v_{t}$ | $\tilde{\nu_{k}}$ | Stoch. |  | Speed at time step $t$ by first model (Expression 4.8 ),
---
or second model (Expression 4.9.
Figure 4.2: Graphical model of the generative model of behavior. Note that the
variables and relationships drawn in yellow and brown are not part of the
internal model, and they describe the conditional dependence of the observed
values to the model variables. See table 4.5 for description of the
relationships.

###  4.4 Estimation of model parameters

Below, the general framework for estimating the model parameters is discussed.
For a fixed spatial model of space $\mathcal{M}_{V,d}$, let $\bm{\theta}$ be
the list of model parameters

$\displaystyle\bm{\theta}$
$\displaystyle\coloneqq(\mathcal{V},d,\eta_{r},\eta_{p},\omega_{1},\omega_{2},\omega_{3},\sigma^{2}_{\tilde{\nu}}),$
(using the second speed model), or $\displaystyle\bm{\theta}$
$\displaystyle\coloneqq(\mathcal{V},d,\eta_{r},\eta_{p},\omega_{1},\omega_{2},\omega_{3},\sigma^{2}_{\tilde{v}},\delta_{v}^{\scalebox{0.5}{+}},\delta_{v}^{\scalebox{0.8}{-}},\epsilon)$
(using the first speed model).

Given the model parameters, and given observational data, parents of $v_{t}$
and $l_{t}$ are deterministically set at each time point (see graphical model
4.2). Therefore, speed and licking are independent. So model likelihood of the
generative model of behaviour at time step $t$ is

$\displaystyle\mathcal{L}\big{(}\bm{\theta}|(v_{t},l_{t})\big{)}$
$\displaystyle=P(v_{t},l_{t}|\bm{\theta})=P(v_{t}|\bm{\theta})\
P(l_{t}|\bm{\theta})$ $\displaystyle\sim
f\big{(}\mu_{t}(\bm{\theta}),\sigma_{t}(\bm{\theta})\big{)}\
g\big{(}l_{t};\lambda_{t}(\bm{\theta})\big{)}$

where $f$ are $g$ are probability mass functions for Gaussian and Poisson
distributions respectively. Note that their distribution parameters are
deterministically fixed at each time point given the model parameters (see
Equations 4.3, 4.8 and 4.9). Then _model evidence_ for the generative model
for up to time step $N$ is

$\displaystyle\mathcal{L}\Big{(}\bm{\theta}\Big{|}\\{(v_{t},l_{t}):t=1\ldots
N\\}\Big{)}\propto\prod_{t=1}^{N}f\big{(}v_{t};\mu_{t}(\bm{\theta}),\sigma_{t}(\bm{\theta})\big{)}\
g\big{(}l_{t};\lambda_{t}(\bm{\theta})\big{)}$ (4.12)

And we can then use the maximum likelihood estimation (MLE) to estimate the
fitted model parameters

$\displaystyle\bm{\bm{\theta}}^{*}=\underset{\bm{\theta}}{\operatorname{argmax}}\sum_{t=1}^{N}ln\Big{[}f\big{(}v_{t};\mu_{t}(\bm{\theta}),\sigma_{t}(\bm{\theta})\big{)}\
g\big{(}l_{t};\lambda_{t}(\bm{\theta})\big{)}\Big{]}$ (4.13)

Note that for each spatial step, the graphical model is used for calculating
the parameters $\mu_{t}(\bm{\theta}),\sigma_{t}(\bm{\theta})$ and
$\lambda_{t}(\bm{\theta})$.

##  Chapter 5 Discussion

The next step in the project is to first complete the model validation on
synthetic data. Before applying the model to real data, it is important to
scrutinize the behaviour of the generative model. We plan to do so by pre-
determining values for a model parameter and generating synthetic behavioural
data. The generated behaviour is then used as a given data set. If the model
is well-behaved, the model parameters should be recoverable from the synthetic
data. As different spatial state structures radically alter the learning
dynamics, we will conduct the parameter recovery for spatial model parameters
more diligently. By considering various alternative hypotheses (different
values for $d$ and $\mathcal{V}$), the model evidence (equation 4.12) of
alternative hypotheses will be compared. For a well-behaved model, the model
evidence for the parameters used to generate data is expected to be the best.

###  5.1 Limitations

While our model assumes fully observable Markov states, noisy observations of
the location and visual stimuli introduce uncertainty about the true current
state of the system. Indeed, observations of the environment are often noisy
and some behavioural models take this into account (Kang et al., n.d.; Kersten
& Mamassian, 2009). While the learning rates of reward probability and
transition probability capture some aspects of noisy observations, they are
not based on normative assumptions. Alternatives should be considered for
future research (Laquitaine & Gardner, 2018). Fortunately, there is an
extensive body of research on partially observable Markov decision processes
(Monahan, 1982; Kaelbling et al., 1996) that would provide a clear path for
improving the current model.

An alternative to estimating the model parameters using MLE in Chapter 4.4 is
to use the maximum a posteriori estimation (MAP) (Murphy, 2012; Griffiths &
Yuille, 2008). In contrast to MLE, which gives one estimated value for each
parameter, MAP gives a distribution for each parameter, characterising the
level of uncertainty about each parameter. Since some of the model parameters
are qualitatively interpretable, MAP may be particularly relevant. In
particular, a distribution over possible options for $\mathcal{V}$, the set of
discriminated visual stimuli, is highly relevant to the imaged activity of the
visual cortex. The potential challenge of MAP is that the computational
difficulty of the calculation may introduce implementation challenges that are
difficult to resolve. Nonetheless, its estimation of model parameters are
potentially more meaningful for studying visual perception.

###  5.2 Implications

During the experiments, two-photon calcium imaging and optogenetics were
performed to determine changes in inputs and activity of individual excitatory
and inhibitory cells within the primary visual cortex. Previously, a
multivariate auto-regressive linear model (MVAR) was fitted to the neuronal
data (Khan et al., 2018):

$\displaystyle\bm{q}_{t+1}=\bm{q}_{t}+A\times\bm{q}_{t}+\bm{u}_{t}+\xi v_{t}$

where $\bm{q}_{t}$ is the vector of response levels at time step $t$ for all
$n$ imaged neurons, $A$ is an $n\times n$ matrix that includes the fitted
interaction parameters, $\bm{u}_{t}$ is a fitted vector for the stimulus-
related input, and $\xi$ is a fitted parameter for the contribution of current
speed $v_{t}$. The MVAR model was used to compare the activity of populations
of different inhibitory and excitatory cell types. The only behavioural term
that was included was speed $v_{t}$, which did not make a significant
contribution. An immediate application of the current behavioural model
presented in this report is to potentially improve the MVAR model by including
parameters related to internal representations, In particular, learned
parameters that are likely to be relevant to behaviour, namely expected reward
probability $\mu(r_{k})$, variance $\sigma^{2}(r_{k})$, and discounted future
reward $\gamma_{k}(\rho)$ could potentially improve the predictive power of
the MVAR model.

If the internal representation terms from the behavioural model improve the
predictive power of the MVAR model, it will give new insights into the
information encoded in neurons located in the primary visual cortex. Future
experiments can then be designed to systematically manipulate these internal
terms to understand the precise representations (Heilbron et al., 2020). This
will help us understand how the structure of the environment changes learning
dynamics and internal representations.

## References

  * (1)
  * Barlow et al. (1961) Barlow, H. B. et al. (1961), ‘Possible principles underlying the transformation of sensory messages’, Sensory communication 1, 217–234.
  * Beck et al. (2008) Beck, J. M., Ma, W. J., Kiani, R., Hanks, T., Churchland, A. K., Roitman, J., Shadlen, M. N., Latham, P. E. & Pouget, A. (2008), ‘Probabilistic population codes for bayesian decision making’, Neuron 60(6), 1142–1152.
  * Berkes et al. (2011) Berkes, P., Orbán, G., Lengyel, M. & Fiser, J. (2011), ‘Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment’, Science 331(6013), 83–87.
  * Bishop (2006) Bishop, C. M. (2006), Pattern recognition and machine learning, springer.
  * Chen et al. (2013) Chen, T.-W., Wardill, T. J., Sun, Y., Pulver, S. R., Renninger, S. L., Baohan, A., Schreiter, E. R., Kerr, R. A., Orger, M. B., Jayaraman, V. et al. (2013), ‘Ultrasensitive fluorescent proteins for imaging neuronal activity’, Nature 499(7458), 295–300.
  * Damodaran (2012) Damodaran, A. (2012), Investment valuation: Tools and techniques for determining the value of any asset, Vol. 666, John Wiley & Sons.
  * Dombeck et al. (2010) Dombeck, D. A., Harvey, C. D., Tian, L., Looger, L. L. & Tank, D. W. (2010), ‘Functional imaging of hippocampal place cells at cellular resolution during virtual navigation’, Nature neuroscience 13(11), 1433–1440.
  * Fiser et al. (2010) Fiser, J., Berkes, P., Orbán, G. & Lengyel, M. (2010), ‘Statistically optimal perception and learning: from behavior to neural representations’, Trends in cognitive sciences 14(3), 119–130.
  * Fishell & Kepecs (2019) Fishell, G. & Kepecs, A. (2019), ‘Interneuron types as attractors and controllers’, Annual review of neuroscience 43.
  * Geisler (2003) Geisler, W. S. (2003), ‘Ideal observer analysis’, The visual neurosciences 10(7), 12–12.
  * Geisler (2011) Geisler, W. S. (2011), ‘Contributions of ideal observer theory to vision research’, Vision research 51(7), 771–781.
  * Griffiths & Yuille (2008) Griffiths, T. & Yuille, A. (2008), ‘A primer on probabilistic inference’, The probabilistic mind: Prospects for Bayesian cognitive science pp. 33–57.
  * Häggström et al. (2002) Häggström, O. et al. (2002), Finite Markov chains and algorithmic applications, Vol. 52, Cambridge University Press.
  * Harvey et al. (2009) Harvey, C. D., Collman, F., Dombeck, D. A. & Tank, D. W. (2009), ‘Intracellular dynamics of hippocampal place cells during virtual navigation’, Nature 461(7266), 941–946.
  * Heeger (2017) Heeger, D. J. (2017), ‘Theory of cortical function’, Proceedings of the National Academy of Sciences 114(8), 1773–1782.
  * Heilbron et al. (2020) Heilbron, M., Richter, D., Ekman, M., Hagoort, P. & De Lange, F. P. (2020), ‘Word contexts enhance the neural representation of individual letters in early visual cortex’, Nature communications 11(1), 1–11.
  * Kaelbling et al. (1996) Kaelbling, L. P., Littman, M. L. & Moore, A. W. (1996), ‘Reinforcement learning: A survey’, Journal of artificial intelligence research 4, 237–285.
  * Kang et al. (n.d.) Kang, Y. H., Mahr, J., Nagy, M., Andrási, K., Csibra, G. & Lengyel, M. (n.d.), ‘Eye movements reflect causal inference during episodic memory retrieval’.
  * Kepecs & Fishell (2014) Kepecs, A. & Fishell, G. (2014), ‘Interneuron cell types are fit to function’, Nature 505(7483), 318–326.
  * Kersten & Mamassian (2009) Kersten, D. & Mamassian, P. (2009), ‘Ideal observer theory’, Encyclopedia of neuroscience 5, 89–95.
  * Khan et al. (2018) Khan, A. G., Poort, J., Chadwick, A., Blot, A., Sahani, M., Mrsic-Flogel, T. D. & Hofer, S. B. (2018), ‘Distinct learning-induced changes in stimulus selectivity and interactions of gabaergic interneuron classes in visual cortex’, Nature neuroscience 21(6), 851–859.
  * Kleinfeld et al. (2006) Kleinfeld, D., Ahissar, E. & Diamond, M. E. (2006), ‘Active sensation: insights from the rodent vibrissa sensorimotor system’, Current opinion in neurobiology 16(4), 435–444.
  * Kriegeskorte & Douglas (2018) Kriegeskorte, N. & Douglas, P. K. (2018), ‘Cognitive computational neuroscience’, Nature neuroscience 21(9), 1148–1160.
  * Laquitaine & Gardner (2018) Laquitaine, S. & Gardner, J. L. (2018), ‘A switching observer for human perceptual estimation’, Neuron 97(2), 462–474.
  * Maloney & Mamassian (2009) Maloney, L. T. & Mamassian, P. (2009), ‘Bayesian decision theory as a model of human visual perception: Testing bayesian transfer’, Visual neuroscience 26(1), 147–155.
  * Monahan (1982) Monahan, G. E. (1982), ‘State of the art—a survey of partially observable markov decision processes: theory, models, and algorithms’, Management science 28(1), 1–16.
  * Murphy (2012) Murphy, K. P. (2012), Machine learning: a probabilistic perspective, MIT press.
  * Orbán et al. (2008) Orbán, G., Fiser, J., Aslin, R. N. & Lengyel, M. (2008), ‘Bayesian learning of visual chunks by human observers’, Proceedings of the National Academy of Sciences 105(7), 2745–2750.
  * Poort et al. (2015) Poort, J., Khan, A. G., Pachitariu, M., Nemri, A., Orsolic, I., Krupic, J., Bauza, M., Sahani, M., Keller, G. B., Mrsic-Flogel, T. D. et al. (2015), ‘Learning enhances sensory and multiple non-sensory representations in primary visual cortex’, Neuron 86(6), 1478–1490.
  * Pouget et al. (2003) Pouget, A., Dayan, P. & Zemel, R. S. (2003), ‘Inference and computation with population codes’, Annual review of neuroscience 26(1), 381–410.
  * Saleem et al. (2018) Saleem, A. B., Diamanti, E. M., Fournier, J., Harris, K. D. & Carandini, M. (2018), ‘Coherent encoding of subjective spatial position in visual cortex and hippocampus’, Nature 562(7725), 124–127.
  * Sutton & Barto (2018) Sutton, R. S. & Barto, A. G. (2018), Reinforcement learning: An introduction, MIT press.
  * Yan et al. (2014) Yan, Y., Rasch, M. J., Chen, M., Xiang, X., Huang, M., Wu, S. & Li, W. (2014), ‘Perceptual training continuously refines neuronal population codes in primary visual cortex’, Nature neuroscience 17(10), 1380–1387.
  * Yang & Maunsell (2004) Yang, T. & Maunsell, J. H. (2004), ‘The effect of perceptual learning on neuronal responses in monkey visual area v4’, Journal of Neuroscience 24(7), 1617–1626.
  * Yap & Greenberg (2018) Yap, E.-L. & Greenberg, M. E. (2018), ‘Activity-regulated transcription: bridging the gap between neural activity and behavior’, Neuron 100(2), 330–348.
  * Zhao & Warren (2015) Zhao, M. & Warren, W. H. (2015), ‘How you get there from here: Interaction of visual landmarks and path integration in human navigation’, Psychological science 26(6), 915–924.