11institutetext: imec, Leuven, Belgium 22institutetext: ESAT, KU Leuven,
Belgium 33institutetext: University of California at San Diego, La Jolla, USA
44institutetext: VERSES Research Lab, Los Angeles, California, USA
44email<EMAIL_ADDRESS>
# Active Inference in Hebbian Learning Networks

Ali Safa 112233 Tim Verbelen 44 Lars Keuninckx 22 Ilja Ocket 22 André Bourdoux
22 Francky Catthoor 1122 Georges Gielen 1122 Gert Cauwenberghs 33

###### Abstract

This work studies how brain-inspired neural ensembles equipped with local
Hebbian plasticity can perform active inference (AIF) in order to control
dynamical agents. A generative model capturing the environment dynamics is
learned by a network composed of two distinct Hebbian ensembles: a posterior
network, which infers latent states given the observations, and a state
transition network, which predicts the next expected latent state given
current state-action pairs. Experimental studies are conducted using the
Mountain Car environment from the OpenAI gym suite, to study the effect of the
various Hebbian network parameters on the task performance. It is shown that
the proposed Hebbian AIF approach outperforms the use of Q-learning, while not
requiring any replay buffer, as in typical reinforcement learning systems.
These results motivate further investigations of Hebbian learning for the
design of AIF networks that can learn environment dynamics without the need
for revisiting past buffered experiences.

###### Keywords:

Active Inference Hebbian Learning Sparse Coding.

##  1 Introduction

The study of Sparse Coding [1], [2], [3], [4] and Predictive Coding [5], [6],
[7] networks has gained much attention for understanding the mechanisms
underlying learning and inference in the brain [8]. In particular, it has been
shown that the learning of the weight dictionary used to project the input
signals into sparse codes can be conducted via the biologically-plausible
Hebbian learning mechanism [9], with experimental evidence behind this
mechanism observed in the brain [10], [11]. Hebbian learning differs from the
widely-used back-propagation of error (backprop) technique due to its local
nature [7], [12], [13], where the weight $w_{j}$ of neuron $i$ is modified via
a combination $f$ of the weight’s input $x_{j}$ and the neuron’s output
$y_{i}$ (with $\eta_{d}$ the learning rate parameter):

$w_{j}\xleftarrow[]{}w_{j}+\eta_{d}f(y_{i},x_{j})$ (1)

When applied to layers that embark some form of competition between their
neurons, the Hebbian mechanism in (1) leads to the unsupervised learning of
complementary features from the input signals [14].

At the same time, Active Inference (AIF) has gained huge interest as a first-
principle theory, explaining how biological agents evolve and perform actions
in their environment [15], [16]. In recent years, the use of deep neural
networks (DNNs) for parameterizing generative models has gained much attention
in AIF research [17], [18], [19]. Deep AIF systems are typically composed of a
posterior network $q_{\Phi_{P}}(s_{l}|o_{l-1},a_{l-1})$, inferring the latent
state $s_{l}$ given an incoming observation-action pair
$\\{o_{l-1},a_{l-1}\\}$, and a state-transition network
$p_{\Phi_{S}}(s_{l}|s_{l-1},a_{l-1})$, predicting the next latent state
$s_{l}$ given the current state-action pair $\\{s_{l-1},a_{l-1}\\}$ [17]. The
state-transition network is used to generate the agent’s roll-outs for
different policies in order to compute the Expected Free Energy associated to
each policy [17]. Finally, a likelihood network $p_{\Phi_{L}}(o_{l}|s_{l})$
reconstructing the input observation $o_{l}$ from the latent state $s_{l}$ can
also be implemented [20]. Each network parameterizes its respective density
function through weight tensors $\Phi_{P},\Phi_{S}$ and $\Phi_{L}$.

In this work, we aim to study how AIF can be performed in Hebbian learning
networks without resorting to backprop (as typically used in deep AIF
systems). Experiments conducted in the OpenAI Mountain Car environment [21]
show that the proposed Hebbian AIF approach outperforms the use of Q-learning
and compares favorably to the backprop-trained Deep AIF system of [17], while
not requiring any replay buffer, as in typical reinforcement learning systems
[22]. Our derivations and experiments add to a growing number of work
addressing the study of Hebbian Active Inference [23], [24].

This paper is organized as follows. Background theory about Hebbian learning
networks is provided in Section 2. Our Hebbian AIF methods are covered in
Section 3. Experimental results are shown in Section 4. Conclusions are
provided in Section 5.

##  2 Background Theory on Hebbian Learning Networks

Inspired by previous works that model the neural activity of biological agents
through Sparse Coding [5], [9] (such as in the mushroom body of an insect’s
brain [25]), we model each individual Hebbian Ensemble layer of our networks
as an identically-distributed Gaussian likelihood model with a Laplacian prior
on the neural activity $c$:

$p(c|o,\Phi)\sim\exp{(-||\Phi c-o||_{2}^{2})}\exp{(-\lambda||c||_{1})}$ (2)

where $o$ is the input of dimension $N$, $c$ is the output of dimension $M$,
$\Phi$ is the $N\times M$ weight matrix of the layer (also called dictionary),
and $\lambda$ is a hyper-parameter setting the scale of the Laplacian prior.
Choosing a Laplacian prior is motivated by the fact that it promotes sparsity
in the output neural code in a way similar to how sparsity is induced in
networks of Spiking Leaky Integrate-and-Fire neurons, modelling cortical
neural activity [9].

Under Sparse Coding (2), inference of $c$ and learning of $\Phi$ is carried
via [9]:

$C,\Phi=\arg\min_{C,\Phi}\sum_{l}||\Phi
c_{l}-o_{l}||_{2}^{2}+\lambda||c_{l}||_{1}\textbf{ with }C=\\{c_{l},\forall
l\\}$ (3)

which can be solved via Proximal Stochastic Gradient Descent [26], by
alternating between: a) the inference of $c_{l}$, given the current input
$o_{l}$ and the weight $\Phi$ and b) the learning of $\Phi$, given the current
$c_{l}$ and $o_{l}$.

Hence, we instantiate Hebbian layers as the dynamical system given in (4),
where $T$ denotes the transpose, $\eta_{c}$ is the coding rate, $\eta_{d}$ is
the learning rate and $\textbf{Prox}_{\lambda||.||_{1}}$ is the proximal
operator to the $l_{1}$ norm (non-linearity) [27]. For each input $o_{i}$, the
neural and weight dynamics of the Hebbian network follows the update rules in
(4) for an arbitrary number of iterations (set to $100$ in this work as a good
balance between speed of convergence and convergence quality), in order to
infer the corresponding $c_{l}$ and learn $\Phi$ [9].

$\begin{cases}c_{l}\leftarrow\textbf{Prox}_{\lambda||.||_{1}}\\{c_{l}-\eta_{c}\Phi^{T}(\Phi
c_{l}-o_{l})\\}\\\ \Phi\leftarrow\Phi-\eta_{d}(\Phi
c_{l}-o_{l})c_{l}^{T}\end{cases}$ (4)

with $\textbf{Prox}_{\lambda||.||_{1}}$ acting as the neural non-linearity:

$\textbf{Prox}_{\lambda||.||_{1}}(c_{i})=\text{sign}(c_{i})\max(0,|c_{i}|-\eta_{c}\lambda),\forall
i$ (5)

From a neural point of view, the dynamical system of (4) can be implemented as
the network architecture in Fig. 1, where all weight updates follow the
standard Hebbian rule (1) [9].

Figure 1: Baseline Hebbian network architecture used in this work. The
dynamics of the network follow (4) and minimize (3), given subsequent input
vectors $o$. Each layer possesses its own weight matrix $\Phi,\Psi$ which
evolve through Hebbian plasticity ($\Psi\sim\Phi^{T}$ in (4), as an
independent, local set of weights).

##  3 Active Inference in Hebbian Learning Networks

In this Section, we show how the Hebbian network described above in Section 2
is utilized in order to build an AIF system. First, we describe how
Variational Free Energy minimization can be performed by a cascade of two
Hebbian networks: a state-transition network predicting the next latent states
given the previous ones, and a posterior network providing latent states given
input observations. Crucially, it is shown that Free Energy minimization
necessitates top-down Hebbian learning connections from the state-transition
network towards the posterior network, steering the posterior output activity
towards the state-transition output during learning. Then, we show how the
Expected Free Energy is computed by generating state transition roll-outs.

###  3.1 Minimizing the Variational Free Energy

The Variational Free Energy can be decomposed as [17] [28] (where $\mathbb{E}$
denotes the expected value):

$\mathcal{F}=D_{KL}[q_{\Phi_{P}}(s_{l}|o_{l-1},a_{l-1})||p_{\Phi_{S}}(s_{l}|s_{l-1},a_{l-1})]-\mathbb{E}_{q}[\log(p_{\Phi_{L}}(o_{l}|s_{l}))]$
(6)

with the parametrized densities $q_{\Phi_{P}},p_{\Phi_{S}},p_{\Phi_{L}}$
described in Section 1. Since the Hebbian network architecture used in this
work intrinsically provides a mean to reconstruct its input $x_{l}$ in (3)
(i.e., likelihood modelling) from its produced latent code $c_{l}$ in (3)
(i.e., posterior modelling), using the same dictionary parameter matrix $\Phi$
in (3) that was used to generate $c_{l}$ via (4), we have $\Phi_{L}=\Phi_{P}$
in (6) and we will solely use $\Phi_{P}$ below to denote the posterior weight
matrix.

Under the assumption of Gaussian likelihood with identity covariance in (2),
the KL divergence $D_{KL}$ in $\mathcal{F}$ can be simplified to [29]:

$\mathcal{F}\sim||\Phi_{S}c_{S,l}-\\{s_{l}(\Phi_{P}),a_{l}\\}||_{2}^{2}+||\Phi_{P}s_{l}-\\{o_{l},a_{l}\\}||_{2}^{2}$
(7)

where $s_{l}(\Phi_{P})$ explicits the dependency of $s_{l}$ on $\Phi_{P}$
($s_{l}$ being the posterior network output activity) and $c_{S}$ denotes the
output activity of the state-transition network given $s_{l}$. The Free Energy
in (7) must be minimized with regard to the state transition weights
$\Phi_{S}$ and the posterior weights $\Phi_{P}$ during learning:

$\Phi_{S},\Phi_{P}=\arg\min_{\Phi_{S},\Phi_{P}}||\Phi_{S}c_{S,l}-\\{s_{l}(\Phi_{P}),a_{l}\\}||_{2}^{2}+||\Phi_{P}s_{l}-\\{o_{l},a_{l}\\}||_{2}^{2},\forall
l$ (8)

This indicates that it is not only the state transition model that must be
steered towards the posterior model, but also, the posterior model must be
steered towards the output of the state-transition network. This effect can be
achieved by re-formulating the optimization in (8) as:

$\begin{cases}\Phi_{S}=\arg\min_{\Phi_{S}}||\Phi_{S}c_{S,l}-\\{s_{l},a_{l}\\}||_{2}^{2},\forall
l&\text{(a)}\\\
\Phi_{P}=\arg\min_{\Phi_{P}}||\Phi_{P}s_{l}-\\{o_{l},a_{l}\\}||_{2}^{2}+||\Phi_{P}(\Phi_{S}c_{S,l})-\\{o_{l},a_{l}\\}||_{2}^{2}&\text{(b)}\end{cases}$
(9)

Intuitively, the right-hand term in (9 b) steers the posterior model towards
the state-transition model by first re-projecting the output activity of the
state-transition network $c_{S}$ into the latent space as $\Phi_{S}c_{S}$
(considering $\Phi_{S}$ fixed). Then, minimizing
$||\Phi_{P}(\Phi_{S}c_{S,l})-\\{o_{l},a_{l}\\}||_{2}^{2}$ modifies $\Phi_{P}$
in order to steer its posterior output $s_{l}$ towards the re-projected state-
transition activity $\Phi_{S}c_{S}$ (considering $\\{o_{l},a_{l}\\}$ fixed).

####  3.1.1 State-Transition Model

Inspired by prior work on dictionary-based sequence modeling [30], we
implement the transition model
$p_{\Phi_{S}}(s_{l}|\tilde{s}_{l-1},\tilde{a}_{l-1})$ as an auto-regressive
Hebbian network (see Fig. 2 a), taking as input a sequence of state-and-action
history $\tilde{s}_{l-1}=[s_{l-1},\ldots,s_{l-L_{buf}}]$,
$\tilde{a}_{l-1}=[a_{l-1},\ldots,a_{l-L_{buf}}]$ and inferring the next state
$\tilde{s}_{l}=[s_{l},\ldots,s_{l-L_{buf}}]$ as the re-projection of its
internal sparse code $c_{S}$ in the input space through the network weights
$\Phi_{S}$:

$\tilde{s}_{l}=\Phi_{S}c_{S,l}$ (10)

where $\Phi_{S,j}$ and $c_{S,j}$ respectively denote the weight vector and the
sparse code of each layer $j$ in the state transition network. Therefore, the
state-transition network effectively projects the $L_{buf}$ previous states
(noted $\tilde{s}_{l-1}$) into a common internal sparse code $c_{S}$ and
reconstructs the next states $\tilde{s}_{l}$ by re-projection of $c_{S}$ into
the input space.

The state-transition network learns its weights $\Phi_{S}$ following (9 a) and
infers its output activity $c_{S}$ via sparse coding (see Section 2):

$\Phi_{S},c_{S,l}=\arg\min_{\Phi_{S},c_{S,l}}||\Phi_{S}c_{S,l}-\tilde{s}_{l}||_{2}^{2}+\lambda_{P}||c_{S,l}||_{1},\forall
l$ (11)

where $\lambda_{P}$ is a parameter that sets the strength of the sparsity of
the state-transition output activity. (11) can therefore be implemented via
the Sparse Coding-based Hebbian learning ensemble described in Section 2. This
auto-regressive strategy enables the network to learn state predictions using
Hebbian learning, without the need for non-bio-plausible back-propagation
through time (BPTT) [30], [31].

In order to prevent the vanishing or exploding of the state transition model
when producing roll-outs further in time, we regularize the norm of the
reconstructed states $s_{l}$ to an arbitrary magnitude $\alpha$ using (12). We
keep $\alpha=5$ in our experiments in Section 4, giving a good balance for the
dynamic range of the network output activity (adjusted empirically).

$s_{l}=\alpha\frac{s_{l}}{||s_{l}||_{2}}$ (12)

Figure 2: Hebbian Active Inference Architecture. a) The state-transition
network takes as input the $L_{buf}$ previous latent states produced by the
posterior network, and projects them onto its internal representation $c_{S}$
via the learned $\Phi_{S}$. When producing roll-outs, the state-transition
network estimates the next state $\hat{s}_{l}$ by re-projecting the output
activity $c_{S}$ due to $[s_{l-L_{buf}},\ldots,s_{l-1}]$ back onto the input
space via (10). b) The posterior network takes observation-action pairs as
input and produces latent states $s$ (corresponding to the output in Fig. 1).
In addition to the Hebbian mechanisms depicted in Fig. 1, the weights
$\Phi_{P}$ of the posterior network are also subject to a top-down Hebbian
learning mechanism for minimizing the second term in (9 b).

####  3.1.2 Posterior Model

Similar to the state-transition model, we use a Hebbian ensemble as posterior
model, where the internal sparse code $c_{P}$ (see Fig. 2 b) is identified as
the hidden state $s_{l}\equiv c_{P,l}$ inferred by the posterior network
$q_{\nu}(s_{l}|o_{l-1},a_{l-1})$, given the observation and action pair
$\\{o_{l-1},a_{l-1}\\}$ in (2).

Therefore, the posterior network learns its weights $\Phi_{P}$ following (9 b)
and infers its output activity $s_{l}=c_{P,l}$ via sparse coding (see Section
2):

$\Phi_{P},c_{P,l}=\arg\min_{\Phi_{P},c_{P,l}}\underbrace{||\Phi_{P}c_{P,l}-\\{o_{l},a_{l}\\}||_{2}^{2}+\lambda_{Q}||c_{P,l}||_{1}}_{\text{Standard
Sparse
Coding}}+\overbrace{||\Phi_{P}(\Phi_{S}c_{S,l})-\\{o_{l},a_{l}\\}||_{2}^{2}}^{\text{{Top-
down Connection}}}$ (13)

for all $l$, where $\lambda_{Q}$ sets the strength of the sparsity of the
posterior output activity. The left-hand standard sparse coding term in (13)
can be implemented via the Hebbian learning ensemble described in Section 2,
while the right-hand term in (13) can be implemented using top-down
connections from the state-transition output activity $c_{S}$ towards the
posterior network, via the state-transition weight matrix $\Phi_{S}$ (see Fig.
2 b).

Finally, here again, we apply the homeostasis rule (12) to the inferred
posterior state, effectively constraining $s_{l}$ to lie on the
$\alpha$-sphere manifold.

###  3.2 Minimizing the Expected Free Energy

Given a policy $\pi$, the Expected Free Energy $G(\pi)$ (EFE) can be written
as [32]:

$G(\pi)=\sum_{l}\mathbb{E}_{q(o_{l},s_{l}|\pi)}[\log q(s_{l}|\pi)-\log
p(s_{l},o_{l}|\pi)]\\\
=\sum_{l}-H\\{q(s_{l}|\pi)\\}-\mathbb{E}_{q(o_{l},s_{l}|\pi)}[\log
p(s_{l},o_{l}|\pi)]$ (14)

where $H$ denotes the Shannon entropy. It can be seen in (14) that selecting a
policy that minimizes the EFE entails the maximization of the posterior
entropy (promoting exploration) and the joint posterior over the states and
observations (reaching the desired goal) [32].

In order to reach the desired goal, we produce roll-outs of states $s_{l}$
given a certain policy and approximate the term $-\mathbb{E}_{q}[\log
p(s_{l},o_{l}|\pi)]$ to be minimized as:

$-\mathbb{E}_{q(o_{l},s_{l}|\pi)}[\log
p(s_{l},o_{l}|\pi)]\sim||s_{l}-s^{*}||_{2}^{2}$ (15)

where $s^{*}$is the desired state that the agent must reach, corresponding to
a desired observation (e.g., the agent’s position). Since the observation
$o_{l}$ can encompass more than just the goal to be reached (i.e., the
observation $o_{l}$ could be both the position and the velocity of an agent,
even though the desired goal is to reach a specific position regardless of the
velocity), we compute the desired goal state $s^{*}$ as:

$s^{*}=\arg\max_{s}\int_{\omega\in D_{\omega}}q(s|\Omega^{*},\omega)d\omega$
(16)

where $\Omega^{*}$ contains all observations that must be reached in order to
attain the desired goal and $\omega$ designates all observation modalities
that are not taking part in defining the goal that must be reached, with
$D_{\omega}$ their domain of definition. In practice, (16) is estimated by
averaging the output of the posterior network, while sweeping $\omega$ for a
grid of possible values and keeping $\Omega^{*}$ fixed.

Regarding the exploration term in (14), our Hebbian network does not directly
allow the estimation of the entropy $H\\{q(o_{l},s_{l}|\pi)\\}$, since the
network does not infer standard deviations as in a variational auto-encoder
(VAE) [17]. We propose to replace the maximization of the entropy
$H\\{q(o_{l},s_{l}|\pi)\\}$ with a surrogate term, crafted to promote
exploration as well. As a surrogate for $H\\{q(o_{l},s_{l}|\pi)\\}$, we choose
to maximize the variance (noted Var) of the state trajectory $s_{l},\forall
l=1,...,L$ along time during the roll-outs. Intuitively, a state trajectory
that presents lots of variation in time will promote the exploration of new
states, providing a similar qualitative effect as maximizing
$H\\{q(o_{l},s_{l}|\pi)\\}$. Therefore, we select the policy $\pi$ such that
the distance to the desired state is minimized, while achieving a state
trajectory variance larger than a certain threshold $t_{v}$.

$\pi^{*}=\arg\min_{\pi}\mathcal{G}(\pi)=\sum_{l=1}^{L}||s_{l}-s^{*}||_{2}^{2}\hskip
10.0pt\text{ s.t. }\hskip
10.0pt\text{Var}(||s_{l}-s^{*}||_{2}^{2},l=1,...,L)\geq t_{v}$ (17)

Given a set of $N_{p}$ policies to try, $t_{v}$ can be determined in an
adaptive way as:

$t_{v}=\beta\times\frac{1}{2}[\boldsymbol{\max_{\pi}}(\text{Var}(||s_{l}(\pi)-s^{*}||_{2}^{2},\forall
l))+\boldsymbol{\min_{\pi}}(\text{Var}(||s_{l}(\pi)-s^{*}||_{2}^{2},\forall
l))]$ (18)

where $\beta$ is the strength hyper-parameter (empirically set to $0.5$ in our
experiments reported below).

##  4 Experimental Results

The aim of our experimental studies is to determine i) how the main network
hyper-parameters (number of neurons, sparsity in output activity,…) impact the
success rate of the proposed Hebbian AIF system; ii) to what extent Hebbian
AIF is robust when learning without using a replay buffer and iii) how Hebbian
AIF compares to Q-learning (which uses dense rewards versus unsupervised
learning in Hebbian AIF).

###  4.1 Mountain Car Environment

We perform experiments in the Mountain Car environment from the OpenAI gym
suite [21]. In this task, a car starts at a random position at the bottom of a
hill and is expected to reach the top of a mountain within $200$ time steps.
The agent is subject to gravity and cannot reach the goal trivially, just by
accelerating towards it. Rather, the agent must learn to gain momentum before
accelerating towards the goal.

In this environment, the x-axis position $x$ and the velocity $v_{x}$ of the
car constitute the input observations to the Hebbian AIF network. Before
feeding the observation tuple $(x,v_{x})$ to our Hebbian network, we normalize
$(x,v_{x})$ using (19) in order to equalize the dynamic range of the position
and velocity signals:

$\begin{cases}x\xleftarrow{}\frac{x-\mu_{x}}{\sigma_{x}}\\\
v_{x}\xleftarrow{}\frac{x-\mu_{v_{x}}}{\sigma_{v_{x}}}\end{cases}$ (19)

where $(\mu_{x},\sigma_{x})$ and $(\mu_{v_{x}},\sigma_{v_{x}})$ denote the
mean and standard deviation of the position and velocity signals respectively
(estimated during random environment runs).

We use an action space constituted by two discrete actions: accelerate to the
left and accelerate to the right. In addition, each action is repeated for
$10$ consecutive time steps once selected during the Expected Free Energy
minimization in (17).

In order to compute the Expected Free Energy, we generate roll-outs of $L=200$
time step predictions for $100$ different random policies $\pi^{j},j=1,...100$
with equal probability of selecting the accelerate to the left or the
accelerate to the right actions.

As learning rate for the Hebbian learning mechanism (4), we use
$\eta_{d}=10^{-4}$ with a decay rate of $0.8$ applied at the end of each
successful episode, i.e. if the episode terminates successfully,
$\eta_{d}\xleftarrow{}\eta_{d}\times 0.8$ (else no decay is applied on
$\eta_{d}$). All weights are initialized randomly from a normal distribution
with standard deviation $0.01$.

In the remainder of this Sections, we perform all our experiments using a
10-fold validation approach, by reporting the success rate curves as averages
over 10 different runs (with $35$ episodes per runs), with different random
network initializations. For each run, we compute the success rate curve using
a moving average window of size $5$, and report the mean success rate curve by
averaging over the $10$ runs, alongside with its standard deviation (see e.g.
Fig. 3). We will now study the impact of the various network hyper-parameters
on the achieved success rates.

###  4.2 Impact of the Number of Neurons in the Posterior and State Transition
Networks

Fig. 3 and 4 show the effect of sweeping the number of coding neurons $M_{Q}$
and $M_{P}$ in both the posterior and state-transition networks. Fig. 3 shows
that for $M_{Q}<8$, the success rate is sub-optimal, but reaches a steady
plateau around $M_{Q}=8$ (orange curve in Fig. 3). Then, as $M_{Q}$ is
increased for $M_{Q}>8$, the success rate becomes sub-optimal again, with dips
in the performance along the episodes (e.g., red curve in Fig. 3). This
phenomenon can be explained as follows: for $M_{Q}<8$, the posterior network
does not have enough parameters to capture the input dynamics into its latent
space and under-fits, while for $M_{Q}>8$, the posterior network starts over-
fitting, reducing the success rate again.

Figure 3: Impact on the success rate when changing the number of neurons
$M_{Q}$ in the posterior network.

Regarding the state-transition network, Fig. 4 shows that the higher the
number of neurons $M_{P}$, the flatter the success rate curves become, leading
to higher performance. The state transition network does not seem to over-fit
as $M_{P}$ is increased (for $\lambda_{P}=10^{-4}$ kept fixed). Rather, Fig. 4
indicates that a higher state-transition network capacity is beneficial for
capturing important dynamics in the latent space, at the output of the
posterior network.

Figure 4: Impact on the success rate when changing the number of neurons
$M_{P}$ in the state transition network.

###  4.3 Impact of the Sparsity of the Output Activity in the Posterior and
State Transition Networks

Fig. 5 and 6 show the effect of sweeping the sparsity-defining hyper-
parameters $\lambda_{Q}$ and $\lambda_{P}$ in both the posterior and state-
transition networks. For the posterior network, Fig. 5 shows that the success
rate performance initially grows as $\lambda_{Q}$ is increased from
$\lambda_{Q}=10^{-6}$ to $\lambda_{Q}=10^{-5}$. Doing so, the non-linearity of
the posterior network is increased, better capturing observation features into
its latent space. Then, as $\lambda_{Q}$ grows past $\lambda_{Q}=10^{-4}$, the
success rate degrades again, indicating a too strong posterior network non-
linearity.

Figure 5: Impact on the success rate when changing the sparsity hyper-
parameter $\lambda_{Q}$ in the posterior network.

Regarding the state-transition network, Fig. 6 shows that the lower
$\lambda_{P}$, the higher the success rate becomes. This suggests that making
the state-transition network more linear (i.e., lower $\lambda_{P}$) better
captures the dynamics of the latent space produced by the posterior network
(other parameters kept fixed).

Figure 6: Impact on the success rate when changing the sparsity $\lambda_{P}$
in the state transition network.

###  4.4 Impact of the Time-Lag Buffer Length on Task Performance

Fig. 7 shows how the length $L_{buf}$ of the time-lag buffer impacts the
achieved success rate. Initially, as $L_{buf}$ increases, the success rate
increases as well, due to an increased availability of past latent states used
by the state-transition network to estimate the next expected state. Then, as
$L_{buf}$ is further increased for $L_{buf}>20$, the success rate drops again
due to the addition of latent states from deep in the past that are less
useful for estimating the present dynamics.

Figure 7: Success rate when changing the time-lag buffer length $L_{buf}$.

###  4.5 Comparing Hebbian AIF against the Use of a Replay Buffer and against
Q-learning

Fig. 8 compares the success rate obtained using our proposed Hebbian AIF
system against a) the use of a replay buffer during learning and b) the use of
a Q-learning agent. Experience replay is done by saving the history of
observation-action pairs in a buffer after each episode. After the end of the
episode, a past experience is randomly selected and used to train the Hebbian
AIF system for one episode.

Regarding the Q-learning setup, we use a standard Q-table learning approach
[22], with the python implementation proposed in [33].

Fig. 8 shows that our Hebbian AIF system converges much faster than the
Q-learning system and behaves in a comparable manner to the Hebbian AIF setup
with a replay buffer. Indeed, the Q-learning agent needs much more episodes in
order to converge, despite the fact that it utilizes the dense rewards
provided by the Mountain Car environment [21] (versus unsupervised learning in
the case of Hebbian AIF). This confirms prior observations about the efficient
convergence of AIF systems, due to their ability to learn a generative model
of environment dynamics (versus supervised learning of a Q-table) [17].

Finally, it is interesting to note that, compared to the Deep AIF results
reported in [17] (using a fully-connected 2-hidden-layer network trained
through backprop), the Hebbian AIF system proposed in this work eventually
reaches $\sim 100\%$ success rate (see red curve in Fig. 6) while the system
in [17] reaches $\sim 95\%$, motivating further investigations of Hebbian
learning for AIF systems.

Figure 8: Hebbian AIF versus the use of a replay buffer and Q-learning (a).
Q-learning needs two orders of magnitude more episodes in order to converge
(b).

##  5 Conclusion

This paper has investigated how neural ensembles equipped with local Hebbian
plasticity can perform active inference for the control of dynamical agents.
First, a Hebbian network architecture performing joint dictionary learning and
sparse coding has been introduced for implementing both the posterior and the
state-transition models forming our generative Active Inference system. Then,
it has been shown how Free Energy minimization can be performed by the
proposed Hebbian AIF system. Finally, extensive experiments for parameter
exploration and benchmarking have been performed to study the impact of the
network parameters on the task performance. Experimental results on the
Mountain Car environment show that the proposed system outperforms the use of
Q-learning, while not requiring the use of a replay buffer during learning,
motivating future investigations of using Hebbian learning for designing
active inference systems.

## Acknowledgement

This research was partially funded by a Long Stay Abroad grant from the
Flemish Fund of Research - Fonds Wetenschappelijk Onderzoek (FWO) - grant
V413023N. This research received funding from the Flemish Government under the
“Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

## References

  * [1] Bruno A. Olshausen, David J. Field (1997). ”Sparse coding with an overcomplete basis set: A strategy employed by V1?.” Vision Research, 37(23), 3311-3325.
  * [2] Fang, M.S., Mudigonda, M., Zarcone, R., Khosrowshahi, A., Olshausen, B. (2022). ”Learning and Inference in Sparse Coding Models With Langevin Dynamics.” Neural Computation, 34(8), 1676-1700.
  * [3] Lee, H., Battle, A., Raina, R., Ng, A. (2006). ”Efficient sparse coding algorithms.” In Advances in Neural Information Processing Systems. MIT Press.
  * [4] Ali Safa, Ilja Ocket, André Bourdoux, Hichem Sahli, Francky Catthoor, Georges Gielen. (2022). ”A New Look at Spike-Timing-Dependent Plasticity Networks for Spatio-Temporal Feature Learning.”
  * [5] Friston, K., Kiebel, S. (2009). ”Predictive coding under the free-energy principle.” Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 364, 1211-21.
  * [6] Friston, K. Does predictive coding have a future?. Nat Neurosci 21, 1019–1021 (2018). https://doi.org/10.1038/s41593-018-0200-7
  * [7] Umais Zahid, Qinghai Guo, Zafeirios Fountas.(2023). ”Predictive Coding as a Neuromorphic Alternative to Backpropagation: A Critical Evaluation.”
  * [8] Olshausen, B., Field, D. (1996). ”Emergence of simple-cell receptive field properties by learning a sparse code for natural images.” Nature, 381, 607-609.
  * [9] Safa, A., Ocket, I., Bourdoux, A., Sahli, H., Catthoor, F., Gielen, G. (2022). ”Event Camera Data Classification Using Spiking Networks with Spike-Timing-Dependent Plasticity.” In 2022 International Joint Conference on Neural Networks (ĲCNN) (pp. 1-8).
  * [10] Guo-qiang Bi, Mu-ming Poo (1998). ”Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type.” Journal of Neuroscience, 18(24), 10464–10472.
  * [11] Rao, R., Ballard, D. (1999). ”Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.” Nature neuroscience, 2(1), 79–87.
  * [12] A. Safa, I. Ocket, A. Bourdoux, H. Sahli, F. Catthoor and G. G. E. Gielen, ”STDP-driven Development of Attention-based People Detection in Spiking Neural Networks,” in IEEE Transactions on Cognitive and Developmental Systems, 2022, doi: 10.1109/TCDS.2022.3210278.
  * [13] Neftci, E., Das, S., Pedroni, B., Kreutz-Delgado, K., Cauwenberghs, G. (2014). ”Event-driven contrastive divergence for spiking neuromorphic systems.” Frontiers in Neuroscience, 7.
  * [14] Dmitry Krotov, John J. Hopfield (2019). ”Unsupervised learning by competing hidden units.” Proceedings of the National Academy of Sciences, 116(16), 7723-7731.
  * [15] Parr, T., Pezzulo, G., Friston, K. (2022). ”Active Inference: The Free Energy Principle in Mind, Brain, and Behavior.” The MIT Press.
  * [16] Isomura, T., Shimazaki, H. Friston, K.J. ”Canonical neural networks perform active inference.” Commun Biol 5, 55 (2022). https://doi.org/10.1038/s42003-021-02994-2
  * [17] Çatal, O., Wauthier, S., De Boom, C., Verbelen, T., Dhoedt, B. (2020). ”Learning Generative State Space Models for Active Inference.” Frontiers in Computational Neuroscience, 14.
  * [18] Ueltzhöffer, K. ”Deep active inference.” Biol Cybern 112, 547–573 (2018). https://doi.org/10.1007/s00422-018-0785-7
  * [19] Fountas, Z., Sajid, N., Mediano, P., Friston, K. (2020). ”Deep active inference agents using Monte-Carlo methods.” In Advances in Neural Information Processing Systems (pp. 11662–11675). Curran Associates, Inc..
  * [20] Van de Maele, T., Verbelen, T., Çatal, O., De Boom, C., Dhoedt, B. (2021). ”Active Vision for Robot Manipulators Using the Free Energy Principle.” Frontiers in Neurorobotics, 15.
  * [21] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W. (2016). ”Openai gym.” arXiv preprint arXiv:1606.01540.
  * [22] Sutton, R., Barto, A. (2018 ). ”Reinforcement Learning: An Introduction.” The MIT Press.
  * [23] Ororbia, A. G., Mali, A. (2022). ”Backprop-Free Reinforcement Learning with Active Neural Generative Coding.” Proceedings of the AAAI Conference on Artificial Intelligence, 36(1), 29-37.
  * [24] Alexander Ororbia, Ankur Mali. ”Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems.” IEEE International Conference on Robotics and Automation (ICRA) 2023.
  * [25] Yuchen Liang, Chaitanya Ryali, Benjamin Hoover, Leopold Grinberg, Saket Navlakha, Mohammed J Zaki, Dmitry Krotov (2021). ”Can a Fruit Fly Learn Word Embeddings?.” In International Conference on Learning Representations.
  * [26] Ablin, P., Moreau, T., Massias, M., Gramfort, A. (2019). ”Learning step sizes for unfolded sparse coding.” In Advances in Neural Information Processing Systems. Curran Associates, Inc.
  * [27] Tsung-Han Lin, Ping Tak Peter Tang (2019). ”Sparse Dictionary Learning by Dynamical Neural Networks.” In International Conference on Learning Representations.
  * [28] Friston, K. ”The free-energy principle: a unified brain theory?.” Nat Rev Neurosci 11, 127–138 (2010). https://doi.org/10.1038/nrn2787
  * [29] J. R. Hershey and P. A. Olsen, ”Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models,” 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07, Honolulu, HI, USA, 2007, pp. IV-317-IV-320, doi: 10.1109/ICASSP.2007.366913.
  * [30] Kim, E., Lawson, E., Sullivan, K., Kenyon, G. (2019). ”Spatiotemporal Sequence Memory for Prediction Using Deep Sparse Coding.” In Proceedings of the 7th Annual Neuro-Inspired Computational Elements Workshop. Association for Computing Machinery.
  * [31] P. J. Werbos, ”Backpropagation through time: what it does and how to do it,” in Proceedings of the IEEE, vol. 78, no. 10, pp. 1550-1560, Oct. 1990, doi: 10.1109/5.58337.
  * [32] Schwartenbeck, P., FitzGerald, T., Dolan, R., Friston, K. (2013). ”Exploration, novelty, surprise, and free energy minimization.” Frontiers in Psychology, 4.
  * [33] https://gist.github.com/gkhayes/3d154e0505e31d6367be22ed3da2e955 (accessed May 1 2023)


# The Flipped Classroom model for teaching Conditional Random Fields in an NLP
course

Manex Agirrezabal
Centre for Language Technology (CST) - Department of Nordic Studies and
Linguistics
University of Copenhagen / Københavns Universitet
Emil Holms Kanal 2
2300 Copenhagen (Denmark)
<EMAIL_ADDRESS>
###### Abstract

In this article, we show and discuss our experience in applying the flipped
classroom method for teaching Conditional Random Fields in a Natural Language
Processing course. We present the activities that we developed together with
their relationship to a cognitive complexity model (Bloom’s taxonomy). After
this, we provide our own reflections and expectations of the model itself.
Based on the evaluation got from students, it seems that students learn about
the topic and also that the method is rewarding for some students.
Additionally, we discuss some shortcomings and we propose possible solutions
to them. We conclude the paper with some possible future work.

##  1 Introduction

In this article we would like to provide experience on developing a flipped
classroom environment in a Natural Language Processing course. The most common
approach of teaching involves a teacher “pouring” knowledge to its students,
as if students were plants and knowledge was water. There is a tendency,
though, to make classes more active and interactive, in which communication
does not only happen from the teacher to the students, but also from student
to student.

Bloom’s taxonomy111https://www.flickr.com/photos/vandycft/29428436431 Bloom et
al. (1956); Anderson et al. (2001) is a model that describes the cognitive
load of different types of work or activities. On the one hand, some
activities, such as remembering or understanding specific characteristics of,
for instance, a model of any kind, would be considered to require a low
cognitive load. On the other hand, evaluating which model is best by
considering the characteristics of each, could be considered to be in a higher
level of complexity in Bloom’s taxonomy.

If we want our students to be more knowledgeable and reflective, we need to go
beyond the lower levels in Bloom’s taxonomy. The traditional approach of
introducing topics in a lecture would make students remember and understand
the covered topics. We believe, though, that with these teaching practices
there is a time limitation to go beyond the first two levels of Bloom’s
taxonomy.

A more efficient approach could be to ask students to work on a set of topics
beforehand, get prepared, and then, work on activities that involve a higher
cognitive load. These activities could involve applying and analyzing the
acquired knowledge to other aspects. In this article, we present our
experience in teaching Conditional Random Fields as a Flipped Classroom. We
teach this in a course at the Master level about Natural Language Processing
(NLP).

The article is structured as follows. First we introduce the flipped classroom
method and some challenges. After that, we present the program, course and
lecture in which the new teaching method will be used. Then, we discuss the
characteristics of the implied student Ulriksen (2009) and also the evaluation
method that we use. Later, we describe the lectures structure and the
different activities and we classify them according to Bloom’s taxonomy. We
then include some discussion, based on the experience. Finally, we conclude
the work and suggest some possible future directions.

##  2 Flipped Classroom: Are we going to turn around the desks?

Flipped Classroom Lage et al. (2000); Brame (2013) is a teaching approach in
which students get first exposure to the material of a lecture outside of
class, and the in-class activities involve applying the learned content.
Provision of lecture material can be done in a number of ways, such as reading
material, video lectures as slideshows, podcasts, and so on.

Students are expected to do the homework, and the majority of the in-class
activities fully depend on that. Because of that, the teacher should make sure
that students do their homework, because even though we name it in various
ways, watching lectures or reading articles is still homework Nielsen (2012),
and the challenge of making students accomplish with that is still there. A
possible idea for making sure that students do the reading homework could be
to ask them to do a quiz or reward them somehow.

Apart from challenges regarding students, we may not forget that all the
activities, homework, readings, and so on have to be retrieved, selected or
produced. Furthermore, as the content covered in class is more complex, the
teacher will have to be more prepared. All this results in a larger working
load in the preparation of the class and its activities.

##  3 Background about program, course and lecture

In this section, we briefly introduce the M. Sc. program, the course and the
specific lecture in which we will be focusing on.

### The IT & Cognition program

The IT & Cognition program at the University of Copenhagen is an international
and interdisciplinary program222https://studies.ku.dk/masters/it-and-
cognition/ that accepts a small group of students every year. The program
covers three main areas: Natural Language Processing, Image Processing and
Cognitive Science.

In the first semester, the students acquire the required basic skills for
their further development in the specialization area in which they are
interested. Scientific Programming, Language Processing I, Cognitive Science I
and Vision and Image Processing are mandatory subjects that cover these basic
skills.

In the second semester students continue learning about Language Processing
and Cognitive Science in further specialized courses, and they also get an
introduction to Data Science. Besides, they start their specialization process
with an elective course.

The third and forth semesters are devoted to a third course about Cognitive
Science and three different electives and after that, students work on their
thesis project.

### Language Processing 1 and 2 (LP1 and LP2)

These courses are taught during a whole academic year (two semesters). There
is one lecture per week which lasts for two hours. Each course, LP1 and LP2,
goes through fourteen weeks in the fall and spring semesters, respectively.

The Intended Learning Outcomes (ILOs) of both courses are quite similar. The
main differences are in the degree of depth in which topics are covered. On
the one hand, the first course offers basic knowledge about different tasks
relevant to Natural Language Processing and their relationship to current
society. On the other hand, the second course is more geared towards the
development of more advanced algorithms and their application in more specific
tasks.

#### Assessment method

We assess students by asking them to work on a predefined topic. The common
procedure is to perform experiments for a relevant NLP task and afterwards,
they should write a scientific article reporting on these experiments.

### Lecture: Conditional Random Fields

One of the topics covered in the second Language Processing course covers
knowledge about sequence tagging. We cover Hidden Markov Models, Maximum
Entropy Markov Models and Conditional Random Fields, as sequential tagging
models. This last model will be covered in one and a half sessions. In the
first session (2 hours) we will cover theoretical questions and students are
expected to understand them. In the following week, there will be one hour,
and students will get hands-on practice about Conditional Random Fields.

The lecture itself has the goal of providing the students an understanding of
how Maximum Entropy Markov Models (MEMMs) McCallum et al. (2000) and
Conditional Random Fields (CRFs) Lafferty et al. (2001) make predictions
compared to hard classification methods. Additionally, they should understand
a common problem of MEMMs (Label Bias problem Bottou (1991); Lafferty et al.
(2001)) and how CRFs solve such limitation. Finally, students should also be
able to use CRFs for their own research after the lectures.

##  4 The implied student

The background of our student group Ulriksen (2009) is very heterogeneous.
Some people may have a Computer Science related background, and therefore,
sufficient experience in programming. Other students do not have the same
background, but they are strong in other aspects, such as linguistics,
neuroscience or psychology. Besides, at the time that our analyzed lecture
happens, students have already received lectures about programming (first
semester), so therefore, this level gap should be significantly smaller.

##  5 Evaluation method

In recent years, the usual teaching practice in the Language Processing series
has been lectures. As the goal of this experiment is to check whether the
flipped classroom can support students in their learning or not, we will
implement this teaching method for the section about Conditional Random
Fields, and analyze how students feel about it.

We evaluate this teaching practice by asking students to fill in a survey. In
the survey we ask students about their general knowledge about some topics
(MEMMs, CRFs, Label Bias problem) but also about whether they would be able to
use CRFs for their own work. Please find below the questions that we asked in
the questionnaire:

  1. 1.

Do you know what a MEMM is (Maximum Entropy Markov Model)?

  2. 2.

Do you know what a CRF is (Conditional Random Field)?

  3. 3.

Do you know what the Label Bias problem is?

  4. 4.

Do you feel capable of using a CRF for your own problems, such as developing a
Named Entity Recognition system?

  5. 5.

Do you feel that this structure (teaching style) is more rewarding?

  6. 6.

Do you feel that this structure is more demanding (mentally)?

The response to these questions could be either “Yes”, “Roughly” or “No”.
Finally, there are two questions related to the specific teaching method, in
which we ask students whether the teaching practice is more demanding
(mentally) and also whether it is more rewarding, compared to the lecturing
approach. The survey was made one week before the lecture day and after the
lecture was done.

##  6 Activities

In this section we describe the activities that we made for students before
the lectures, during the lectures and after the lectures. The activities will
be made public, hoping that they will be useful for other NLP teachers and/or
researchers.

###  6.1 Before lecture

Before the lecture, students were asked to watch two video lectures. The two
videos were available at the university learning platform and they were made
by ourselves. The first video covers Maximum Entropy Markov Models and we
introduce them by showing the relationship to Logistic Regression and Hidden
Markov Models. These last models (HMMs) were introduced two weeks before in
this same course. We use Maximum Entropy Markov Models as a middle step in
order to understand Conditional Random Fields, which are discussed in the
second video. We talk about the Label Bias problem and show how this is solved
by using global normalization in Conditional Random Fields.

Students should also read the paper that introduces Conditional Random Fields
Lafferty et al. (2001)333We believe that including an additional article about
Maximum Entropy Markov Models McCallum et al. (2000) is relevant and very
helpful for students. Unfortunately, we did not include it this year..

t | 0 | 1 | 2 | 3 | 4
---|---|---|---|---|---
WORDS | My | smartphone | worked | very | well
POS tags | PRP | N | VP | MOD | RB
Features | 1,2,0 | 0,10,0 | 0,6,1 | 0,4,0 | 0,4,0

###  6.2 In classroom (online)

As mentioned before, CRFs will be covered in one and a half lecture sessions.
These sessions are held online because of the current situation, following our
health authorities requirements.

Before we start the lecture, students are asked whether there are questions
regarding the reading and watching activities. We will briefly recapitulate
some aspects, such as where could sequential models like HMM, MEMM or CRFs be
useful: Part-of-Speech (POS) tagging, Named Entity Recognition (NER), and
besides, any other task that requires the production of tags for a given
sequence of elements.

Then, the goal of the remaining time in the session is threefold: (1) to
revise and get an understanding of why sequential models such as HMMs, MEMMs
or CRFs are more powerful than hard classification models, e.g. Maximum
Entropy, Support Vector Machines (SVM), and so on; (2) to make it clear what
the Label Bias problem is; and (3) to show how CRFs solve the Label Bias
problem by using global normalization. After this session, there will be time
to show CRFs working in practice, so that students get hands-on experience.

We describe below four different exercises that students will have to do in
small groups (3-4 people). These exercises have an increasing level of
complexity, as it will be seen.

#### Exercise 1

Understand how prediction is made in a Maximum Entropy POS tagger. Students
are given one sentence and the POS tags for each word in that sentence. They
are told that the model is trained using three very simple features:
isUpperCase, length_chars, endsInEd), and they have to simulate by hand how
predictions are made for each word. We also provide a list of 10 possible POS
tags. In order to make sure that the concepts about the weight matrix are
understood, we ask some checkpoint questions, such as the shape of the weight
matrix, provided that the input matrix has a size of 1x3 and the output matrix
has a size of 1x10.

Each group of students should provide two outputs to the teacher. Given an
input and a weight matrix,444we also provide the dot product output to save
time they should be able to see which POS tag would be returned by the model.
We also ask them to describe, in one sentence with their own words, how the
model produces the output for each word.

Considering Bloom’s taxonomy Bloom et al. (1956), this exercise could be
considered an exercise to remember, understand and apply concepts, and thus,
in the three lower levels.

####  6.2.1 Exercise 2

In this exercise, students are given the same exact sentence as before. The
difference is that the POS tagger with which the students will work is a
Maximum Entropy Markov Model (MEMM), and therefore, there is no independent
predictions.

Students already learned about the Viterbi algorithm for Hidden Markov Models
(HMM). The goal of this exercise is to remember how this algorithm works and
also to try to be aware which is the difference between HMMs and MEMMs, i.e.
the use of an extended set of features besides single words and tags.

In order to raise that awareness, the exercise is to go through the pseudo-
code of the Viterbi algorithm for HMMs (Jurafsky and Martin, 2008, p. 220),
understand it and find out which are the specific elements that have to be
changed, so that this works with MEMMs. As a hint in order to guess what
should be put there, we provide students a trellis of the example above. We
highlight one node in that trellis and discuss what its outgoing arcs
represent: A probability distribution
$P_{S\textquoteright}(S|0)=[P_{1},P_{2},...,P_{k}]$. The value of $k$ depends
on the number of states, i.e. output classes.

This exercise requires understanding the Viterbi algorithm for HMMs and
understanding what should be modified to make it work with MEMMs. As students
have to draw connections between these two models, we consider that the
cognitive load, based on Bloom’s taxonomy, is higher than in exercise 1.

Figure 1: The left plot shows the responses to the survey that we did one week
before the flipped classroom session started. The one on the right shows the
responses to the same questions, but after the session was over. Based on the
responses, we can say that students felt that they understood these four
relevant aspects. It looks like, though, that in the future we should
emphasize in the practical aspect of the taught models (Look at the right
column in the second figure).

#### Exercise 3

In the previous exercise, when we mention the probability distribution
$P_{S^{\prime}}(S|O)$, we mention that some of those probabilities could be
zero. Because of that, in this exercise we ask students to reflect on what
would happen if we have a node with many zero probabilities. For example, what
happens if a node has 2 non-zero probabilities? What if it has 10 non-zero
probabilities? We ask them this question to think about those probabilities
and make them aware of the Label Bias problem.

In Bloom’s taxonomy this exercise could be seen as evaluating and/or
analyzing, as students should be able to find out the problem by themselves.
We do not ask them to apply what they know, but to think about how having more
or less arcs could affect the probabilities of nodes, and thus, the final
sequence probabilities.

#### Exercise 4

As they already identified the problem, the last exercise is to suggest a
solution for that problem. They should analyze the trellis given in exercise 2
and think about a possible solution. Students are given 5-10 minutes to
discuss the topic. Afterwards, we discuss their possible solutions and if
nobody reaches the actual solution, we introduce global normalization (per
observation sequence), which is used in Conditional Random Fields.

In this case, the students would be in a similar scenario as the researchers
that found out the Label Bias problem. As such, we consider that this is at
the highest level of Bloom’s taxonomy, in which students discuss and
conjecture about a possible solution for the problem.

#### Practical applications

After covering the previous exercises, in the next session, we cover
Conditional Random Fields from a practical perspective, for which we show
students how to process text to use it with Conditional Random Fields. We also
discuss how to extract relevant features.

###  6.3 After classroom (exam project)

With regards to the final project, there will be a practical session in which
different methods, including CRFs, will be shown. The students will be then
able to apply the obtained knowledge.

##  7 Discussion and evaluation

Our expectation is that students will learn more, in a better way with a
similar effort (from their side). We think that the reward (for students) of
this teaching method is high, on the one hand. On the other hand, the required
time for preparation (for teachers) is higher.

The preparation time increase is directly related with the fact that the
topics that are covered are expected to be more complex. As students already
got lecturing as homework, then the in-class activities become more
complicated, and it is because of this that the teacher must have a deeper
understanding of the topic in question. Besides, in our specific case the
video lectures were made by us. We could save time by using the available
resources in online learning platforms, but we decided to make them ourselves,
making the preparation more time-consuming.

The evaluation, based on a survey that students had to answer, seems positive,
although there are issues. Considering how much students learned, we can say
that the learning goals were satisfied, as it can be seen in Figure 1. Each
column in the plots represents the answers of our students to a question
regarding knowledge on different concepts. They had to answer either Yes,
Roughly or No. As we can see, students did not have much expertise on the
topic before the lecture and preparation. Afterwards, the responses show that
students got the required understanding.

Besides, we also asked whether this teaching method is more rewarding. 7 out
of 15 answered yes. There was one that answered no. Finally, there were other
7 out of 15 people that did not take any stance, as they responded “I don’t
know”. From these results we cannot strongly confirm that the method, in the
way that we implemented it, is rewarding. It seems quite rewarding, though.

Together with these answers, we gave the students the option of writing
further comments. There were some positive comments, and some others were
possible issues with suggestions for improvements. One mentions that it is
hard not to be able to ask questions when you watch the lecture, and that it
is hard to remember the context of the question in class. A possible solution
to this issue could be to include a discussion forum for each video lecture,
so that students could post the question immediately in the forum. Another
student pointed out that homework distribution was far from being optimal, as
all homework was given in the first week and in the second week there was
nothing. This should definitely be thought in a way that the homework load is
more balanced. On the bright side, it seemed positive to have the chance of
watching again the lecture videos. Also, some felt that group work was better
and that it was nice to have more time to understand code and exercises.

##  8 Conclusion and Future directions

In this paper, we presented a possible class structure for teaching
Conditional Random Fields in almost two lectures. This class is formulated as
a Flipped Classroom, in which there is a strong workload in the students’
preparation and this allows students to get a further understanding of the
topic, compared to traditional lecturing.

We contemplate the Flipped Classroom as a relevant teaching method for
teaching complex topics. We believe that by asking students to do part of the
work beforehand has allowed us to go one step beyond in the understanding of
CRFs. Furthermore, the exercise types that we covered were in the highest
orders in Bloom’s taxonomy, showing the efficiency of the method.

The feedback that we got from students seems to show that, in general, they
learn about the topic in question. It further seems that the method is
rewarding for some students. We believe that the methodology has advantages,
e.g. students get a deeper understanding of the topic, and disadvantages, for
instance a higher preparation time. Considering both aspects, a possibility
could be to find a balance between flipping only a portion of the whole
lecture and having the other portion as a more traditional lecture.

In our prepared lectures, we decided to emphasize on a problem that
Conditional Random Fields fix, the Label Bias problem. In the future, it would
be interesting to include a discussion about the observation bias Klein and
Manning (2002), which happens when the prediction is made by totally ignoring
the labels.

As students have a very varied background, we could already observe
differences in the time of execution of the exercises. A possible solution to
this is to apply differentiated teaching Rock et al. (2008), where the
teaching content is adjusted to some groups of students, making the activities
more tailored to those student clusters.

## Acknowledgements

First of all, I would like to acknowledge the whole Teaching and Learning in
Higher Education (TLHE) program taught at the University of Copenhagen,
especially to my pedagogical and academic supervisors, Lis Lak Risager (TEACH
centre) and Patrizia Paggio (Centre for Language Technology), respectively. I
would like to thank also the students at Language Processing 2 (Spring
semester, course 2020/2021, IT & Cognition program) for being so helpful in
the development of this model. Last, but not least, I thank the anonymous
reviewers for their valuable comments and revisions. These reviews are not
only useful for this article, but also for the development of the whole course
of Language Processing.

## References

  * Anderson et al. (2001) L.W. Anderson, B.S. Bloom, D.R. Krathwohl, P. Airasian, K. Cruikshank, R. Mayer, P. Pintrich, J. Raths, and M. Wittrock. 2001.  _A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives_.  Longman.
  * Bloom et al. (1956) Benjamin S Bloom et al. 1956.  Taxonomy of educational objectives. vol. 1: Cognitive domain.  _New York: McKay_ , 20:24.
  * Bottou (1991) Léon Bottou. 1991.  _Une Approche théorique de l’Apprentissage Connexionniste: Applications à la Reconnaissance de la Parole_.  Ph.D. thesis, Université de Paris XI, Orsay, France.
  * Brame (2013) Cynthia Brame. 2013.  Flipping the classroom (retrieved last 2021-03-15).
  * Jurafsky and Martin (2008) Daniel Jurafsky and James H Martin. 2008.  Speech and language processing: An introduction to speech recognition, computational linguistics and natural language processing.  _Upper Saddle River, NJ: Prentice Hall_.
  * Klein and Manning (2002) Dan Klein and Christopher D Manning. 2002.  Conditional structure versus conditional estimation in nlp models.  In _Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)_ , pages 9–16.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
  * Lage et al. (2000) Maureen J Lage, Glenn J Platt, and Michael Treglia. 2000.  Inverting the classroom: A gateway to creating an inclusive learning environment.  _The journal of economic education_ , 31(1):30–43.
  * McCallum et al. (2000) Andrew McCallum, Dayne Freitag, and Fernando CN Pereira. 2000.  Maximum entropy markov models for information extraction and segmentation.  In _Icml_ , volume 17, pages 591–598.
  * Nielsen (2012) Lisa Nielsen. 2012.  Five reasons i’m not flipping over the flipped classroom.  _Technology & Learning_, 32(10):46–46.
  * Rock et al. (2008) Marcia L Rock, Madeleine Gregg, Edwin Ellis, and Robert A Gable. 2008.  REACH: A framework for differentiating classroom instruction.  _Preventing School Failure: Alternative Education for Children and Youth_ , 52(2):31–47.
  * Ulriksen (2009) Lars Ulriksen. 2009.  The implied student.  _Studies in Higher Education_ , 34(5):517–532.