marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

Duality Principle and Biologically Plausible Learning:
Connecting the Representer Theorem and Hebbian Learning

Yanis Bahroun 1  Dmitri B. Chklovskii 1 2  Anirvan M. Sengupta 1 3

††footnotetext: *Equal contribution 1Flatiron Institute, Simons Foundation,
NY, USA 2Department of Physics and Astronomy, Rutgers University, NJ, USA
3Neuroscience Institute, NYU Langone Medical Center, NY, USA. Correspondence
to: Yanis Bahroun<EMAIL_ADDRESS>
ICML 2023 Workshop on Duality for Modern Machine Learning, Honolulu, Hawaii,
USA. Copyright 2023 by the author(s).

###### Abstract

A normative approach called Similarity Matching was recently introduced for
deriving and understanding the algorithmic basis of neural computation focused
on unsupervised problems. It involves deriving algorithms from computational
objectives and evaluating their compatibility with anatomical and
physiological observations. In particular, it introduces neural architectures
by considering dual alternatives instead of primal formulations of popular
models such as PCA. However, its connection to the Representer theorem remains
unexplored. In this work, we propose to use teachings from this approach to
explore supervised learning algorithms and clarify the notion of Hebbian
learning. We examine regularized supervised learning and elucidate the
emergence of neural architecture and additive versus multiplicative update
rules. In this work, we focus not on developing new algorithms but on showing
that the Representer theorem offers the perfect lens to study biologically
plausible learning algorithms. We argue that many past and current
advancements in the field rely on some form of dual formulation to introduce
biological plausibility. In short, as long as a dual formulation exists, it is
possible to derive biologically plausible algorithms. Our work sheds light on
the pivotal role of the Representer theorem in advancing our comprehension of
neural computation.

##  1 Introduction

Machine learning (ML) and neuroscience have influenced each other in many ways
Bengio et al. (2015); Guerguiev et al. (2017). In particular, the workings of
the human brain have inspired the development of ML algorithms Fukushima
(1980), in particular artificial neural networks (NN). At the same time,
neuroscience has benefited from the insights and models developed in ML
Schrimpf et al. (2018).

One of the main drivers of biological plasticity is the Hebbian principle Hebb
(1949). It is based on the idea that the strength of a connection between
neurons increases when they are activated together. Hebbian learning models
many different aspects of learning in the brain, and numerical implementations
of it are known to learn reoccurring activity patterns Oja (1982); Rumelhart &
Zipser (1985); Sanger (1989); Olshausen & Field (1996). An algorithm using
such rules can process and adapt to one input at a time with memory
requirements independent of the number of samples. Indeed, inspired by the
Hebbian principle, the contributions above enabled substantial developments in
ML and the understanding of neural mechanisms. Regrettably, they are often
based on ad hoc learning rules, and there are few rigorous formulations.

The Representer theorem Wahba (1990); Schölkopf et al. (2001); Argyriou et al.
(2009) is a crucial component for connecting standard and kernel formulations
of ML algorithms. The theorem states that the optimal solution to a particular
class of problems can be expressed as a linear combination of the training
samples in a feature space. The Representer theorem has been used to develop
efficient algorithms for solving many ML problems, including regression,
classification, and clustering. Recently, kernel methods have been omnipresent
for studying deep NNs, either from NTK Jacot et al. (2018); Arora et al.
(2019) or using techniques from statistical mechanics Bordelon et al. (2020);
Canatar et al. (2021). However, although the kernel approach to artificial NNs
has provided insights into the inner workings of artificial NNs, it is unclear
how they can benefit our understanding of biological ones.

This paper highlights that primal-dual formulations of regularized models
provide a perfect framework for understanding the relationships between
Hebbian learning and the Representer theorem. More precisely, by connecting
regularized models with neural architectures in the primal space, Hebbian
learning rules in the dual space, and the primal-dual with neural dynamics,
the theory of duality provides a powerful tool for understanding the
mechanisms of learning in ML and neuroscience. The link between Hebbian
learning and the Representer theorem is an example of the deep insights that
can be gained by exploring the relationships between these two fields. In
practice, this can also lead to developing local and biologically plausible
alternatives to algorithms, such as backpropagation (BP) or neuromorphic
implementation of powerful deep NNs. This work aims at extending the existing
work from Similarity Matching (SM) to supervised learning algorithms. The work
of Pehlevan et al. (2018) first focused on connecting Hebbian learning rules
and similarity-based objective functions.

Outline. In Section 2, we describe the background of Hebbian learning,
synaptic plasticity, and a recently developed framework called Similarity
Matching (SM). SM borrows from the duality theory to derive biologically
plausible algorithms for unsupervised learning. In Section 3, we recall the
theory of duality for models with regularized objectives with convex risks,
the key aspect being the existence of the Representer theorem. In Section 4,
we explore the connection between neural dynamics, Hebbian rules, and the dual
formulation of commonly used models, e.g., Ridge regression, SVM, Logistic
regression, and Entropy-regularized problems. In Section 5, we further explain
the consequences and advantages of this method for studying learning
algorithms. Finally, in Section 6, we discuss the implication and how it
elucidates various methods used in recent years to develop biologically
plausible and local alternatives to BP.

##  2 Background

In recent years, there has been a growing interest in building local and
biologically plausible alternatives to the popular BP algorithm Rumelhart et
al. (1986) for several reasons.

While highly effective for training DNNs, BP relies on the computation of
gradients across all network layers, which is not biologically plausible.
Moreover, BP requires extensive computations and information exchange
throughout the network, which can be computationally expensive and energy-
consuming. Finally, BP typically requires access to large batches of data for
practical training, making it less suitable for online learning scenarios.

Biological systems must instead rely on local computations, are energy-
efficient, and are trained online. As a result, developing algorithms that
align with the constraints and mechanisms observed in biological systems is
important for understanding neural processes and potentially uncovering novel
learning mechanisms. It can also lead to more efficient and scalable learning
processes and enable continual learning. Such developments could also be
crucial for the further development of neuromorphic computing.

###  2.1 Hebbian learning and neural dynamics

The notion of local and online learning rules is essential in neuroscience
because it reflects the idea that locally available quantities drive changes
in neural connections and activity.

#### Hebbian rule.

The Hebbian principle Hebb (1949) is a famous local learning rule stating that
if a pre-synaptic neuron $x_{i}$, and a post-synaptic neuron, $z_{j}$, are
active at the same time, the strength of the connection between them, denoted
by $w_{ij}$ should increase as

$\displaystyle\Delta w_{ij}\propto\eta x_{i}z_{j}~{}~{},$ (1)

where $\Delta w_{ij}$ is the change in the weight, and $\eta$ a learning rate.
The Hebbian principle is important because it provides a mechanism for neurons
to learn associations between stimuli and responses, a fundamental aspect of
many types of learning and memory. We do not discuss the limitation of this
principle here, and we direct the reader toward the literature on BCM
Bienenstock et al. (1982) or STDP Sjöström et al. (2001) if interested.

#### Neural dynamics.

Without being a dynamical systems approach to neuroscience, we assume that
neural activity is a function of its input and not a clipped version of it.
The solution is obtained by running the following dynamical systems until
convergence

$\displaystyle\frac{dz_{t}(\gamma)}{d\gamma}=\Gamma\left({\bf w},{\bf
x}_{t},y_{t},z_{t}(\gamma)\right)~{}~{},$ (2)

where $\Gamma(\cdot)$ is a possibly nonlinear function, and $\gamma$ the steps
in the dynamics.

###  2.2 Similarity matching framework

Interestingly, in the SM framework, unlike in other models for biologically
plausible models, the authors start from the dual problem, which they solve in
the primal by introducing auxiliary variables.

#### Dual problem.

In one of the first papers on SM Pehlevan et al. (2015), the authors borrowed
the objective from Classical Multidimensional scaling Cox & Cox (2000) (CMDS),
using the equivalence that exists between the solution of the CMDS and kernel
PCA Schölkopf et al. (1998) with linear kernel. The objective function that
the authors propose to solve is the following. Given an input data matrix
${\bf X}=[{\bf x}_{1},\ldots,{\bf x}_{T}]\in{\mathbb{R}}^{n\times T}$, find an
output matrix $\hat{{\bf Z}}\in{\mathbb{R}}^{m\times T}$, with $m\leq n$, such
that

$\displaystyle\hat{{\bf Z}}=\underset{{\bf
Z}}{\operatorname{arg}\,\operatorname{min}}\;-2\operatorname{Tr}({\bf
X}^{\top}{\bf X}{\bf Z}^{\top}{\bf Z})+\|{\bf Z}^{\top}{\bf
Z}\|_{F}^{2}~{}~{}.$ (3)

The $\hat{{\bf Z}}$ variables, in Eq. (3), represent the dual representation
of the weight parameters, ${\bf W}$ defined in PCA Williams (2001) as

$\displaystyle\hat{{\bf W}}=\underset{{\bf W}\in{\mathbb{R}}^{m\times
n}}{\operatorname{arg}\,\operatorname{max}}\;\frac{1}{T}\operatorname{Tr}\left({\bf
W}{\bf X}{\bf X}^{\top}{\bf W}^{\top}\right)$ (4)

under the orthonormality constraint ${\bf W}{\bf W}^{\top}={\bf I}_{m}$, with
$m\leq n$. This result derives from the duality relationship that exists
between the eigenvalue decomposition of the covariance matrix ${\bf X}{\bf
X}^{\top}$ and that of the Gram matrix ${\bf X}^{\top}{\bf X}$. An issue with
the dual formulation is that there is no obvious formulation in terms of
synaptic plasticity.

#### Primal-dual formulation.

The authors of Pehlevan & Chklovskii (2015); Pehlevan et al. (2018) proposed
to solve the dual problem, Eq. (3), by reformulating it as the following
primal-dual min-max optimization problem:

$\displaystyle\min_{{\bf Z}}\max_{{\bf M}}\min_{{\bf
W}}-2\operatorname{Tr}({\bf Z}^{\top}{\bf W}{\bf X})+\operatorname{Tr}({\bf
Z}^{\top}{\bf M}{\bf Z})$ $\displaystyle+\operatorname{Tr}({\bf W}^{\top}{\bf
W})-\tfrac{1}{2}\operatorname{Tr}({\bf M}^{\top}{\bf M})~{}~{}.$ (5)

The reformulated problem Eq. (2.2) did not introduce one but two auxiliary
(primal) variables ${\bf W}$ and ${\bf M}$, which have different neural
interpretations.

#### Neural implementation.

The resulting model exhibits a different architecture than primal-based models
Oja (1992), as it incorporates lateral inhibitory connections encoded by the
weight matrix ${\bf M}$. The neural dynamics of the resulting algorithm and
update rules are obtained by gradient optimization applied to the objective
function defined in Eq.(2.2) as follows

Neural dynamics: $\displaystyle\frac{d{\bf Z}(\gamma)}{d\gamma}={\bf W}{\bf
X}-{\bf M}{\bf Z}(\gamma)~{},$ (6) Update rules: $\displaystyle\Delta{\bf
W}={\bf Z}{\bf X}^{\top}~{},~{}\Delta{\bf M}=-{\bf Z}{\bf Z}^{\top}~{}.$ (7)

The neural dynamics Eq. (6), and the update rules Eq. (7) respectively satisfy
the basics definition for biological plausibility defined in Eq. (2) and Eq.
(1).

In a series of recent works (Bahroun et al., 2019; Lipshutz et al., 2020;
2021; Bahroun et al., 2021; Lipshutz et al., 2022), the authors extended the
SM framework to include objectives for more complex learning tasks. Examples
include Canonical Correlation Analysis (CCA) Hotelling (1936), Slow Feature
Analysis Wiskott & Sejnowski (2002), and Independent Component Analysis
Hyvärinen & Oja (2000), which are all generalized eigenvalue (GEV) problems.
Given that GEVs admit dual formulations, the algorithms derived from these
objectives naturally map onto NNs with local forms of plasticity. Reviews of
the SM framework can be found in Pehlevan & Chklovskii (2019); Lipshutz et al.
(2023).

##  3 Duality principle for regularized models

The SM framework mentioned above leverages the primal-dual equivalence of
problems to develop biologically plausible learning rules for unsupervised
learning problems. In this work, the question is about the implication of
applying such an idea to supervised learning problems. In this section, we
recall the duality framework in a supervised setting.

###  3.1 Regularized systems with convex risks

Although the set of models considered can be more general, for practical
relevance, we consider linear (possibly point-wise nonlinear, e.g., ReLU)
prediction problems such as linear regression and classification. In the
following, we restate the more general case and then decompose the problem in
determining the dual for the loss and the regularizer, respectively. For a
detailed presentation of the theory of duality and convex optimization, we
invite the reader to refer to the work of Schölkopf et al. (2001) and Zhang
(2002).

We first set the notations. The data corresponds to samples ${\bf
x}_{t}\in\mathcal{X}=\mathbb{R}^{n}$ and associated labels ${\bf
y}_{t}\in\mathcal{Y}$, $\forall t\in\\{1,\ldots,T\\}$. The models we consider
in the following are those of regularized systems with convex loss. Supervised
learning problems are often phrased as minimizing an objective function with
respect to a set of parameters ${\bf w}\in\mathcal{X}$ as

$\displaystyle\hat{{\bf w}}=\underset{{\bf
w}\in\mathbb{R}^{n}}{\operatorname{arg}\,\operatorname{min}}\;\left[\frac{1}{T}\sum_{t=1}^{T}\ell\left({\bf
y}_{t},f({\bf w}^{\top}{\bf x}_{t})\right)+\lambda g({\bf w})\right]~{}.$ (8)

The general assumptions are the following, $\ell(a,b)$ is a convex function of
$b$, $g(\cdot)$ is a convex function of ${\bf w}$, and $\lambda>0$ is a
regularization parameter. Given this setup, we recall below the Representer
theorem.

######  Theorem 1.

Representer Theorem Schölkopf et al. (1998) Suppose we are given a non-empty
set $\mathcal{X}$, a positive real-valued kernel $k$ on
$\mathcal{X}\times\mathcal{X}$, training sample $({\bf
x}_{1},y_{1}),\ldots,({\bf x}_{T},y_{T})\in\mathcal{X}\times\mathbb{R}$, a
strictly monotonically increasing real-valued function $g$ on $[0,+\infty[$,
an arbitrary cost function
$\ell:{\mathbb{R}}\times{\mathbb{R}}\rightarrow\mathbb{R}\cup\\{\infty\\}$,
and a class of functions

$\displaystyle\mathcal{F}=\left\\{f\in{\mathbb{R}}^{\mathcal{X}}|f(\cdot)=\sum_{i=1}^{\infty}\beta_{i}k(\cdot,{\bf
h}_{i}),\beta_{i}\in{\mathbb{R}},\right.$ $\displaystyle\left.{\bf
h}_{i}\in\mathcal{X},\|f\|<\infty\right\\}~{}.$ (9)

Here, $\|\cdot\|$ is the norm in the RKHS $H_{k}$ associated with $k$. Then
any $f\in\mathcal{F}$ minimizing the regularized risk functional

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}\ell(y_{t},f({\bf x}_{t}))+\lambda
g(\|f\|)~{},$ (10)

admits a representation of the form

$\displaystyle f(\cdot)=\sum_{t=1}^{T}z_{t}k(\cdot,{\bf x}_{t})~{}~{}.$ (11)

Although there is potentially a limitless amount of such objectives, it
appears that the literature often revolves around the following few. The
square loss, $\ell({\bf y},f({\bf w}^{\top}{\bf x}))=\frac{1}{2}\|{\bf
y}-f({\bf w}^{\top}{\bf x})\|^{2}$, for regularized linear regression. The SVM
loss, $\ell({\bf y},f({\bf w}^{\top}{\bf x}))=\ell({\bf y}f({\bf w}^{\top}{\bf
x}))=1_{{\bf y}f({\bf x})<0}$, and cross-entropy as in logistic regression
$\ell({\bf y},f({\bf w}^{\top}{\bf x}))=-{\bf y}\log(f({\bf w}^{\top}{\bf
x}))-(1-{\bf y})\log(1-f({\bf w}^{\top}{\bf x}))$ for classification. For
regression tasks consider $\mathcal{Y}=\mathbb{R}$ or $\mathbb{R}^{n}$, and
instead $\mathcal{Y}=\\{-1,1\\}$ for classification tasks.

###  3.2 Dual formulation of the loss term $\ell(\cdot)$

We first address the dual of the loss term, which is not only a function of
the parameters, ${\bf w}$, and the samples, ${\bf x}$, but also of the labels
${\bf y}$. Let $c(y,\cdot)$ be the dual transform of $\ell(y,\cdot)$ defined
as

$\displaystyle c(y,v)=\sup_{u}(uv-\ell(y,u)),$ (12)

implies that for all $t\in\\{1,\ldots,T\\}$

$\displaystyle\ell(y_{t},{\bf w}^{\top}{\bf
x}_{t})=\sup_{z_{t}}\left(z_{t}{\bf w}^{\top}{\bf
x}_{t}-c(y_{t},-z_{t})\right).$ (13)

Using this definition, we can now introduce an auxiliary, dual variable ${\bf
z}$, with component $z_{t}$, such that the following objective

$\displaystyle R({\bf w},{\bf
z})=\frac{1}{T}\sum_{t=1}^{T}\left(-c(y_{t},-z_{t})-z_{t}{\bf w}^{\top}{\bf
x}_{t}\right)+\lambda g({\bf w}),$ (14)

and the optimization problem, Eq. (8), are equivalent the optimal solution
$\hat{{\bf w}}$ can be equivalently rewritten as

$\displaystyle\hat{{\bf w}}=\arg\inf_{{\bf w}}\sup_{{\bf z}}R({\bf w},{\bf
z}).$ (15)

Switching the order of $\inf_{\bf w}$ and $\sup_{{\bf z}}$ in the above
minimax convex-concave programming problem is valid. Given $\hat{{\bf w}}$ a
solution of the problem Eq. (15), there is a solution ${\bf z}$ to the
following problem

$\displaystyle\hat{{\bf z}}=\arg\sup_{{\bf z}}\inf_{{\bf w}}R({\bf w},{\bf
z}).$ (16)

###  3.3 Dual of the regularizer $g(\cdot)$

The goal now is to have a formulation of Eq (16) only as a function of ${\bf
z}$. To do so, we now consider $h(\cdot)$ the dual transform of $g(\cdot)$ the
regularizer defined by

$\displaystyle h(v)=\sup_{u}(u^{\top}v-g(u))~{},$ (17)

where $h(\cdot)$ is also a convex function. Using the dual formulation of the
regularizer, the optimization problem Eq. (16) can be rewritten as the
following dual formulation

$\displaystyle\hat{{\bf
z}}=\arg\inf_{z}\left[\frac{1}{T}\sum_{t=1}^{T}c(y_{t},-z_{t})+\lambda
h\left(\frac{1}{\lambda T}\sum_{t=1}^{T}z_{t}{\bf x}_{t}\right)\right]~{},$
(18)

strictly in terms of ${\bf z}$.

###  3.4 General implications

As a result, we see that the optimal solution $\hat{{\bf w}}$ to the primal
problem in Eq. (8) is given by

$\displaystyle\hat{{\bf w}}=\nabla h\left(\frac{1}{\lambda
T}\sum_{t=1}^{T}\hat{z_{t}}{\bf x}_{t}\right)~{}.$ (19)

Looking at Eq. (19), it becomes clear that it greatly resembles the form of
the Hebbian rule, Eq. (1), where the synaptic weights are a weighted sum of
the samples.

Interestingly, we also have

$\displaystyle\hat{z_{t}}=-\ell^{\prime}(y_{t},\hat{{\bf w}}^{\top}{\bf
x}_{t})~{}~{},$ (20)

where $\ell^{\prime}$ denotes a (sub)gradient of $\ell(a,b)$ with respect to
$b$. The dual variables $z_{t}$, Eq. (20), now have clearly defined neural
dynamics as the solution to an optimization problem. It is a function of the
samples, weights, and labels as defined in Eq (2).

To clarify this point, we study in the following section (Sec. 4) some of the
most widely used models for regression and classification. In Section 6, we
will discuss recent developments of biologically plausible learning
alternatives to BP and show how heavily they rely on the Representer theorem
often implicitly. We also describe in Section 6, what canonical neural
components are used in some of the more elaborate biologically plausible
implementations.

##  4 Neural dynamics and plasticity rule

Using the general formulation of the learning problem presented in the
previous section, we recall the dual formulation of popular models, such as
Ridge regression, SVM, and logistic regression. We present the implications of
choosing a regularizer and a cost function to the form of the update rule,
i.e., additive ($L_{2}$) or multiplicative (Entropy), and either product or
difference of kernels of label and samples.

The purpose of this section is to lay the groundwork for identifying the
critical component for “biological interpretation”.

###  4.1 Ridge Regression

We consider $\mathcal{X}\in\mathbb{R}^{n}$ and $\mathcal{Y}\in\mathbb{R}$,
with $f:\mathbb{R}\mapsto\mathbb{R}$ the simple linear function $f({\bf
w}^{\top}{\bf x})={\bf w}^{\top}{\bf x}$. We use here the quadratic loss
defined as follows

$\displaystyle\min_{{\bf w}\in\mathbb{R}^{n}}\sum_{t=1}^{T}(y_{t}-{\bf
w}^{\top}{\bf x}_{t})^{2}+\frac{\lambda}{2}{\bf w}^{\top}{\bf w}~{}~{}.$ (21)

The problem Eq. (21), being part of the framework Sec. 3, admits a dual
formulation of the loss. We now introduce the variables
$z_{t=1..T}\in\mathbb{R}$, which identify with a prediction error term
$(y_{t}-{\bf w}^{\top}{\bf x}_{t})$ as

$\displaystyle(y_{t}-{\bf w}^{\top}{\bf
x}_{t})^{2}=\max_{z_{t}}z_{t}(y_{t}-{\bf w}^{\top}{\bf
x}_{t})-\frac{1}{2}z_{t}^{2}.$ (22)

As a result, we can rewrite the original objective as a min-max objective, Eq.
(21)

$\displaystyle\min_{{\bf w}}\max_{{\bf z}}R({\bf w},{\bf z})=\min_{{\bf
w}}\max_{z_{t=1..T}}\sum_{t=1}^{T}z_{t}(y_{t}-{\bf w}^{\top}{\bf x}_{t})$
$\displaystyle-\tfrac{1}{2}\sum_{t=1}^{T}z^{2}_{t}+\tfrac{1}{2}{\bf
w}^{\top}{\bf w}.$ (23)

We swap the order of optimization to obtain the optimal set of parameters,
$\hat{{\bf w}}$, as a function of the dual variables, $z_{t}$ as

$\displaystyle\hat{{\bf w}}=\sum_{t=1}^{T}z_{t}{\bf x}_{t}~{}~{}.$ (24)

#### Dual objective function.

By replacing $\hat{{\bf w}}$ we obtain the following dual objective function
for linear regression as

$\displaystyle\max_{z_{t}}R(\hat{{\bf
w}},z_{t})=\max_{z_{t}}\sum_{t=1}^{T}z_{t}y_{t}-\frac{1}{2}\sum_{t^{\prime}=1}^{T}z_{t}z_{t^{\prime}}{\bf
x}_{t^{\prime}}^{\top}{\bf x}_{t}$
$\displaystyle-\frac{1}{2}\sum_{t=1}^{T}z^{2}_{t}~{}~{}.$ (25)

With abuse of notations, $\operatorname{diag}({\bf z})={\bf Z}$, we can
rewrite the problem as

$\displaystyle\max_{{\bf Z}}\operatorname{Tr}({\bf Y}^{\top}{\bf
Z})-\frac{1}{2}\operatorname{Tr}({\bf X}^{\top}{\bf X}{\bf Z}^{\top}{\bf
Z})-\frac{1}{2}\operatorname{Tr}({\bf Z}^{\top}{\bf Z})~{}~{}.$ (26)

#### Connection with Similarity Matching.

The dual objective above shares a lot of similarities with the SM objective,
Eq.(3). Indeed, the synaptic weights, ${\bf w}$, arise from the kernel-to-
kernel relationship between samples and neural output variable,
$\operatorname{Tr}({\bf X}^{\top}{\bf X}{\bf Z}^{\top}{\bf Z})$.
Interestingly, although here, the optimization is a maximization rather than a
minimization, it is the same object that is considered. This naturally leads
to the fact that the neural variables, ${\bf Z}$, in a supervised setting
correspond to errors.

#### Architecture, plasticity, and neural dynamics.

By construction of $z_{t}$ and ${\bf w}$, it is already clear that the
plasticity of this connection is Hebbian and that we can derive neural
dynamics for $z_{t}$ and plasticity rules as

$\displaystyle\text{Neural
dynamics:}~{}~{}~{}\tfrac{dz_{t}(\gamma)}{d\gamma}=(y_{t}-{\bf w}^{\top}{\bf
x}_{t})-z_{t}(\gamma),$ (27) $\displaystyle\text{Plasticity
rules:}~{}~{}~{}~{}\Delta{\bf w}=\eta z_{t}{\bf x}_{t}~{}~{}.$ (28)

The neural dynamics above describe a simple feedforward NN with inputs ${\bf
x}_{t}$ and $y_{t}$. The input ${\bf x}_{t}$ is weighted by the synaptic
connections ${\bf w}$, while the input $y_{t}$ has a fixed neural connection
to the output neuron $z_{t}$. The update rule is a simple Hebbian one for
${\bf w}$ with pre-synaptic activities, ${\bf x}_{t}$, and post-synaptic
activities $z_{t}$ as defined by Eq. (1) in Sec. 2. Introducing the dual
variable $z_{t}$ leads to neural dynamics and Hebbian rules. The biological
realism of such an “error” computing neuron is mentioned in Sec. 6.

###  4.2 Support Vector Machine

In the following, we consider the SVM problem introduced by Cortes & Vapnik
(1995) and formulated as follows

$\displaystyle\min_{{\bf w}\in\mathbb{R}^{n}}\frac{1}{2}\|{\bf
w}\|^{2}~{}~{}\text{s.t.}~{}~{}y_{t}{\bf w}^{\top}{\bf x}_{t}\geq 1~{}.$ (29)

#### Dual objective function.

The dual objective function for SVM is the following

$\displaystyle\max_{z_{t}\geq
0}-\frac{1}{2}\operatorname{Tr}(\chi^{\top}\chi{\bf Z}^{\top}{\bf
Z})+\left(\sum_{t}z_{t}~{}~{}-\frac{\kappa}{2}z_{t}^{2}\right)~{}~{},$

with $\chi_{t}=y_{t}{\bf x}_{t}$. An important difference with the objective
Eq. (4.3), is that SVM considers the product of kernels, ${\bf X}^{\top}{\bf
X}{\bf Y}^{\top}{\bf Y}$ rather than their difference as in Eq. (26).
Nonetheless, the dual variables still account for some form of error.

#### Architecture, plasticity, and neural dynamics.

The neural dynamics differ as it is a “function” of the loss, not the
regularizer

$\displaystyle\text{Neural dynamics:}~{}~{}z_{t}=\frac{1}{\kappa}[1-y_{t}{\bf
w}_{t}^{\top}{\bf x}]_{+}$ (30) $\displaystyle\text{Hebbian
update:}~{}~{}\Delta{\bf w}=\eta z_{t}y_{t}{\bf x}_{t}$ (31)

This dual formulation of the update gives a simple version of the passive-
aggressive algorithm introduced by Crammer et al. (2006).

The neural architecture arising from the primal-dual problem is that of a
feedforward NN with rectified units. The interesting aspect of the
architecture is that the output is a function of the product $y_{t}{\bf
x}_{t}$, unlike in the regression problem Sec. 4.1. This is also the case for
the Hebbian update rule for the weights ${\bf w}$, with pre-synaptic input
${\bf x}_{t}$, post-synaptic output $z_{t}$, and $y_{t}$ possibly operating as
a modulating factor. Such an interaction can be speculated in the cerebellum,
where this learning mechanism could be attributed to the interaction between
Purkinje cells, $z_{t}$, parallel fibers ${\bf x}_{t}$, and climbing fibers
$y_{t}$ Moldwin & Segev (2020). SVM is also a candidate for odor
classification in the mushroom body Muezzinoglu et al. (2008); Nowotny &
Huerta (2012).

#### Connection with Similarity Matching.

Again, as in Ridge regression, the synaptic weights, ${\bf w}$, arise from the
kernel-to-kernel relationship between samples and neural output variable,
$\operatorname{Tr}(\chi^{\top}\chi{\bf Z}^{\top}{\bf Z})$. Maximization
replaces minimization, and the input kernel considered here is the kernel
product between labels and samples.

###  4.3 Logistic Regression

Another prevalent classification model is that of Logistic Regression (LR) Cox
(1958). The objective function for LR is that of binary cross-Entropy or
Logloss. The LR model is

$\displaystyle p(y=\pm 1|{\bf x},{\bf w})=\frac{1}{1+\exp(-y{\bf w}^{\top}{\bf
x})}~{}~{}.$ (32)

#### Dual objective function.

In this context, Jaakkola & Haussler (1999) introduced the dual algorithm for
LR as

$\displaystyle R({\bf z})$
$\displaystyle=-\frac{1}{2}\sum_{t,t^{\prime}}z_{t}z_{t^{\prime}}y_{t}y_{t^{\prime}}{\bf
x}_{t}^{\top}{\bf x}_{t^{\prime}}+\sum_{t}F(z_{t})~{},$
$\displaystyle=-\frac{1}{2}\operatorname{Tr}(\chi^{\top}\chi{\bf Z}^{\top}{\bf
Z})+\sum_{t}F(z_{t})~{},$ (33)

with $F(z)=-z\log(z)-(1-z)\log(1-z)$, and $\chi$ defined as in Sec. 4.2.

#### Architecture, plasticity, and neural dynamics.

Given the formulation of the dual objective function for LR, Eq. (4.3), the
update rules and neural dynamics are the following

$\displaystyle\text{Neural
dynamics:}~{}~{}\tfrac{dz_{t}(\gamma)}{d\gamma}=y_{t}{\bf w}_{t}^{\top}{\bf
x}_{t}-F^{\prime}(z_{t}(\gamma))~{},$ (34) $\displaystyle\text{Hebbian
update:}~{}~{}\Delta{\bf w}=\eta z_{t}y_{t}{\bf x}_{t}~{},$ (35)

with $z_{t}\in[0,1]$. Here the neural activity is obtained as a fixed point of
the neural dynamics. Such neural dynamics in the context of unsupervised
learning can also be found in Rozell et al. (2008); Obeid et al. (2019).

The neural dynamics, as a function of the loss, differs from that of the SVM
above. Here the neural activity is bounded between 0 and 1, as is expected
from a sigmoid function. However, the update rule is the same. This is an
example where more than knowing the update rule is required for distinguishing
the underlying task being performed. However, with the power of the duality
framework, looking at the neural dynamics in concert with the update rule
should help in reverse-engineering neural computation.

###  4.4 Multiplicative rules from Entropy regularization

We have emphasized the importance of the $L_{2}$ regularization. However,
other types of regularization are also useful. In the maximum entropy
framework for density estimation, one seeks a density vector ${\bf w}$ that
maximizes the relative entropy as

$\displaystyle\hat{{\bf w}}=\underset{{\bf
w}\in\mathbb{R}^{n}_{+}}{\operatorname{arg}\,\operatorname{max}}\;g({\bf
w})~{}\text{with}~{}y_{t}={\bf w}^{\top}{\bf x}_{t},~{}\forall t\in\\{1,n\\}.$
(36)

The regularization condition, $g(\cdot)$ is formulated as

$\displaystyle g({\bf
w})=\sum_{k=1}^{n}w_{k}\ln\left(\frac{w_{k}}{\mu_{k}}\right)~{}~{},~{}~{}w_{k}\geq
0~{},$ (37)

which is the unnormalized entropy regularization. It has been popular in the
online learning literature for leading to exponentiated gradient rules Kivinen
& Warmuth (1995). This learning problem has for dual variables neural
dynamics, and update rules the following

$\displaystyle\text{Neural
dynamics:}~{}~{}\tfrac{dz_{t}(\gamma)}{d\gamma}=(y_{t}-{\bf w}^{\top}{\bf
x}_{t})-z_{t}(\gamma),$ (38) $\displaystyle\text{Hebbian
update:}~{}~{}~{}~{}\hat{w}_{k,t+1}\propto\hat{w}_{k,t}\exp\left(z_{t}{\bf
x}_{t}\right)~{}.$ (39)

We can view this update as a multiplicative Hebbian rule. As the task is
closely related to a regression model, the neural architecture and neural
dynamics are unsurprisingly the same as that of RR. However, the difference
lies in the update rule, determined by the regularization term. Indeed, the
update is multiplicative. Again, the duality framework with constraints on the
neural dynamics and the update allows us to narrow down the computation
performed in specific neural components.

##  5 Connection between the duality principle and biologically learning
algorithms

Based on the different examples given in Section 4, we highlight how looking
through the framework of duality can enlighten how we see neural plasticity
and possibly reverse-engineer some observed anatomical and physiological
observations.

###  5.1 Sparse or dense dual variables

The dual variables in the supervised setting had the property of accounting
for the error of the current model when evaluated on a new sample. This
starkly contrasts with the type of dual variables mentioned in the SM
framework, which is more unsupervised learning focused. Indeed, in an
unsupervised setting, the dual variables account instead for the alignment
between the sample and the current estimate.

This has important implications for the nature of the update that can be
tested in practice. In practice, supervised learning models will show sparse
updates in time, while similarity-based models will show dense updates in
time. Indeed, in unsupervised models, a better update typically results in a
larger update. In contrast, a correct prediction does not require an update in
supervised learning. Specifically, the dual variables $z_{t}$ become inactive
in supervised learning once the optimal hyperplane is found for linearly
separable problems.

###  5.2 Additive or multiplicative updates

From an ML perspective, we mentioned additive and multiplicative updates in
the previous section. The distinction also exists in neuroscience as they
correspond to two types of synaptic plasticity in the brain. Additive
plasticity refers to changes in the synaptic strength independent of the
synapse’s current strength. In contrast, multiplicative plasticity refers to
changes proportional to the synapse’s current strength.

In the brain, additive plasticity is considered important for regulating the
overall level of synaptic activity in a NN. On the other hand, multiplicative
plasticity is important for regulating the relative strength of different
synapses in a network Loewenstein et al. (2011). This helps maintain the
synapses’ relative strength and prevent the weaker synapse from becoming
saturated.

We observed that such rules could be employed for the same regression task but
imply different regularization. This dichotomy is what distinguishes, for
example, perceptron-like models from boosting-type models Kivinen & Warmuth
(1999); Zemel & Pitassi (2000). Entropy regularization is also a type of
regularization often present in information-theoretic formulations. For
example, in the recent work of Halvagal & Zenke (2022), they observe
“multiplicative” update rules such as the BCM rule. Similarly, for the work of
Bahroun et al. (2022).

Ultimately, both additive and multiplicative plasticity is essential, and in
this work, we highlighted that a choice of learning rule can be directly
traced back to the choice of the regularizer. We showed that the Representer
theorem and duality allow reverse-engineering of the objective function and
understanding of the neural circuit’s task.

###  5.3 $L_{1}$ penalty on the weights ${\bf w}$

An important type of regularization that we have not covered yet that of
sparsity-inducing regularization. From an ML perspective, the sparsity
constraint is integral to the LASSO method. LASSO (least absolute shrinkage
and selection operator) Tibshirani (1996); Donoho & Johnstone (1994) is a
regularized linear regression with $L_{1}$ penalty stated as

$\displaystyle\min_{{\bf w}\in\mathbb{R}^{n}}\sum_{t=1}^{T}(y_{t}-{\bf
w}^{\top}{\bf x}_{t})^{2}+\lambda\sum_{i=1}^{n}|w_{i}|~{}~{}.$ (40)

The LASSO method is often used in practice for feature selection, to select a
few valuable and ignore the many that are not. We refer the reader to the work
of Bach et al. (2011) for an overview of convex optimization with sparsity-
inducing norms.

#### Sparse connections in biological NNs.

Synaptic weights in the brain are sparse. Indeed, only a small subset of the
connections between neurons in the brain exists Song et al. (2005), meaning
that the networks are sparsely connected.

This has to be distinguished from the more common sparsity at the level of
neural activity, which has been broadly studied as part of the theory of
sparse coding Olshausen & Field (1997), and sparse dictionary learning Aharon
et al. (2006). Biologically plausible implementations in the SM framework have
also been studied Pehlevan & Chklovskii (2014); Bahroun & Soltoggio (2017);
Lipshutz et al. (2022).

Connection sparsity is important in neuroscience for at least two reasons.
Firstly, a sparsely connected network is considered important because it
allows the brain to perform complex computations with relatively few active
connections, reducing energy consumption and processing time. Secondly, it can
be thought that the brain might be performing a form of a sparse mixture of
experts, where not all features, i.e., upstream information, are relevant,
allowing the extraction of only relevant information. In other words, synaptic
weight sparsity is a form of neural efficiency.

#### Plausible implementation.

The work of Connor et al. (2015) showed that by using a different prior than
the Laplace prior, which they describe as “offsetting or raising” the
Laplacian prior distribution, they could propose a biological algorithm for
Bayesian LASSO. As a result, they observed that the learning rule is now
affected by a different learning rate, which in our framework can be
represented by a set of dual variables. Interestingly, they argued that
raising the Laplacian prior led to updated rules that better match the limited
loss of association seen between days in the human data of the perceptual
expertise task.

##  6 Discussion of existing methods

In this section, we discuss recent developments that rely on duality and some
of the outlined elements in Sec. 4. These works develop online algorithms for
deep learning and biologically plausible alternatives to BP.

###  6.1 Local learning rules in Deep Learning

Various recent contributions have been developed based on duality in deep
learning. In particular, one of the earlier works is the Method of Auxiliary
Coordinates (MAC) from Carreira-Perpinan & Wang (2014)

The idea is the following, training a fully-connected NN with $L$ hidden
layers consists of minimizing the loss $\mathcal{L}({\bf y},\sigma({\bf
W},{\bf x}_{L}))$ involving a nested function

$\displaystyle f({\bf W},{\bf x}_{L})=f_{L}({\bf W}_{L},\sigma_{L-1}({\bf
W}_{L-1},\ldots,\sigma_{1}({\bf W}_{1},{\bf x}_{L}))~{},$ (41)

which can be re-written as constrained optimization:

$\displaystyle\min_{{\bf W}}\sum_{t=1}^{n}\ell({\bf y}_{t},{\bf
a}_{t}^{L},{\bf W}_{L+1})~{}~{},~{}~{}\text{where}~{}~{}{\bf
a}_{t}^{l}=\sigma_{l}({\bf c}_{t}^{l}),~{}~{}$ (42)
$\displaystyle\text{s.t.}~{}~{}{\bf c}_{l}^{t}={\bf W}_{l}{\bf
a}^{l-1}_{t},~{}\forall l\in\\{1,L\\}~{}~{}\text{and}~{}~{}{\bf
a}_{t}^{0}={\bf x}_{t}.$ (43)

The NN becomes a succession of regularized objectives that we detailed above,
which are types of RR. The generality of the duality approach can be seen in
the fact that a lot of novelties introduced by Carreira-Perpinan & Wang (2014)
could be carried away to the work of Choromanska et al. (2019) by changing the
objective function. And the work of Choromanska et al. (2019) recovering
Hebbian-like learning rules and local quantities is a direct consequence of
the Representer theorem and choice of the regularizer.

Other objective functions, which directly consider the notion of similarities
for optimizing NNs as in Pogodin & Latham (2020); Ma et al. (2020), have been
proposed and, as part of the framework, lead to local learning rules or
alternatives to BP. Although the works mentioned above show some biological
plausibility, they are arguably separated from the more neuroscience-driven
approaches, which we detail in the following section.

###  6.2 Local learning for neuroscience

In the last few years, various attempts have been made at building
biologically plausible alternatives to BP. Interestingly, the approaches were
similar to those in Sec. 6.1. Although not always explicitly mentioned, those
models fall in the category of nested objectives, where each layer can be
phrased as in Sec. 4.

Those approaches focused on building or mapping the resulting component onto
anatomical and physiological observations. The two approaches detailed below
are to either consider the dual variables as quantities computed in a
dendritic compartment of pyramidal-like cells or to have them computed by
error neurons.

#### Dendritic computation of error signal.

In their work, Sacramento et al. (2018) proposed representing feedback error
signals in dendritic compartments. Similar ideas were also explored in the
work of Golkar et al. (2020) and Meulemans et al. (2021), each adding various
valuable biological components to their models. It is important to note that
all these models have in common to impose regularization properties to their
models so that local (dual) variables exist and can be interpreted as
biophysical quantities.

This line of work has also been furthered in Meulemans et al. (2022); Zucchet
& Sacramento (2022) by introducing bilevel optimization and dual variables to
produce quantities that can be computed locally. We also refer the reader to
work on deep equilibrium models Bai et al. (2019) for completeness.

#### Error and inter-neurons.

Another line of work focused on the predictive coding theory that local error
neurons can use two distinct populations of neurons Whittington & Bogacz
(2017). Error neurons measure deviations of the actual state of prediction
neurons from the expected state. This is very similar to what the dual
formulation offers. Indeed, the probabilistic formulation of predictive coding
makes it similar to regularized generalized linear models and that of
Carreira-Perpinan & Wang (2014).

More recently, the work of Golkar et al. (2022) has shown that the
introduction of optimization constraints related to whitening allowed the
introduction of interneurons alleviating various limitations on the biological
implementations of the predictive framework. The reader can find other
discussions on the possible implementation of the PC framework in the work of
Ororbia & Mali (2019).

##  7 Conclusion

In this work, we aimed to show that local learning rules, the Hebbian
principle, and the Representer theorem, although coming from different fields,
can be considered similar concepts and should be used in concert. We have
provided well-known examples of ML algorithms that can be naturally phrased as
biologically plausible. We also emphasized how the variety of approaches
covered by Representer can account for biologically relevant features.

In conclusion, adopting a viewpoint based on dual variables and the
Representer theorem during the development of algorithms offers a twofold
advantage. Firstly, it enables the creation of biologically plausible
algorithms that align with observable physiological and anatomical
observations. Secondly, it facilitates the reverse-engineering process,
allowing for the interpretation and mapping of local variables to canonical
components found in the brain. Researchers are bridging the gap between
computational models and biological systems by leveraging dual variables,
paving the way for advancements in our understanding of neural processes.

## References

  * Aharon et al. (2006) Aharon, M., Elad, M., and Bruckstein, A.  K-svd: An algorithm for designing overcomplete dictionaries for sparse representation.  _IEEE Transactions on signal processing_ , 54(11):4311–4322, 2006.
  * Argyriou et al. (2009) Argyriou, A., Micchelli, C. A., and Pontil, M.  When is there a representer theorem? vector versus matrix regularizers.  _The Journal of Machine Learning Research_ , 10:2507–2529, 2009.
  * Arora et al. (2019) Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R.  On exact computation with an infinitely wide neural net.  _Advances in neural information processing systems_ , 32, 2019.
  * Bach et al. (2011) Bach, F., Jenatton, R., Mairal, J., Obozinski, G., et al.  Convex optimization with sparsity-inducing norms.  _Optimization for Machine Learning_ , 5:19–53, 2011.
  * Bahroun & Soltoggio (2017) Bahroun, Y. and Soltoggio, A.  Online representation learning with single and multi-layer Hebbian networks for image classification.  In _Artificial Neural Networks and Machine Learning–ICANN 2017: 26th International Conference on Artificial Neural Networks, Alghero, Italy, September 11-14, 2017, Proceedings, Part I 26_ , pp. 354–363. Springer, 2017\.
  * Bahroun et al. (2019) Bahroun, Y., Sengupta, A., and Chklovskii, D. B.  A similarity-preserving network trained on transformed images recapitulates salient features of the fly motion detection circuit.  In _Advances in Neural Information Processing Systems_ , pp. 14178–14189, 2019.
  * Bahroun et al. (2021) Bahroun, Y., Chklovskii, D., and Sengupta, A.  A normative and biologically plausible algorithm for independent component analysis.  _Advances in Neural Information Processing Systems_ , 34:7368–7384, 2021.
  * Bahroun et al. (2022) Bahroun, Y., Acharya, A., Chklovskii, D., and Sengupta, A. M.  Similarity-preserving neural networks from GPLVM and information theory.  In _NeurIPS 2022 Workshop on Information-Theoretic Principles in Cognitive Systems_ , 2022.  URL https://openreview.net/forum?id=T8C8m_20jiE.
  * Bai et al. (2019) Bai, S., Kolter, J. Z., and Koltun, V.  Deep equilibrium models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Bengio et al. (2015) Bengio, Y., Lee, D.-H., Bornschein, J., and Lin, Z.  Towards biologically plausible deep learning.  _arXiv preprint arXiv:1502.04156_ , 2015.
  * Bienenstock et al. (1982) Bienenstock, E. L., Cooper, L. N., and Munro, P. W.  Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex.  _Journal of Neuroscience_ , 2(1):32–48, 1982\.
  * Bordelon et al. (2020) Bordelon, B., Canatar, A., and Pehlevan, C.  Spectrum dependent learning curves in kernel regression and wide neural networks.  In _International Conference on Machine Learning_ , pp. 1024–1034. PMLR, 2020.
  * Canatar et al. (2021) Canatar, A., Bordelon, B., and Pehlevan, C.  Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks.  _Nature communications_ , 12(1):2914, 2021.
  * Carreira-Perpinan & Wang (2014) Carreira-Perpinan, M. and Wang, W.  Distributed optimization of deeply nested systems.  In _Artificial Intelligence and Statistics_ , pp. 10–19. PMLR, 2014\.
  * Choromanska et al. (2019) Choromanska, A., Cowen, B., Kumaravel, S., Luss, R., Rigotti, M., Rish, I., Diachille, P., Gurev, V., Kingsbury, B., Tejwani, R., et al.  Beyond backprop: Online alternating minimization with auxiliary variables.  In _International Conference on Machine Learning_ , pp. 1193–1202. PMLR, 2019.
  * Connor et al. (2015) Connor, P., Hollensen, P., Krigolson, O., and Trappenberg, T.  A biological mechanism for bayesian feature selection: weight decay and raising the lasso.  _Neural Networks_ , 67:121–130, 2015.
  * Cortes & Vapnik (1995) Cortes, C. and Vapnik, V.  Support-vector networks.  _Machine learning_ , 20(3):273–297, 1995.
  * Cox (1958) Cox, D. R.  The regression analysis of binary sequences.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 20(2):215–232, 1958.
  * Cox & Cox (2000) Cox, T. F. and Cox, M. A.  _Multidimensional scaling_.  Chapman and hall/CRC, 2000.
  * Crammer et al. (2006) Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., and Singer, Y.  Online passive-aggressive algorithms.  _Journal of Machine Learning Research_ , 7(Mar):551–585, 2006.
  * Donoho & Johnstone (1994) Donoho, D. L. and Johnstone, I. M.  Ideal spatial adaptation by wavelet shrinkage.  _biometrika_ , 81(3):425–455, 1994.
  * Fukushima (1980) Fukushima, K.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position.  _Biological cybernetics_ , 36(4):193–202, 1980\.
  * Golkar et al. (2020) Golkar, S., Lipshutz, D., Bahroun, Y., Sengupta, A., and Chklovskii, D.  A simple normative network approximates local non-hebbian learning in the cortex.  _Advances in neural information processing systems_ , 33:7283–7295, 2020.
  * Golkar et al. (2022) Golkar, S., Tesileanu, T., Bahroun, Y., Sengupta, A., and Chklovskii, D.  Constrained predictive coding as a biologically plausible model of the cortical hierarchy.  _Advances in Neural Information Processing Systems_ , 35:14155–14169, 2022.
  * Guerguiev et al. (2017) Guerguiev, J., Lillicrap, T. P., and Richards, B. A.  Towards deep learning with segregated dendrites.  _Elife_ , 6:e22901, 2017.
  * Halvagal & Zenke (2022) Halvagal, M. S. and Zenke, F.  The combination of hebbian and predictive plasticity learns invariant object representations in deep sensory networks.  _bioRxiv_ , pp. 2022–03, 2022.
  * Hebb (1949) Hebb, D. O.  _The organization of behavior: A neuropsychological approach_.  John Wiley & Sons, 1949.
  * Hotelling (1936) Hotelling, H.  Relations between two sets of variates.  _Biometrika_ , 28(3-4):321–377, 1936.
  * Hyvärinen & Oja (2000) Hyvärinen, A. and Oja, E.  Independent component analysis: algorithms and applications.  _Neural networks_ , 13(4-5):411–430, 2000.
  * Jaakkola & Haussler (1999) Jaakkola, T. S. and Haussler, D.  Probabilistic kernel regression models.  In _AISTATS_ , 1999.
  * Jacot et al. (2018) Jacot, A., Gabriel, F., and Hongler, C.  Neural tangent kernel: Convergence and generalization in neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Kivinen & Warmuth (1995) Kivinen, J. and Warmuth, M. K.  Additive versus exponentiated gradient updates for linear prediction.  In _Proceedings of the twenty-seventh annual ACM symposium on Theory of computing_ , pp. 209–218, 1995.
  * Kivinen & Warmuth (1999) Kivinen, J. and Warmuth, M. K.  Boosting as entropy projection.  In _Proceedings of the twelfth annual conference on Computational learning theory_ , pp. 134–144, 1999.
  * Lipshutz et al. (2020) Lipshutz, D., Windolf, C., Golkar, S., and Chklovskii, D. B.  A biologically plausible neural network for slow feature analysis.  _Advances in Neural Information Processing Systems_ , 33:14986–14996, 2020.
  * Lipshutz et al. (2021) Lipshutz, D., Bahroun, Y., Golkar, S., Sengupta, A. M., and Chklovskii, D. B.  A biologically plausible neural network for multichannel canonical correlation analysis.  _Neural Computation_ , 33(9):2309–2352, 2021\.
  * Lipshutz et al. (2022) Lipshutz, D., Pehlevan, C., and Chklovskii, D. B.  Biologically plausible single-layer networks for nonnegative independent component analysis.  _Biological Cybernetics_ , 116(5–6):557–568, 2022.
  * Lipshutz et al. (2023) Lipshutz, D., Bahroun, Y., Golkar, S., Sengupta, A. M., and Chklovskii, D. B.  A normative framework for deriving neural networks with multi-compartmental neurons and non-hebbian plasticity.  _arXiv preprint arXiv:2302.10051_ , 2023.
  * Loewenstein et al. (2011) Loewenstein, Y., Kuras, A., and Rumpel, S.  Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo.  _Journal of Neuroscience_ , 31(26):9481–9488, 2011.
  * Ma et al. (2020) Ma, W.-D. K., Lewis, J., and Kleijn, W. B.  The hsic bottleneck: Deep learning without back-propagation.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 5085–5092, 2020.
  * Meulemans et al. (2021) Meulemans, A., Tristany Farinha, M., Garcia Ordonez, J., Vilimelis Aceituno, P., Sacramento, J., and Grewe, B. F.  Credit assignment in neural networks through deep feedback control.  _Advances in Neural Information Processing Systems_ , 34:4674–4687, 2021.
  * Meulemans et al. (2022) Meulemans, A., Zucchet, N., Kobayashi, S., Von Oswald, J., and Sacramento, J.  The least-control principle for local learning at equilibrium.  _Advances in Neural Information Processing Systems_ , 35:33603–33617, 2022.
  * Moldwin & Segev (2020) Moldwin, T. and Segev, I.  Perceptron learning and classification in a modeled cortical pyramidal cell.  _Frontiers in computational neuroscience_ , 14:33, 2020\.
  * Muezzinoglu et al. (2008) Muezzinoglu, M., Vergara, A., Huerta, R., Nowotny, T., Rulkov, N., Abarbanel, H., Selverston, A., and Rabinovich, M.  Artificial olfactory brain for mixture identification.  _Advances in Neural Information Processing Systems_ , 21, 2008.
  * Nowotny & Huerta (2012) Nowotny, T. and Huerta, R.  On the equivalence of hebbian learning and the svm formalism.  In _2012 46th Annual Conference on Information Sciences and Systems (CISS)_ , pp. 1–4. IEEE, 2012.
  * Obeid et al. (2019) Obeid, D., Ramambason, H., and Pehlevan, C.  Structured and deep similarity matching via structured and deep hebbian networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Oja (1982) Oja, E.  Simplified neuron model as a principal component analyzer.  _J. Math. Biol._ , 15(3):267–273, 1982.
  * Oja (1992) Oja, E.  Principal components, minor components, and linear neural networks.  _Neural Networks_ , 5(6):927–935, 1992.
  * Olshausen & Field (1996) Olshausen, B. A. and Field, D. J.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images.  _Nature_ , 381:607–609, 1996.
  * Olshausen & Field (1997) Olshausen, B. A. and Field, D. J.  Sparse coding with an overcomplete basis set: A strategy employed by v1?  _Vision research_ , 37(23):3311–3325, 1997.
  * Ororbia & Mali (2019) Ororbia, A. G. and Mali, A.  Biologically motivated algorithms for propagating local target representations.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 4651–4658, 2019.
  * Pehlevan & Chklovskii (2015) Pehlevan, C. and Chklovskii, D.  A normative theory of adaptive dimensionality reduction in neural networks.  In _Advances in neural information processing systems_ , pp. 2269–2277, 2015.
  * Pehlevan & Chklovskii (2014) Pehlevan, C. and Chklovskii, D. B.  A Hebbian/anti-Hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features.  In _2014 48th Asilomar Conference on Signals, Systems and Computers_ , pp. 769–775. IEEE, 2014.
  * Pehlevan & Chklovskii (2019) Pehlevan, C. and Chklovskii, D. B.  Neuroscience-inspired online unsupervised learning algorithms: Artificial neural networks.  _IEEE Signal Processing Magazine_ , 36(6):88–96, 2019.
  * Pehlevan et al. (2015) Pehlevan, C., Hu, T., and Chklovskii, D. B.  A Hebbian/anti-Hebbian neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data.  _Neural computation_ , 27(7):1461–1495, 2015\.
  * Pehlevan et al. (2018) Pehlevan, C., Sengupta, A. M., and Chklovskii, D. B.  Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?  _Neural Computation_ , 30(1):84–124, 2018.
  * Pogodin & Latham (2020) Pogodin, R. and Latham, P.  Kernelized information bottleneck leads to biologically plausible 3-factor hebbian learning in deep networks.  _Advances in Neural Information Processing Systems_ , 33:7296–7307, 2020.
  * Rozell et al. (2008) Rozell, C. J., Johnson, D. H., Baraniuk, R. G., and Olshausen, B. A.  Sparse coding via thresholding and local competition in neural circuits.  _Neural computation_ , 20(10):2526–2563, 2008\.
  * Rumelhart & Zipser (1985) Rumelhart, D. E. and Zipser, D.  Feature discovery by competitive learning.  _Cognitive Science_ , 9(1):75–112, 1985.
  * Rumelhart et al. (1986) Rumelhart, D. E., Hinton, G. E., and Williams, R. J.  Learning representations by back-propagating errors.  _nature_ , 323(6088):533–536, 1986.
  * Sacramento et al. (2018) Sacramento, J., Costa, R. P., Bengio, Y., and Senn, W.  Dendritic cortical microcircuits approximate the backpropagation algorithm.  In _Advances in Neural Information Processing Systems_ , pp. 8721–8732, 2018.
  * Sanger (1989) Sanger, T. D.  Optimal unsupervised learning in a single-layer linear feedforward neural network.  _Neural Networks_ , 2(6):459–473, 1989.
  * Schölkopf et al. (1998) Schölkopf, B., Smola, A., and Müller, K.-R.  Nonlinear component analysis as a kernel eigenvalue problem.  _Neural computation_ , 10(5):1299–1319, 1998\.
  * Schölkopf et al. (2001) Schölkopf, B., Herbrich, R., and Smola, A. J.  A generalized representer theorem.  In _International conference on computational learning theory_ , pp. 416–426. Springer, 2001.
  * Schrimpf et al. (2018) Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al.  Brain-score: Which artificial neural network for object recognition is most brain-like?  _BioRxiv_ , pp. 407007, 2018.
  * Sjöström et al. (2001) Sjöström, P. J., Turrigiano, G. G., and Nelson, S. B.  Rate, timing, and cooperativity jointly determine cortical synaptic plasticity.  _Neuron_ , 32(6):1149–1164, 2001.
  * Song et al. (2005) Song, S., Sjöström, P. J., Reigl, M., Nelson, S., and Chklovskii, D. B.  Highly nonrandom features of synaptic connectivity in local cortical circuits.  _PLoS biology_ , 3(3):e68, 2005.
  * Tibshirani (1996) Tibshirani, R.  Regression shrinkage and selection via the lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 58(1):267–288, 1996.
  * Wahba (1990) Wahba, G.  _Spline models for observational data_.  SIAM, 1990.
  * Whittington & Bogacz (2017) Whittington, J. C. and Bogacz, R.  An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity.  _Neural computation_ , 29(5):1229–1262, 2017\.
  * Williams (2001) Williams, C. K.  On a connection between kernel PCA and metric multidimensional scaling.  In _Advances in neural information processing systems_ , pp. 675–681, 2001.
  * Wiskott & Sejnowski (2002) Wiskott, L. and Sejnowski, T. J.  Slow feature analysis: Unsupervised learning of invariances.  _Neural computation_ , 14(4):715–770, 2002.
  * Zemel & Pitassi (2000) Zemel, R. and Pitassi, T.  A gradient-based boosting algorithm for regression problems.  _Advances in neural information processing systems_ , 13, 2000.
  * Zhang (2002) Zhang, T.  On the dual formulation of regularized linear systems with convex risks.  _Machine Learning_ , 46(1-3):91–129, 2002.
  * Zucchet & Sacramento (2022) Zucchet, N. and Sacramento, J.  Beyond backpropagation: implicit gradients for bilevel optimization.  _arXiv preprint arXiv:2205.03076_ , 2022.


predictions of NFCL-V and C. However, NFCL-D demonstrates similar prediction
results to other models for series 2. In regions 2 and 3, NFCL-D demonstrates
incorrect pattern predictions. Furthermore, in Region 3, all NFCL variants
display inaccurate time-series forecast results, with NFCL-D notably
exhibiting a larger error compared to the other NFCLs.

Dataset | Split | Variation | MAE | MSE | SMAPE | $R^{2}$
---|---|---|---|---|---|---
ILL | Train | NFCL-V | 0.390 | 0.387 | 37.846 | 0.629
NFCL-C | 0.288 | 0.224 | 30.394 | 0.785
NFCL-D | 0.287 | 0.213 | 30.451 | 0.796
Val | NFCL-V | 0.453 | 0.472 | 40.218 | 0.366
NFCL-C | 0.366 | 0.288 | 32.873 | 0.599
NFCL-D | 0.378 | 0.290 | 33.719 | 0.582
Test | NFCL-V | 1.261 | 5.197 | 42.969 | -0.211
NFCL-C | 1.320 | 7.211 | 40.082 | -0.680
NFCL-D | 1.497 | 11.411 | 40.067 | -1.569
Weather | Train | NFCL-V | 0.171 | 0.276 | 20.017 | 0.724
NFCL-C | 0.159 | 0.250 | 19.656 | 0.750
NFCL-D | 0.163 | 0.257 | 19.924 | 0.743
Val | NFCL-V | 0.189 | 0.348 | 20.431 | 0.706
NFCL-C | 0.183 | 0.319 | 20.491 | 0.730
NFCL-D | 0.186 | 0.328 | 20.741 | 0.723
Test | NFCL-V | 0.109 | 0.091 | 16.712 | 0.704
NFCL-C | 0.108 | 0.089 | 16.691 | 0.716
NFCL-D | 0.110 | 0.090 | 16.975 | 0.711
Table 15. The train-validation-test performance comparison according to NFCLs.

A glimpse into the cause of NFCL-D’s poor performance can also be obtained
from Table 15. The first row of the table outlines the dataset used for
learning, the data division method, and the type of NFCL employed. NFCL
utilized one layer and 32 hidden units, while NFCL-D’s decomposition was
configured as 10,4,1. Furthermore, each experiment was conducted only once.

Results from the experiments indicate that in the case of the Weather dataset,
which contains a substantial amount of data, both NFCL-C and NFCL-D
consistently exhibit performance enhancements over NFCL-V across all training,
validation, and evaluation datasets. However, when considering the ILL
dataset, NFCL-D demonstrated notably strong performance during training but
displayed poor performance indicators during testing, specifically on the
evaluation dataset. This discrepancy highlights an instance of NFCL-D
overfitting to the training data, as evidenced by inappropriate pattern
predictions (as illustrated in Figure 9). Consequently, it becomes evident
that NFCLs can effectively capture the data distribution given a sufficient
volume of data.