SSupplementary References

# The least-control principle
for local learning at equilibrium

Alexander Meulemans ${}^{~{}\text{ 1}}$, Nicolas Zucchet${}^{*\text{1}}$,
Seijin Kobayashi${}^{*\dagger\text{1}}$
Johannes von Oswald${}^{\text{1}}$, João Sacramento${}^{\text{2}}$

${}^{\text{1}}$Department of Computer Science, ETH Zürich
${}^{\text{2}}$Institute of Neuroinformatics, University of Zürich and ETH
Zürich
{ameulema, nzucchet, seijink, voswaldj<EMAIL_ADDRESS>Equal contributionWork
partly done at the Institute of Neuroinformatics, University of Zürich & ETH
Zürich

###### Abstract

Equilibrium systems are a powerful way to express neural computations. As
special cases, they include models of great current interest in both
neuroscience and machine learning, such as deep neural networks, equilibrium
recurrent neural networks, deep equilibrium models, or meta-learning. Here, we
present a new principle for learning such systems with a temporally- and
spatially-local rule. Our principle casts learning as a _least-control_
problem, where we first introduce an optimal controller to lead the system
towards a solution state, and then define learning as reducing the amount of
control needed to reach such a state. We show that incorporating learning
signals within a dynamics as an optimal control enables transmitting activity-
dependent credit assignment information, avoids storing intermediate states in
memory, and does not rely on infinitesimal learning signals. In practice, our
principle leads to strong performance matching that of leading gradient-based
learning methods when applied to an array of problems involving recurrent
neural networks and meta-learning. Our results shed light on how the brain
might learn and offer new ways of approaching a broad class of machine
learning problems.

###  1 Introduction

The neural networks of the cortex are both layered and highly recurrent [1,
2]. Their high degree of recurrence and relatively low depth stands in
contrast to the prevailing design of artificial neural networks, which have
high depth and little to no recurrence. This discrepancy has triggered a
recent wave of research into recurrent networks that are more brain-like and
which achieve high performance in perceptual tasks [3, 4, 5, 6, 7].
Concurrently, another line of recent work has shown that repeating a short
sequence of neural computations until convergence can lead to large gains in
efficiency, reaching the state-of-the-art in various machine learning problems
while reducing model size [8, 9, 10].

As we develop models that come closer to cortical networks by way of their
recurrence, the precise mechanisms supporting learning in the brain remain
largely unknown. While gradient-based methods currently dominate machine
learning, standard methods for efficient gradient computation result in non-
local rules that are hard to interpret in biological terms [11, 12]. This
issue is particularly aggravated when applying these methods to complex
systems involving recurrence [13]. Indeed, while multiple interesting
proposals [14, 15, 16, 17, 18, 19, 20, 21, 22, 23] have emerged for how to
efficiently compute loss function gradients for feedforward networks in
biologically-plausible ways, apart from a few notable exceptions [24, 25] much
less progress has been made for recurrent networks. Furthermore, the majority
of these methods requires that error feedback does not influence network
activity, a property at odds with many experimental findings on activity-
dependent plasticity [26]. In this paper, we focus on the problem of learning
such recurrent systems using biologically-plausible, activity-dependent local
rules. To make progress in this longstanding question we make the assumption
that our system is at equilibrium, and formalize learning as the following
optimization problem:

$\min{}_{\\!\theta}\,L(\phi^{*})\quad\mathrm{s.t.}\enspace
f(\phi^{*},\theta)=0,$ (1)

where $\theta$ are the parameters we wish to learn, $L$ is a loss function
which measures performance, and $\dot{\phi}=f(\phi,\theta)$ is the system
dynamics which is at an equilibrium point $\phi^{*}$. Our model is thus
_implicitly_ defined through a dynamics that is at equilibrium. Expressing our
problem in this general form allows us to model a very broad class of learning
systems, without being restricted to a particular type of neural network [27,
28, 29]. In particular, we cover both feedforward and recurrent neural
architectures. More generally, $f$ is not even restricted to being a neural
dynamics. Consider the case where $f$ defines a learning algorithm governing
the dynamics of the _weights_ of a network; in this case (1) becomes a meta-
learning problem, where the goal is to tune the (meta-)parameters of a
learning algorithm such that its performance improves.

Inspired by a recent control-based learning method for feedforward neural
networks [30], we present a new principle for solving problems of the form (1)
which yields learning rules that are (i) gradient-following, (ii) local in
space, (iii) local in time, and in particular do not require storing
intermediate states, (iv) activity-dependent by embedding error information
into the network activity, and (v) not reliant on infinitesimally small
learning signals. To meet all five criteria at once, we depart from direct
gradient-based optimization of the loss, and we reformulate gradient-based
learning within the framework of optimal control as a problem of _least-
control_.

Such problems can be approached in two steps. First, an optimal controller
provides additional feedback input leading the dynamical system to a least-
control state: an equilibrium point in which the loss $L$ is minimized with
the least amount of control. Subsequently, the parameters are changed to
further reduce the amount of control at the resulting equilibria. Critically,
this minimization can be achieved using a local gradient-based rule for which
all the required information is available at the controlled equilibria. This
should be contrasted to the ubiquitous backpropagation of error algorithm [31,
32] (and its variant for equilibrium systems [33, 34]) which yields non-local
parameter updates based on the outcome of two separate phases. Importantly,
our least-control problem is intimately related to the original learning
problem; we make the connection between the two precise by identifying
mathematical conditions under which least-control solutions correspond to
minima of the original objective $L(\phi^{*})$, or a bound of it.

We apply our principle to learn equilibrium neural networks featuring a
variety of architectures, including both fully-connected and convolutional
feedfoward networks, laterally-connected recurrent networks of leaky
integrator neurons, as well as deep equilibrium models [8, 9, 10], a recent
family of high-performance models which repeat until convergence a sequence of
complex computations. To further demonstrate the generality of our principle,
we then consider a recently studied meta-learning problem where the goal is to
change the internal state of a complex synapse such that future learning
performance is improved [35]. We find that our single-phase local learning
rules yield highly competitive performance in both application domains. Our
results extend the limits of what can be achieved with local learning rules,
opening novel avenues to an old problem.

###  2 The least-control principle

Figure 1: (A, B) Intuition behind the least-control principle. (A) During
learning, the free dynamics (dark blue) is augmented with a controller (light
blue) that drives the system towards an equilibrium that minimizes the loss
function $L$ (grey, the darker the smaller the loss is). Our principle
prescribes minimizing the amount of optimal control
$\psi_{\scalebox{0.42}{$\bigstar$}}$ (light green) needed at an equilibrium
that minimizes $L$. (B) After learning, the free equilibrium coincides with
the controlled equilibrium and hence minimizes the loss $L$. (C) Example of an
instantiation of the principle for the supervised learning of a feedforward
neural network. The controller both drives the output units $y$ of the network
towards a target value $y^{\mathrm{target}}$ and influences the rest of the
network. The control signal $\psi$ can be computed directly from the output
error or indirectly by leveraging internal feedback within the network.

####  2.1 The principle

We introduce the least-control principle for learning dynamical systems that
evolve in time according to $\dot{\phi}(t)=f(\phi(t),\theta)$ until an
equilibrium $\phi^{*}$ is reached. Instead of directly minimizing a loss
evaluated at equilibrium $L(\phi^{*})$ as in (1), we augment the dynamics
$\dot{\phi}(t)=f(\phi(t),\theta)+\psi(t)$ with a control signal $\psi(t)$ that
drives the system towards a new controlled equilibrium where the loss is
minimized (c.f. Figure 1). Our principle recasts learning as minimizing the
amount of control needed to reach that state:

$\min_{\theta,\phi,\psi}\,\frac{1}{2}\lVert\psi\rVert^{2}\quad\mathrm{s.t.}\enspace
f(\phi,\theta)+\psi=0,~{}\nabla_{\phi}L(\phi)=0.$ (2)

The two constraints in the equation above ensure that we reach a steady state
of the controlled dynamics, and that the loss $L$ is minimized at this state.
Without loss of generality, we consider a single data point above (c.f.
Section S5 in the supplementary materials).

To solve (2), we first fix the parameters $\theta$. We then find an optimal
control $\psi_{\scalebox{0.42}{$\bigstar$}}$ and controlled equilibrium
$\phi_{\scalebox{0.42}{$\bigstar$}}$ that minimize the following least-control
objective, for the current setting of $\theta$,

$\min_{\phi,\psi}\,\frac{1}{2}\lVert\psi\rVert^{2}\quad\mathrm{s.t.}\enspace
f(\phi,\theta)+\psi=0,~{}\nabla_{\phi}L(\phi)=0.$ (3)

We call $\phi_{\scalebox{0.42}{$\bigstar$}}$ the least-control state and
$\psi_{\scalebox{0.42}{$\bigstar$}}$ the least control. Here,
$\psi_{\scalebox{0.42}{$\bigstar$}}$ represents an optimal control with
terminal cost $\lVert\psi\rVert^{2}$, and a terminal constraint enforcing that
the optimally controlled equilibrium $\phi_{\scalebox{0.42}{$\bigstar$}}$
minimizes the loss $L$. Critically, both credit assignment and system dynamics
are now combined into a single phase through the optimal control
$\psi_{\scalebox{0.42}{$\bigstar$}}$. We then take a gradient-following update
in $\theta$ on
$\mathcal{H}(\theta):=\frac{1}{2}\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert^{2}$,
the least-control objective of Eq. 3, which implicitly depends on $\theta$
through $\psi_{\scalebox{0.42}{$\bigstar$}}$.

Theorem 1 shows that the optimal control $\psi_{\scalebox{0.42}{$\bigstar$}}$
contains enough information about the learning objective so that the gradient
is easy to compute. We prove Theorem 1, along with all the theoretical results
from Section 2, in Section S2. We use $\partial_{x}$ to denote partial
derivatives w.r.t. the variable $x$.

######  Theorem 1 (Informal).

Under some mild regularity conditions, the least-control principle yields the
following gradient for $\theta$:

$\displaystyle\nabla_{\theta}\mathcal{H}(\theta)$
$\displaystyle=-\partial_{\theta}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}\psi_{\scalebox{0.42}{$\bigstar$}}=\partial_{\theta}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta).$
(4)

To illustrate what our principle achieves, let us briefly consider the setting
where $f(\phi,\theta)$ implements a feedforward neural network, typically
learned by backpropagation of a supervised error signal. In this case, the
variable $\phi$ corresponds to the state of the neurons (e.g. their firing
rate or postsynaptic voltage) and $\theta$ to the synaptic connection weights.
The least control $\psi_{\scalebox{0.42}{$\bigstar$}}$ then drives the network
to an equilibrium that minimizes the output loss, while being of minimum norm.
The resulting weight update (c.f. Section S6.2) now represents a local Hebbian
rule multiplying presynaptic input with the postsynaptic control signal: the
neural activity implicitly encodes credit assignment information, in such a
way that local learning becomes possible. By contrast, in the backpropagation
algorithm, errors do not directly influence neural activity and are computed
in a separate phase.

The simplicity of our parameter update becomes apparent when contrasting it
with the gradient associated with the original objective (1), as the implicit
function theorem (c.f. Section S3) gives

$\nabla_{\theta}L(\phi^{*})=-\partial_{\theta}f(\phi^{*},\theta)^{\top}\left(\partial_{\phi}f(\phi^{*},\theta)\right)^{-\top}\nabla_{\phi}L(\phi^{*})\\!.$
(5)

Calculating this gradient directly requires inverting the Jacobian
$\partial_{\phi}f(\phi^{*},\theta)\\!$, which is intractable for large-scale
systems. The standard way of dealing with this issue is to resort to the two-
phase recurrent backpropagation algorithm, which estimates
$\nabla_{\theta}L(\phi^{*})$ by running a second linear dynamics until
equilibrium, while holding the first equilibrium state $\phi^{*}$ in memory.
For acyclic computation graphs, of which feedforward neural networks are a
prime example, it simplifies into the error backpropagation algorithm. Those
procedures are still more complicated than ours: they require implementing a
specialized auxiliary dynamical system, holding intermediate states in memory,
and alternating between inference and gradient computation phases.

####  2.2 A general class of dynamics leads to the same optimal control and
parameter update

Our principle is agnostic to the dynamics $\dot{\psi}(t)$ of the controller;
the only assumption made is that its value $\psi_{*}$ at the steady state is
optimal, i.e. $\psi_{*}=\psi_{\scalebox{0.42}{$\bigstar$}}$. We use the
subscripts ∗ to denote equilibrium states of the controlled dynamics and
${}_{\scalebox{0.42}{$\bigstar$}}$ to indicate optimality on the least-control
objective of Eq. 3, and the superscript ∗ for a free (not controlled)
equilibrium. In particular, the parameter update only depends on information
available at the controlled equilibrium state $(\phi_{*},\psi_{*})$. Here, we
capitalize on this important flexibility by designing different simple
feedback controller dynamics that lead to the same optimal steady state. We
consider feedback controllers which take the following general form:

$\dot{\phi}=f(\phi,\theta)+Q(\phi,\theta)u,\quad\text{and}\quad\dot{u}=-\nabla_{\phi}L(\phi)-\alpha
u,$ (6)

where $u$ is a controller defined on the output units and $Q(\phi,\theta)$
determines the influence of the controller on the entire system. We note that
here and throughout our paper we omit the dependence on time (above, of
$\phi(t)$ and $u(t)$) from our equations to unclutter the presentation. In
(6), the mapping $Q(\phi,\theta)$ can either be a direct mapping from the
output controller $u$ to the entire system, or an indirect one leveraging
internal feedback mechanisms in the system (c.f. Figure 1C). The controller
$u$ may be seen as a leaky integral controller with leak rate $\alpha$, that
uses the output error $\nabla_{\phi}L$ to drive the network towards a minimal-
loss state (c.f. Section S2). Returning to our example of a neural network,
$u$ leads the network into a state configuration where the output neurons $y$
are at the target value $y^{\mathrm{target}}$, when $\alpha=0$.

We use leaky integral control in (6) for explanatory purposes and as we use it
in our experiments. We remark however that our theory goes beyond this form of
output control; in particular, it generalizes to any controller satisfying
$\alpha u_{*}+\nabla_{\phi}L(\phi_{*})=0$ at equilibrium. For example, the
proportional control $u=-\beta\nabla_{\phi}L(\phi)$ satisfies this condition
for $\alpha=\beta^{-1}$.

Theorem 2 provides general conditions for the dynamics (6) to converge to an
optimal control signal needed for obtaining first-order parameter updates in
Theorem 1.

######  Theorem 2.

Let $(\phi_{*},u_{*})$ be a steady state of the generalized dynamics (6).
Assume that $\partial_{\phi}{f}(\phi_{*},\theta)$ is invertible and that the
following column space condition is satisfied at equilibrium:

$\displaystyle\mathrm{col}\left[Q(\phi_{*},\theta)\right]=\mathrm{row}\left[[0,\mathrm{Id}]\,\partial_{\phi}f(\phi_{*},\theta)^{-1}\right]$
(7)

where the matrix $[0,\mathrm{Id}]$ has $|\phi|$ columns and as many rows as
the system has output units. Then, $\psi_{*}=Q(\phi_{*},\theta)u_{*}$ is an
optimal control $\psi_{\scalebox{0.42}{$\bigstar$}}$ for the least-control
objective (3), in the limit $\alpha\rightarrow 0$.

Interestingly, Theorem 2 shows that many different feedback mappings from the
output controller $u$ towards the system state $\phi$ are possible, as long as
the mapping $Q(\phi_{*},\theta)$ satisfies the column space condition (7) at
equilibrium, in the limit $\alpha\rightarrow 0$. Furthermore, the control
$\psi$ need not be instantaneous and can have its own temporal dynamics
$\dot{\psi}$ alongside the output controller dynamics $\dot{u}$, as the column
space condition (7) is only defined at equilibrium. We exploit this property
and construct a general-purpose algorithm that computes an optimal control
$\psi_{\scalebox{0.42}{$\bigstar$}}$:

$\displaystyle\dot{\phi}=f(\phi,\theta)+\psi,\quad\text{and}\quad\dot{\psi}=\partial_{\phi}f(\phi,\theta)^{\top}\psi+u$
(8)

with $u$ the leaky integral controller defined in (6). This inversion dynamics
has two special properties. First, it satisfies the conditions of Theorem 2 by
construction (c.f. Section S2.3). Second, as defined above, $\dot{\psi}$
resembles the second-phase dynamics of the (recurrent) backpropagation
algorithm (c.f. Section S3), or in case of feedforward networks, the layerwise
backpropagation of error signals in the error backpropagation algorithm.
Intuitively, in this case we find an optimally-controlled equilibrium by
_simultaneously_ (in a single phase) running the first- and second-phase
dynamics of (recurrent) backpropagation, with the critical difference that
here the dynamical equations of both phases interact through the relation
$\dot{\phi}=f(\phi,\theta)+\psi$.

####  2.3 The least-control principle solves the original learning problem

Iterating our parameter update minimizes the least-control objective
$\mathcal{H}(\theta)$ and not directly the original objective of learning (1).
The latter is however the objective of ultimate interest, used to measure the
performance of the system after learning, when it is no longer under the
influence of a controller. We now show that there is a close link between the
two objectives and identify a wide range of conditions under which least-
control solutions also solve the original learning problem (1).

Proposition 3 shows the intuitive result that if the optimal control is
minimized to zero, the equilibria of the controlled and free dynamics coincide
and hence the loss $L$ is minimized at the free equilibrium as well (c.f.
Figure 1.B).

######  Proposition 3.

Assuming $L$ is convex on the system output, we have that the optimal control
$\psi_{\scalebox{0.42}{$\bigstar$}}$ is equal to 0 iff. the free equilibrium
$\phi^{*}$ minimizes $L$.

As
$\psi_{\scalebox{0.42}{$\bigstar$}}=-f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$,
reaching $\psi_{\scalebox{0.42}{$\bigstar$}}=0$ can only be done if the model
is powerful enough to perfectly fit the controlled equilibria for all data
points. We show in Proposition 4, that overparameterization, i.e., having more
parameters $\theta$ than system states $\phi$, indeed helps solve the original
learning problem (1). The condition
$\partial_{\theta}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ being of full
row rank, that is needed in this proposition, can only be satisfied when the
dimensions satisfy $|\theta|\geq|\phi|$, with $\phi$ the concatenated states
of all data points, hence providing another perspective on why
overparameterization helps.

######  Proposition 4.

Assuming $L$ is convex on the system output, a local minimizer $\theta$ of the
least control objective $\mathcal{H}$ is a global minimizer of the original
learning problem (1), under the sufficient condition that
$\partial_{\theta}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ has full row
rank.

Fully minimizing the amount of control to zero is not always possible, which
highlights the need to understand the relation between the least-control
objective
$\mathcal{H}(\theta)=\frac{1}{2}\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert^{2}$
and the loss $L(\phi^{*})$ from the original learning problem (1). Proposition
5 shows that we can do so, as minimizing
$\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert^{2}$ indirectly minimizes the
loss $L(\phi^{*})$, under some regularity and strong convexity assumptions.

######  Proposition 5.

If $\lVert f\rVert^{2}$ is $\mu$-strongly convex, $L$ is $M$-Lipschitz
continuous and the minimum of $L$ is 0, then

$L(\phi^{*})\leq\frac{\sqrt{\mu}}{\sqrt{2}M}\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert=\frac{\sqrt{\mu}}{\sqrt{2}M}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert.$

Approximate equilibria. Our least-control theory requires the system to be at
equilibrium, and that the loss is minimized at the controlled equilibrium.
This is almost never the case in practice, for instance because the dynamics
are run for a finite amount of time, the column space condition for $Q$ is not
perfectly satisfied, or because $\alpha$ is not zero in the output controller
$u$ dynamics. The update we take therefore does not strictly follow the
gradient of the least-control objective $\mathcal{H}(\theta)$. We formally
show in Section S2.6 that our update resulting from an approximate optimally-
controlled equilibrium is close to the gradient $\nabla_{\theta}\mathcal{H}$.
More precisely, under some regularity conditions on $f$, the error in the
gradient estimation is upper bounded by the distance between
$\phi_{\scalebox{0.42}{$\bigstar$}}$ and its estimate. Additionally, if we
reach equilibrium but $\alpha$ is non-zero the update
$-\partial_{\theta}f(\phi_{*},\theta)^{\top}\psi_{*}$ follows the gradient of
an objective very similar to the least-control one $\mathcal{H}$, under strict
conditions on the feedback mapping $Q$.

####  2.4 The least-control principle as constrained energy minimization

We now provide a dual view of our least-control principle from the perspective
of energy minimization. This will allow designing a second general-purpose
dynamics for computing the least control $\psi_{\scalebox{0.42}{$\bigstar$}}$,
and establishing a link to another principle for learning known as the free-
energy principle. To arrive at an energy function, we rewrite the least-
control objective (3) by substituting the constraint $\psi=-f(\phi,\theta)$
into this objective, leading to the following constrained optimization
problem:

$\phi_{\scalebox{0.42}{$\bigstar$}}=\operatorname*{arg\,min}{}_{\\!\phi}\,\tfrac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}\quad\mathrm{s.t.}\enspace\nabla_{\phi}L(\phi)=0$ (9)

The role of optimal control can therefore be reinterpreted as finding the
state that is the closest to equilibrium under the free dynamics, among the
states that minimize the loss function.

Next, we introduce the augmented energy
$F(\phi,\theta,\beta):=\frac{1}{2}\lVert f(\phi,\theta)\rVert^{2}+\beta
L(\phi)$, which adds a nudging potential to the equilibrium energy $\lVert
f(\phi,\theta)\rVert^{2}$, and take the _perfect control limit_ ,
$\beta\to\infty$, where this potential dominates the energy. As we show in
Section S2.5, minimizing the augmented energy with respect to the state
variable $\phi$ leads to fulfilling our objective (9). Interestingly, when
$\beta\to\infty$ and $f(\phi,\theta)$ implements a feedforward neural network,
we recover the free-energy function which governs the predictive coding model
of Whittington and Bogacz [36], which was obtained from an entirely different
route based on variational expectation maximization for a probabilistic latent
variable model [37] (c.f. Section S4).

Energy-based dynamics as optimal control. We leverage this connection to
design a second general-purpose dynamics that computes an optimal control, by
using the gradient flow on $F$:

$\displaystyle\dot{\phi}=-\partial_{\phi}f(\phi,\theta)^{\top}f(\phi,\theta)-\beta\nabla_{\phi}L(\phi)$
(10)

Here, the control $\psi$ is implicitly contained in the dynamics of $\phi$.
Part of the dynamics, $-\beta\nabla_{\phi}L(\phi)$, plays the role of an
infinitely-strong proportional controller on the system output; another part
is in charge of optimally sending the teaching signal from the output back to
the rest of the system. We refer to Section S2.5 for more details.

Two opposing limits: perfect control vs. weak nudging. The update
$\partial_{\theta}f^{\top}f$ that we obtain in Theorem 1 has been extensively
studied in the weak nudging limit $\beta\rightarrow 0$ [38, 36, 39], which
sits at the opposite end of the $\beta$-spectrum of the perfect control limit
studied here. A seminal result shows that as $\beta\to 0$ the gradient
$\nabla_{\theta}L(\phi^{*})$ of the original objective function is recovered
[24, 40]. However, this update is known to be sensitive to noise as the value
of $f$ can be very small at the weakly-nudged equilibrium [35]. Our least-
control principle works at the opposite perfect control end of the spectrum
($\beta\rightarrow\infty$), and it is therefore more resistant to noise. One
of our main contributions is to show that using the same update, but instead
evaluated at an optimally-controlled state
$\phi_{\scalebox{0.42}{$\bigstar$}}$, still performs principled gradient-based
optimization, however now on the least-control objective
$\mathcal{H}(\theta)$.

###  3 Applications of the least-control principle

The least-control principle applies to a general class of dynamical systems at
equilibrium. By contrast, the majority of previous work on local learning
focused on designing circuits and rules tailored towards feedforward neural
networks [14, 41, 36, 17, 42, 43, 44, 45, 23]. We now demonstrate the
generality of our principle by applying it to two problems of interest in
neuroscience and machine learning: deep and recurrent neural network learning,
and meta-learning.111Source code for all experiments is available at
github.com/seijin-kobayashi/least-control In both cases, our principle leads
to activity-dependent local learning rules by leveraging Theorem 1, while
making use of simple and flexible optimal controllers to feed back credit
assignment information (c.f. Theorem 2). We test our learning and meta-
learning systems on standard benchmarks and find that they perform
competitively when compared to conventional gradient-based methods.

####  3.1 The least-control principle for feedforward and recurrent neural
networks

As a first demonstration of learning according to our principle we consider
feedforward neural networks with multiple layers of hidden neurons, the
current preferred architecture for a large array of perceptual problems.
Moreover, prior work on biologically-plausible alternatives to backpropagation
has mostly focused on feedforward networks, making them a natural first choice
to study our principle.

Motivated by the massive recurrence of cortical networks, we then turn to
equilibrium recurrent neural networks as a second application of our
principle, aiming to demonstrate its applicability beyond feedforward models.
While there is debate over the precise functional roles of recurrent
processing in the brain, there is strong experimental evidence that these
connections are not limited to transmitting learning signals and play an
active role in information processing [e.g., 46, 47, 48, 49]. From the
viewpoint of machine learning, it has been shown that equilibrium RNNs are
often more parameter-efficient than feedforward networks, while reaching
state-of-the-art performance [3, 7, 9].

More concretely, we consider a generic neural network driven by a fixed input
$x$ assumed to converge to an equilibrium state, obeying the following free
dynamics (e.g. Figure 1.C):

$\dot{\phi}=f(\phi,\theta)=-\phi+W\sigma(\phi)+Ux$ (11)

with $\phi$ the neural activities, $W$ the internal synaptic weight matrix,
$U$ the input weight matrix and $\sigma$ the activation function. Note that
this notation is general enough to encompass feedforward and recurrent network
architectures; while freely chosen $W$ and $U$ yield a vanilla RNN,
structuring $W$ and $U$ appropriately (with block-diagonal structure) allows
recovering standard feedforward fully-connected or convolutional deep neural
networks. We point to Section S6 for full architectural details. The learning
problem then consists of optimizing the network weights $\theta=\\{W,U\\}$ on
a loss $L(\phi^{*})$ at the equilibrium activity $\phi^{*}$. This loss is only
measured on the output neurons, a subset of $\phi$.

We now add a control signal $\psi$ to the dynamics (11). Theorem 1 guarantees
that if the controlled steady state $\psi_{*}$ is optimal, the following
updates minimize the least-control objective $\mathcal{H}(\theta)$:

$\displaystyle\Delta
W=\psi_{*}\sigma(\phi_{*})^{\top},\quad\mathrm{and}\quad\Delta
U=\psi_{*}x^{\top}\\!.$ (12)

To showcase the flexibility of our principle, we use Theorem 2 to design
various simple feedback circuits that compute these control signals, as we
detail next. We visualize those circuits in Figure S2.

#####  3.1.1 A simple controller with direct linear feedback

To instantiate the least-control principle in its simplest form, we project
the output controller $u$ onto the hidden neurons $\phi$ with direct linear
feedback weights $Q$, which is a generalization of a recent feedforward
network learning method [30] to equilibrium neural networks. Broadcasting
output errors directly [50] may be seen as one of the simplest possible ways
of avoiding the weight transport problem [11, 12, 16]. This results in the
following dynamics:

$\displaystyle\dot{\phi}=-\phi+W\sigma(\phi)+Ux+Qu,\quad\mathrm{and}\quad\dot{u}=-\nabla_{\phi}L(\phi)-\alpha
u$ (13)

with $u$ a simple leaky integral output controller (c.f. Section 2.2), and
$\nabla_{\phi}L$ the output error.

Theorem 2 guarantees that $Qu_{*}$ is an optimal control in the limit of
$\alpha\rightarrow 0$, if the feedback weights $Q$ satisfy the column space
condition
$\mathrm{col}[Q]=\mathrm{row}\left[[0,\mathrm{Id}](\mathrm{Id}-W\sigma^{\prime}(\phi_{*}))^{-1}\right]$
for all data samples. For a linear feedback mapping, this condition cannot be
exactly satisfied for multiple samples as the row space is data-dependent,
whereas $Q$ is not. Still, we learn the feedback weights $Q$ to approximately
satisfy this column space condition, by using a local Hebbian rule that
operates simultaneously to the learning of the other weights, inspired by
recent work on feedforward networks [30, 51] (details in Section S6). The
resulting update provably finds feedback weights $Q$ that satisfy the column
space condition for one sample, and we empirically verify that the control
signal it gives in the multiple samples regime is close to optimal (c.f.
Section S6).

Despite the simplicity of this linear feedback controller, we show that the
training procedure described above is still powerful enough to learn both a
two-hidden-layer (each of 256 neurons) fully-connected feedforward network and
an equilibrium RNN with a fully-connected recurrent layer of 256 neurons on
the MNIST digit classification task [52] (c.f. Table 1, LCP-LF). Notably, it
performs almost as well as backpropagation (BP) and recurrent backpropagation
(RBP), the current methods of choice for equilibrium RNN training. We use
fixed point iterations to find the steady states $\phi_{*}$ and $u_{*}$ in Eq.
(13) for computational efficiency (c.f. Section S6 for more simulation
details). We also observe that the empirical performance improves by changing
the parameter updates to $\Delta
W=\sigma^{\prime}(\phi_{*})Qu_{*}\sigma(\phi_{*})^{\top}$ and $\Delta
U=\sigma^{\prime}(\phi_{*})Qu_{*}x^{\top}$ when the column space condition is
not perfectly satisfied, corroborating the findings of Meulemans et al. [30,
51] for feedforward neural networks.

Method | FF | RNN
---|---|---
LCP - LF | 97.73±0.07 | 97.70±0.11
LCP - DI | 98.11±0.07 | 97.58±0.12
LCP - DI (KP) | 98.14±0.09 | 97.75±0.11
LCP - EBD | 98.00±0.03 | 97.60±0.15
BP/RBP | 98.29±0.14 | 97.87±0.19
Table 1: MNIST test set classification accuracy (%) for a feedforward network
(2x256 neurons) and an RNN (256 neurons) trained by resp. backpropagation (BP)
and recurrent backpropagation (RBP), and by the least-control principle (LCP)
with linear feedback (LF), energy-based dynamics (EBD), dynamic inversion (DI)
and with learned feedback weights (DI KP). Mean $\pm$ std computed over 3
seeds.

#####  3.1.2 Learning with general-purpose optimal control dynamics

In the previous section we designed and tested a simple error broadcast
circuit and showed that we can learn it such that the conditions of Theorem 2
are approximately met. This approximate feedback is likely insufficient for
more complex tasks which require harnessing depth or strongly-recurrent
dynamics [53]. This leads us to consider two feedback circuits with more
detailed architectures, which have the capacity to steer the network to an
(exact) optimally-controlled equilibrium.

The first circuit is based on our general-purpose dynamic inversion (8).
Applied to RNNs it yields:

$\displaystyle\dot{\phi}=-\phi+W\sigma(\phi)+Ux+\psi,\quad\text{and}\quad\dot{\psi}=-\psi+\sigma^{\prime}(\phi)S\psi+u.$
(14)

The second circuit exploits the dual energy minimization view of our principle
presented in Section 2.4. Taking the energy-descending dynamics
$\dot{\phi}=-\partial_{\phi}F(\phi,\theta,\beta)^{\top}$ of Eq. 10, we obtain

$\displaystyle\dot{\phi}=-\psi+\sigma^{\prime}(\phi)S\psi+u,\quad\text{and}\quad\psi=\phi-W\sigma(\phi)-Ux,$
(15)

where we use the output controller $u$ of Eq. 6 as a generalization of the
proportional control $-\beta\nabla_{y}L$ to make the two circuits more
directly comparable. Above, we introduce decoupled feedback weights $S$ to
avoid sharing weights between $\phi$ and $\psi$, and we learn $S$ to align
with $W^{\top}$ by using the Kolen-Pollack local learning rule [54, 42], which
simply transposes the weight update $\Delta W$ (12) and adds weight decay to
both update rules: $\Delta S=\sigma(\phi_{*})\psi_{*}^{\top}-\gamma S$.

Both circuits implement optimal control through neural dynamics, avoiding
explicit matrix inverses and phases. In biological terms, they lead to
different interpretations and implementation possibilities, highlighting the
flexibility of our learning principle. The dynamic inversion circuit (14) fits
naturally with dendritic error implementations, where $\psi$ can be linked to
feedback dendritic compartments whose signals steer plasticity [14, 41, 17,
20, 23, 51, 30, 55]. In particular, by leveraging burst-multiplexing circuits
[23, 14] or interneuron microcircuits [17], all information for the dynamics
and weight updates can be made locally available (cf. Section S6). On the
other hand, the energy-minimizing dynamics (15) naturally leads to an
implementation based on prediction and error neuron populations,
characteristic of predictive coding circuits [56, 57, 36, 58]. We provide a
high-level discussion of the two circuit alternatives (14, 15) in Section S6.

Figure 2: Training loss $L$ and the amount of control
$\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert=\sqrt{2\mathcal{H}}$ after
training untill convergence on MNIST with various recurrent layer widths.

First, we test our optimally-controlled neural networks by training a fully-
connected equilibrium RNN on MNIST with circuit (14), while varying the
recurrent layer width, cf. Figure 2. This analysis shows that our theoretical
results of Section 2.3 translate to practice, as for strongly
overparameterized networks, both $\mathcal{H}$ and $L$ are minimized to zero,
whereas for underparameterized networks, minimizing the least-control
objective $\mathcal{H}(\theta)$ leads to a significant decrease in $L$.
Moreover, our feedforward (2x256 hidden neurons) and recurrent (256 hidden
neurons) networks achieve competitive test performance compared to
backpropagation (recurrent backpropagation, for RNNs), both when the feedback
weights are trained (LCP - DI KP) and when they are tied ($S=W^{\top}$; LCP -
DI and LCP-EBD), irrespective of whether we use the energy-based (15) or
dynamic inversion (14) optimal control dynamics, cf. Table 1.

Table 2: C10: CIFAR-10 test accuracy (%) of a convolutional feedforward (FF) and equilibrium RNN (RNN), learned by dynamic inversion (DI; Kollen-Pollack learning without weight sharing: KP) and by standard/recurrent backpropagation (BP/RBP). INR: the peak signal-to-noise ratio (in dB) of an implicit neural representation learned on a natural image dataset [59]. Mean $\pm$ std computed over 3 seeds.  Method | C10 (FF) | C10 (RNN) | INR
---|---|---|---
BP/RBP | 77.58±0.14 | 80.14±0.20 | 25.47±4.16
LCP-DI | 77.28±0.10 | 80.26±0.17 | 25.11±2.90
LCP-KP | 77.16±0.10 | – | –

Next, we test whether the circuits and rules obtained through our principle
scale to more complex architectures and tasks and move to the CIFAR-10 image
classification benchmark [60]. We study two distinct models: a standard
feedforward convolutional neural network, and a deep equilibrium model (DEQ)
consisting of a recurrent block of convolutions [9]. The latter can be
interpreted as an efficient infinite-depth residual neural network with
weight-sharing [61, 62]. Full architectural details are provided in Sections
S6.4.3 and S6.5.2; we note that to showcase our least-control principle as a
scalable learning method in the DEQ experiments, we make a number of design
choices that include using automatic differentiation to compute certain
derivatives. For our feedforward network experiments we make no such
concessions, and we further investigate a weight-transport-free variant using
Kollen-Pollack learning. We show on Table 2 that the networks trained by our
principle are able to match those trained with backpropagation. Interestingly,
we observe that the DEQ models considered here are not overparameterized
enough to not need control, but that this does not impair final predictive
performance (c.f. Table S5).

Lastly, we turn to implicit neural representations [63], yet another task for
which recurrent connections are known to be particularly beneficial [10]. The
network there has to represent an image in its weights so that it is able to
map pixel coordinates to their color values. Briefly, we once again find that
our principle performs competitively to RBP. We refer full details in Section
S6 for more details.

####  3.2 The least-control principle for meta-learning

The generality of the least-control principle enables considering learning
paradigms other than supervised learning. Here, we show that it can be applied
to meta-learning, a framework that aims at improving a learning algorithm by
leveraging shared structure in encountered tasks [64]. This framework is
gaining traction in neuroscience, having been featured in recent work aiming
at understanding which systems [65, 66, 67] and rules [35] could support meta-
learning in the brain.

Meta-learning complex synapses. Following [35, 68], we model the learning
algorithm for training a neural network with synaptic weights $\phi$ on some
task $\tau$ as the minimization of a data-driven loss
$L^{\text{train}}_{\tau}(\phi)$, regularized by a quadratic term
$\lambda||\phi-\omega||^{2}$. This regularizer can be interpreted as a model
of a complex synapse; similar models have been introduced in studies of
continual learning and long-term memory [69, 70, 71, 72, 73]. In the context
of meta-learning, the fast weights $\phi$ quickly learn how to solve a task
and $\lambda$ determines how strongly $\phi$ is pulled towards the slow
weights $\omega$, which consolidate the knowledge that has been previously
acquired. The capabilities of the learning algorithm are measured by
evaluating $L^{\mathrm{eval}}_{\tau}(\phi^{*}_{\tau})$, the performance of the
learned neural network on data from task $\tau$ that was held out during
learning. Meta-learning finally corresponds to improving the learning
performance on many tasks by adjusting the meta-parameters $\theta=\omega$,
leading to the following problem:

$\min_{\theta}\,\mathbb{E}_{\tau}\\!\left[L^{\mathrm{eval}}_{\tau}(\phi^{*}_{\tau})\right]\quad\mathrm{s.t.}\enspace\phi_{\tau}^{*}=\operatorname*{arg\,min}_{\phi}L^{\mathrm{learn}}_{\tau}(\phi)+\frac{\lambda}{2}\lVert\phi-\omega\rVert^{2}.$
(16)

Above, the parameters $\phi$ no longer represent neural activities and
$\theta$ synaptic weights as in the previous section; $\phi$ and
$\theta=\omega$ now designate synaptic weights and meta-parameters,
respectively.

Method | Test accuracy
---|---
FOMAML [74] | 89.40±0.50
T1-T2 [75] | 89.72±0.43
Reptile [76] | 89.43±0.14
LCP (ours) | 91.00±0.24
iMAML [68] | 94.46±0.42
CML [35] | 94.16±0.12
Table 3: Meta-learning neural networks with complex synapses on Omniglot
20-way 1-shot learning tasks. Mean $\pm$ std test set classification accuracy
(%) computed over 3 seeds.

Meta-parameter updates. Using the first-order conditions
$-\nabla_{\phi}L^{\mathrm{train}}_{\tau}(\phi_{\tau}^{*})-\lambda(\phi_{\tau}^{*}-\theta)=0$
associated to $\phi_{\tau}^{*}$ being a minimizer in the previous equation
yields an optimization problem on which the least-control principle can be
applied. Our principle prescribes running the controlled dynamics

$\begin{split}\dot{\phi}&=-\nabla_{\phi}L^{\mathrm{learn}}_{\tau}(\phi)-\lambda(\phi-\omega)+u,\\\
\dot{u}&=-\alpha u-\nabla_{\phi}L^{\mathrm{eval}}_{\tau}(\phi)\end{split}$
(17)

or any other process that finds the same fixed points
$(\phi_{*}^{\tau},u_{*}^{\tau})$. The theory we developed in Section 2.2
guarantees that $u_{*}^{\tau}$ is an optimal control as long as
$\alpha\rightarrow 0$ and the Hessian of the evaluation loss
$\partial_{\phi}^{2}L^{\mathrm{eval}}_{\tau}$ at the equilibrium
$\phi_{*}^{\tau}$ is invertible. Hence, we do not need complex dynamics to
compute the gradient, highlighting the flexibility and simplicity of learning
according to the least-control principle. The resulting single-phase meta-
learning rule is local to every synapse (c.f. Section S7):

$\displaystyle\Delta\omega=\lambda u_{*}^{\tau}.$ (18)

Experimental results. Our meta-learning algorithm stands out from competing
methods as being both principled, first-order and single-phase. On the one
hand, existing first-order and single-phase methods either approximate meta-
gradients (e.g., FOMAML [74] and T1-T2 [75]) or rely on heuristics (e.g.
Reptile [76]), and therefore do not provably minimize a meta-objective
function. On the other hand, full meta-gradient descent methods require two
phases to backpropagate through learning [74], invert an intractable Hessian
[77, 68], or learn a slightly modified task [35], and often require second
derivatives [74, 77, 68]. Here, we compare our rule to competing first-order
methods (FOMAML, T1-T2 and Reptile) and meta-gradient approximation algorithms
that use the same synaptic model (implicit MAML and contrastive meta-learning;
iMAML and CML) on the Omniglot few-shot visual classification benchmark [78].
Each Omniglot task consists of distinguishing 20 classes after seeing only 1
sample per class; we choose this 20-way 1-shot regime as the gap between
first- and second-order methods is reputedly the largest compared to other
variants of this benchmark. We find that our meta-learning rule significantly
outperforms other single-phased first-order methods, while still lagging
behind iMAML, a second-order and two-phased method, and CML, a two-phased
first-order method that can approximate the meta-gradient arbitrarily well.
These findings are particularly interesting since our synaptic model (16) is
in the underparameterized regime: there is a separate set of weights
$\phi_{\tau}$ per task $\tau$ but only one set of consolidated weights
$\omega$ for the whole task distribution. The strong performance observed here
is another demonstratation that least-control solutions can perform well even
when overparameterization arguments (cf. Section 2.3) do not hold.

###  4 Discussion

Control theories of neural network learning. The relationship between control
theory and neural network learning is an old and intricate one. It is by now
well known that backpropagation, the method of choice for computing gradients
in deep learning, can be seen as the discrete-time counterpart of the adjoint
method developed in control theory for continuous-time systems [79, 80]. One
of the early papers realizing this connection formulated neural network
learning as an optimization problem under equilibrium constraints, as in the
original problem statement (1), and rederived the recurrent backpropagation
algorithm [81]. More recent work has explored using feedback neural control
circuits which influence the neural dynamics of feedforward [51, 30] and
recurrent [82, 83, 84] networks to deliver learning signals. Different from
such approaches, our least-control principle is rooted in _optimal_ control:
instead of setting the control from an ansatz, we derived it from first
principles. This allowed us to arrive at a simple local rule for gradient-
based learning, and to identify general sufficient conditions for the feedback
control signals.

Predictive coding from the least-control perspective. We have shown that
least-control problems correspond to free-energy minimization problems. This
dual view of least-control problems allows connecting to influential theories
of neural information processing based on predictive coding [56, 37, 36, 58,
19]. Our results elucidate predictive coding theories in at least three ways.
First, Theorem 1 justifies the parameter update taken in supervised predictive
coding networks [36] as gradient-based optimization of our least-control
objective. This connection offers a novel view of the predictive coding
dynamics as an optimal control algorithm in charge of controlling hidden
neurons, while ‘clamping’ output neurons to the respective targets can be seen
as a form of output control. Previous arguments for gradient-based learning
were only known for the weak nudging limit ($\beta\to 0$) where learning
signals are vanishingly small [24], which is problematic in noisy systems such
as the cortex [85]; our novel theoretical and experimental findings
characterize the opposite perfect control ($\beta\to\infty$) end of the
spectrum. Second, Theorem 2 broadens the class of neural network circuits
which may be used to implement predictive coding, beyond the precisely-
constructed microcircuits that have been proposed so far [36, 17]. Finally,
our experiments on recurrent neural networks using free-energy-minimizing
dynamics to assign credit are to the best of our knowledge the first of the
kind, and serve to validate the effectiveness of predictive coding circuits
beyond feedforward architectures.

Limitations. A limiting factor of our theory is that in its current form it is
only applicable to equilibrium systems. While the study of neural dynamics at
equilibrium is an old endeavor in neuroscience and neural network theory [86,
87], it remains unclear how strong an assumption it is when modeling the
networks of the brain. Extending our principle to out-of-equilibrium dynamic
is an exciting direction for future work. Another limitation concerning the
breadth of problem (1) is that we did not allow the loss function $L$ to
directly depend on the parameters $\theta$. We discuss extensions to this more
general case in Section S2.2, however not all loss functions $L$ in this more
general class satisfy the conditions needed to obtain a local parameter
update. Lastly, it should be noted that although our principle requires only
one relaxation phase, its computational cost often exceeds the combined cost
of performing the two phases of recurrent backpropagation (cf. Section S6.6).

Concluding remarks. We have presented a new theory that enables learning
equilibrium systems with local gradient-based rules. Our learning rules are
driven by changes in activity generated by an optimal control, in charge of
feeding back credit assignment information into the system as it evolves.
Taken together, our results push the boundaries of what is possible with
single-phase, activity-dependent local learning rules, and suggest an
alternative to direct gradient-based learning which yields high performance in
practice.

### Acknowledgments and Disclosure of Funding

This research was supported by an Ambizione grant (PZ00P3_186027) from the
Swiss National Science Foundation and an ETH Research Grant (ETH-23 21-1)
awarded to João Sacramento. Seijin Kobayashi was supported by the Swiss
National Science Foundation (SNF) grant CRSII5_173721. Johannes von Oswald was
funded by the Swiss Data Science Center (J.v.O. P18-03). We thank Angelika
Steger, Benjamin Scellier, Walter Senn, Il Memming Park, Blake A. Richards,
Rafal Bogacz, Yoshua Bengio, and members of the Senn, Bogacz and Richards labs
for discussions.

### References

  * Felleman and van Essen [1991] D. J. Felleman and D. C. van Essen.  Distributed hierarchical processing in the primate cerebral cortex.  _Cerebral Cortex_ , 1(1):1–47, 1991.
  * Douglas and Martin [2004] Rodney J. Douglas and Kevan A.C. Martin.  Neuronal circuits of the neocortex.  _Annual Review of Neuroscience_ , 27(1):419–451, 2004.
  * Liao and Poggio [2016] Qianli Liao and Tomaso Poggio.  Bridging the gaps between residual learning, recurrent neural networks and visual cortex.  _arXiv preprint arXiv:1604.03640_ , 2016.
  * Nayebi et al. [2018] Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J. DiCarlo, and Daniel L. Yamins.  Task-driven convolutional recurrent models of the visual system.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Kubilius et al. [2019] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. Yamins, and James J. DiCarlo.  Brain-like object recognition with high-performing shallow recurrent ANNs.  In _Advances in Neural Information Processing Systems_ , 2019\.
  * van Bergen and Kriegeskorte [2020] Ruben S. van Bergen and Nikolaus Kriegeskorte.  Going in circles is the way forward: the role of recurrence in visual inference.  _Current Opinion in Neurobiology_ , 65:176–193, 2020.
  * Linsley et al. [2020] Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre.  Stable and expressive recurrent vision models.  In _Advances in Neural Information Processing Systems_ , 2020\.
  * Bai et al. [2019] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun.  Deep equilibrium models.  _Advances in Neural Information Processing Systems_ , 2019.
  * Bai et al. [2020] Shaojie Bai, Vladlen Koltun, and J. Zico Kolter.  Multiscale deep equilibrium models.  _Advances in Neural Information Processing Systems_ , 2020.
  * Huang et al. [2021] Zhichun Huang, Shaojie Bai, and J. Zico Kolter.  Implicit${}^{\textrm{2}}$: implicit layers for implicit representations.  In _Advances in Neural Information Processing Systems_ , 2021\.
  * Grossberg [1987] Stephen Grossberg.  Competitive learning: From interactive activation to adaptive resonance.  _Cognitive science_ , 11(1):23–63, 1987.
  * Crick [1989] Francis Crick.  The recent excitement about neural networks.  _Nature_ , 337:129–132, 1989.
  * Lillicrap and Santoro [2019] Timothy P. Lillicrap and Adam Santoro.  Backpropagation through time and the brain.  _Current Opinion in Neurobiology_ , 55:82–89, 2019.
  * Körding and König [2001] Konrad P Körding and Peter König.  Supervised and unsupervised learning with two sites of synaptic integration.  _Journal of Computational Neuroscience_ , 11(3):207–215, 2001.
  * Lee et al. [2015] Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio.  Difference target propagation.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , 2015.
  * Lillicrap et al. [2016] Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman.  Random synaptic feedback weights support error backpropagation for deep learning.  _Nature Communications_ , 7(1):13276, 2016.
  * Sacramento et al. [2018] João Sacramento, Rui P. Costa, Yoshua Bengio, and Walter Senn.  Dendritic cortical microcircuits approximate the backpropagation algorithm.  In _Advances in Neural Information Processing Systems_ , 2018\.
  * Roelfsema and Holtmaat [2018] Pieter R. Roelfsema and Anthony Holtmaat.  Control of synaptic plasticity in deep cortical networks.  _Nature Reviews Neuroscience_ , 19(3):166–180, 2018.
  * Whittington and Bogacz [2019] James C. R. Whittington and Rafal Bogacz.  Theories of error back-propagation in the brain.  _Trends in Cognitive Sciences_ , 23(3):235–250, 2019.
  * Richards and Lillicrap [2019] Blake A. Richards and Timothy P. Lillicrap.  Dendritic solutions to the credit assignment problem.  _Current Opinion in Neurobiology_ , 54:28–36, 2019.
  * Richards et al. [2019] Blake A. Richards, Timothy P. Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lindsay, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording.  A deep learning framework for neuroscience.  _Nature Neuroscience_ , 22(11):1761–1770, 2019\.
  * Lillicrap et al. [2020] Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton.  Backpropagation and the brain.  _Nature Reviews Neuroscience_ , 21(6):335–346, 2020.
  * Payeur et al. [2021] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud.  Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits.  _Nature Neuroscience_ , 24(7):1010–1019, 2021\.
  * Scellier and Bengio [2017] Benjamin Scellier and Yoshua Bengio.  Equilibrium propagation: bridging the gap between energy-based models and backpropagation.  _Frontiers in Computational Neuroscience_ , 11, 2017.
  * Bellec et al. [2020] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass.  A solution to the learning dilemma for recurrent networks of spiking neurons.  _Nature Communications_ , 11(1):3625, 2020.
  * Martin et al. [2000] Stephen J. Martin, Paul D. Grimwood, and Richard G.M. Morris.  Synaptic plasticity and memory: an evaluation of the hypothesis.  _Annual Review of Neuroscience_ , 23(1):649–711, 2000.
  * Amos [2019] Brandon Amos.  _Differentiable optimization-based modeling for machine learning_.  PhD Thesis, Carnegie Mellon University, 2019.
  * Gould et al. [2021] Stephen Gould, Richard Hartley, and Dylan John Campbell.  Deep declarative networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * El Ghaoui et al. [2021] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai.  Implicit deep learning.  _SIAM Journal on Mathematics of Data Science_ , 3(3):930–958, 2021.
  * Meulemans et al. [2022] Alexander Meulemans, Matilde Tristany Farinha, Maria R. Cervera, João Sacramento, and Benjamin F. Grewe.  Minimizing control for credit assignment with strong feedback.  _arXiv preprint arXiv:2204.07249_ , 2022.
  * Werbos [1974] Paul Werbos.  _Beyond regression: new tools for prediction and analysis in the behavioral sciences_.  Ph.D. thesis, Harvard University, 1974.
  * Rumelhart et al. [1986] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.  Learning representations by back-propagating errors.  _Nature_ , 323(6088):533–536, 1986.
  * Almeida [1989] Luís B. Almeida.  Backpropagation in perceptrons with feedback.  In Rolf Eckmiller and Christoph v.d. Malsburg, editors, _Neural Computers_ , pages 199–208. Springer Berlin Heidelberg, 1989.
  * Pineda [1989] Fernando J. Pineda.  Recurrent backpropagation and the dynamical approach to adaptive neural computation.  _Neural Computation_ , 1(2):161–172, 1989.
  * Zucchet et al. [2022] Nicolas Zucchet, Simon Schug, Johannes von Oswald, Dominic Zhao, and João Sacramento.  A contrastive rule for meta-learning.  _Advances in Neural Information Processing Systems_ , 2022.
  * Whittington and Bogacz [2017] James C. R. Whittington and Rafal Bogacz.  An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity.  _Neural Computation_ , 29(5):1229–1262, 2017\.
  * Friston [2009] Karl Friston.  The free-energy principle: a rough guide to the brain?  _Trends in Cognitive Sciences_ , 13(7):293–301, 2009.
  * Carreira-Perpinan and Wang [2014] Miguel Carreira-Perpinan and Weiran Wang.  Distributed optimization of deeply nested systems.  In _Artificial Intelligence and Statistics_ , 2014.
  * Dold et al. [2019] Dominik Dold, Akos F. Kungl, João Sacramento, Mihai A. Petrovici, Kaspar Schindler, Jonathan Binas, Yoshua Bengio, and Walter Senn.  Lagrangian dynamics of dendritic microcircuits enables real-time backpropagation of errors.  In _Computational and Systems Neuroscience (Cosyne)_ , 2019\.
  * Scellier [2021] Benjamin Scellier.  _A deep learning theory for neural networks grounded in physics_.  PhD Thesis, Université de Montréal, 2021.
  * Guerguiev et al. [2017] Jordan Guerguiev, Timothy P. Lillicrap, and Blake A. Richards.  Towards deep learning with segregated dendrites.  _eLife_ , 6:e22901, 2017.
  * Akrout et al. [2019] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy P. Lillicrap, and Douglas B. Tweed.  Deep learning without weight transport.  In _Advances in Neural Information Processing Systems_ , 2019\.
  * Meulemans et al. [2020] Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin F. Grewe.  A theoretical framework for target propagation.  In _Advances in Neural Information Processing Systems_ , 2020\.
  * Podlaski and Machens [2020] William F. Podlaski and Christian K. Machens.  Biological credit assignment through dynamic inversion of feedforward networks.  In _Advances in Neural Information Processing Systems_ , 2020\.
  * Pozzi et al. [2020] Isabella Pozzi, Sander Bohte, and Pieter Roelfsema.  Attention-gated brain propagation: how the brain can implement reward-based error backpropagation.  _Advances in Neural Information Processing Systems_ , pages 2516–2526, 2020.
  * Gilbert and Li [2013] Charles D. Gilbert and Wu Li.  Top-down influences on visual processing.  _Nature Reviews Neuroscience_ , 14(5):350–363, 2013.
  * Manita et al. [2015] Satoshi Manita, Takayuki Suzuki, Chihiro Homma, Takashi Matsumoto, Maya Odagawa, Kazuyuki Yamada, Keisuke Ota, Chie Matsubara, Ayumu Inutsuka, Masaaki Sato, Masamichi Ohkura, Akihiro Yamanaka, Yuchio Yanagawa, Junichi Nakai, Yasunori Hayashi, Matthew E. Larkum, and Masanori Murayama.  A top-down cortical circuit for accurate sensory perception.  _Neuron_ , 86(5):1304–1316, 2015.
  * Marques et al. [2018] Tiago Marques, Julia Nguyen, Gabriela Fioreze, and Leopoldo Petreanu.  The functional organization of cortical feedback inputs to primary visual cortex.  _Nature Neuroscience_ , 21(5):757–764, 2018.
  * Kirchberger et al. [2021] Lisa Kirchberger, Sreedeep Mukherjee, Ulf H. Schnabel, Enny H. van Beest, Areg Barsegyan, Christiaan N. Levelt, J. Alexander Heimel, Jeannette A. M. Lorteije, Chris van der Togt, Matthew W. Self, and Pieter R. Roelfsema.  The essential role of recurrent processing for figure-ground perception in mice.  _Science Advances_ , 7(27):eabe1833, 2021.
  * Nøkland [2016] Arild Nøkland.  Direct feedback alignment provides learning in deep neural networks.  In _Advances in Neural Information Processing Systems_ , 2016\.
  * Meulemans et al. [2021] Alexander Meulemans, Matilde Tristany Farinha, Javier Garcia Ordonez, Pau Vilimelis Aceituno, João Sacramento, and Benjamin F. Grewe.  Credit assignment in neural networks through deep feedback control.  In _Advances in Neural Information Processing Systems_ , 2021\.
  * LeCun [1998] Yann LeCun.  The MNIST database of handwritten digits.  _Available at http://yann. lecun. com/exdb/mnist_ , 1998.
  * Bartunov et al. [2018] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E. Hinton, and Timothy P. Lillicrap.  Assessing the scalability of biologically-motivated deep learning algorithms and architectures.  In _Advances in neural information processing systems_ , 2018.
  * Kolen and Pollack [1994] John F. Kolen and Jordan B. Pollack.  Back-propagation without weight transport.  In _Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94)_ , 1994.
  * Mikulasch et al. [2022] Fabian A. Mikulasch, Lucas Rudelt, Michael Wibral, and Viola Priesemann.  Dendritic predictive coding: A theory of cortical computation with spiking neurons.  _arXiv preprint arXiv:2205.05303_ , 2022.
  * Rao and Ballard [1999] Rajesh P. N. Rao and Dana H. Ballard.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.  _Nature Neuroscience_ , 2(1):79–87, 1999.
  * Bastos et al. [2012] Andre M. Bastos, W. Martin Usrey, Rick A. Adams, George R. Mangun, Pascal Fries, and Karl J. Friston.  Canonical microcircuits for predictive coding.  _Neuron_ , 76(4):695–711, 2012.
  * Keller and Mrsic-Flogel [2018] Georg B. Keller and Thomas D. Mrsic-Flogel.  Predictive processing: a canonical cortical computation.  _Neuron_ , 100(2):424–435, 2018.
  * Fathony et al. [2020] Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J. Zico Kolter.  Multiplicative filter networks.  September 2020.
  * Krizhevsky and Hinton [2009] Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016.
  * Chen et al. [2018] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud.  Neural ordinary differential equations.  In _Advances in Neural Information Processing Systems_ , 2018\.
  * Sitzmann et al. [2020] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In _Advances in Neural Information Processing Systems_ , 2020\.
  * Thrun and Pratt [1998] Sebastian Thrun and Lorien Pratt.  _Learning to learn_.  Springer US, 1998.
  * Behrens et al. [2018] Timothy E. J. Behrens, Timothy H Muller, James C. R. Whittington, Shirley Mark, Alon B. Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson.  What is a cognitive map? Organizing knowledge for flexible behavior.  _Neuron_ , 100(2):490–509, 2018.
  * Wang et al. [2018] Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, and Matthew Botvinick.  Prefrontal cortex as a meta-reinforcement learning system.  _Nature Neuroscience_ , 21(6):860–868, 2018.
  * Wang [2021] Jane X. Wang.  Meta-learning in natural and artificial intelligence.  _Current Opinion in Behavioral Sciences_ , 38:90–95, 2021\.
  * Rajeswaran et al. [2019] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine.  Meta-learning with implicit gradients.  In _Advances in Neural Information Processing Systems_ , 2019\.
  * Fusi et al. [2005] Stefano Fusi, Patrick J. Drew, and Larry F. Abbott.  Cascade models of synaptically stored memories.  _Neuron_ , 45(4):599–611, 2005.
  * Ziegler et al. [2015] Lorric Ziegler, Friedemann Zenke, David B. Kastner, and Wulfram Gerstner.  Synaptic consolidation: from synapses to behavioral modeling.  _Journal of Neuroscience_ , 35(3):1319–1334, 2015\.
  * Benna and Fusi [2016] Marcus K. Benna and Stefano Fusi.  Computational principles of synaptic memory consolidation.  _Nature Neuroscience_ , 19(12):1697–1706, 2016\.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _International Conference on Machine Learning_ , 2017.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences of the United States of America_ , 114(13):3521–3526, 2017.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , 2017.
  * Luketina et al. [2016] Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko.  Scalable gradient-based tuning of continuous regularization hyperparameters.  In _International Conference on Machine Learning_ , 2016.
  * Nichol et al. [2018] Alex Nichol, Joshua Achiam, and John Schulman.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Lorraine et al. [2020] Jonathan Lorraine, Paul Vicol, and David Duvenaud.  Optimizing millions of hyperparameters by implicit differentiation.  In _International Conference on Artificial Intelligence and Statistics_ , 2020.
  * Lake et al. [2015] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum.  Human-level concept learning through probabilistic program induction.  _Science_ , 350(6266):1332–1338, 2015.
  * Bryson and Ho [1969] A. E. Bryson and Y. C. Ho.  _Applied optimal control: optimization, estimation, and control_.  Blaisdell Pub. Co., 1969.
  * Baydin et al. [2018] Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind.  Automatic differentiation in machine learning: a survey.  _Journal of Marchine Learning Research_ , 18:1–43, 2018\.
  * LeCun [1988] Yann LeCun.  A theoretical framework for back-propagation.  In _Proceedings of the 1998 Connectionist Models Summer School_ , 1988.
  * Gilra and Gerstner [2017] Aditya Gilra and Wulfram Gerstner.  Predicting non-linear dynamics by stable local learning in a recurrent spiking neural network.  _eLife_ , 6:e28295, 2017.
  * Denève et al. [2017] Sophie Denève, Alireza Alemi, and Ralph Bourdoukan.  The brain as an efficient and robust adaptive learner.  _Neuron_ , 94(5):969–977, 2017.
  * Alemi et al. [2018] Alireza Alemi, Christian Machens, Sophie Deneve, and Jean-Jacques Slotine.  Learning nonlinear dynamics in efficient, balanced spiking networks using local plasticity rules.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Rusakov et al. [2020] Dmitri A. Rusakov, Leonid P. Savtchenko, and Peter E. Latham.  Noisy synaptic conductance: Bug or a feature?  _Trends in Neurosciences_ , 43(6):363–372, 2020\.
  * Hopfield [1984] J. J. Hopfield.  Neurons with graded response have collective computational properties like those of two-state neurons.  _Proceedings of the National Academy of Sciences_ , 81(10):3088–3092, 1984.
  * Cohen and Grossberg [1983] Michael A. Cohen and Stephen Grossberg.  Absolute stability of global pattern formation and parallel memory storage by competitive neural networks.  _IEEE Transactions on Systems, Man, and Cybernetics_ , SMC-13(5):815–826, 1983.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015\.

##  Supplementary Materials

Alexander Meulemans*, Nicolas Zucchet*, Seijin Kobayashi*, Johannes von
Oswald,
João Sacramento

###  Appendix S1 Notation

In this section, we provide an extensive list of the notations we use. In
particular, we detail what the different quantities precisely mean for
learning recurrent neural networks and for meta-learning.

####  S1.1 General notations

Notation |  Meaning
---|---
$\frac{\partial}{\partial x}=\partial_{x}$ |  Partial derivative of a function. The derivative of a scalar function is a row vector of size $|x|$.
$\frac{\mathrm{d}}{\mathrm{d}x}=\mathrm{d}_{x}$ |  Total derivative of a function. The derivative of a scalar function is a row vector of size $|x|$.
$\nabla_{x}$ |  Gradient of a scalar function (column vector). We have $\nabla_{x}\,\cdot=(\mathrm{d}_{x}\,\cdot)^{\top}\\!$.
$\dot{x}$ |  Time derivative of $x$ (column vector).
$\lVert\,\cdot\,\rVert$ |  Euclidean (L2) norm.
$\min_{x}$ |  Minimum of a function with respect to $x$.
$\operatorname*{arg\,min}_{x}$ |  Set of $x$ that (locally) minimize a function.
$\mathrm{Id}$ |  Identity matrix (size defined by the context).
$\mathrm{col}[A]$ |  Column space of $A$, that is $\mathrm{Im}[A]$.
$\mathrm{row}[A]$ |  Row space of $A$, that is $\mathrm{Im}[A^{\top}]$.

####  S1.2 Notations for the least-control principle in general

Notation |  Meaning
---|---
$\phi$ |  Dynamical parameters of the system.
$\theta$ |  Parameters of the system.
$f(\phi,\theta)$ |  Vector-field of the (free) dynamics on $\phi$.
$L$ |  Learning loss that the system should minimize.
$y=h(\phi)$ |  Output units (i.e. units on which the loss is evaluated) of the system.
$\mathcal{H}$ |  Least-control objective.
$\psi$ |  Controller whose goal is to help the system reaching a loss-minimizing state.
$u$ |  Controller on the output units.
$Q(\phi,\theta)$ |  General feedback mapping from output control $u$ to the entire system control $\psi$. We have $\psi=Q(\phi,\theta)u$ at equilibrium.
$\phi^{*}$ |  Equilibrium of the free dynamics.
$\phi_{\scalebox{0.42}{$\bigstar$}}$, $\psi_{\scalebox{0.42}{$\bigstar$}}$, $u_{\scalebox{0.42}{$\bigstar$}}$ |  Optimal state, control and output control according for the least-control objective (c.f. Eq. 3).
$\phi_{*}$, $\psi_{*}$, $u_{*}$ |  Equilibrium of the controlled dynamics. The least-control principle aims for $\psi_{*}=\psi_{\scalebox{0.42}{$\bigstar$}}$, that is to find an optimal control through the steady state of controlled dynamics.
$\alpha$ |  Leakage strength in a leaky integral controller ($\alpha=0$ yields pure integral control).
$\beta$ |  Strength of the proportional control, or of nudging in the energy-based view.

####  S1.3 Notations for recurrent neural networks

Notation |  Meaning
---|---
$\phi$ |  Neural activities of all the neurons in the network (column vector).
$\theta=\\{W,S\\}$ |  Learnable synaptic connection weights of the network.
$f(\phi,\theta)$ |  Dynamics of the neural network.
$y=D\phi$ |  The binary matrix $D$ selects output neurons $u$ from the rest of them $\phi$.
$\sigma$ |  Non-linear activation function.
$x$ |  Input of the neural network.
$y^{\mathrm{true}}$ |  Desired output corresponding to an input $x$.
$L$ |  Loss measuring the discrepancy between outputs of the network $y$ (that depends on $x$) and target output $y^{\mathrm{true}}$, averaged over all the pairs $(x,y^{\mathrm{true}})$ from a data set.
$\gamma$ |  Weight decay for the Kollen-Pollack learning-rule.

####  S1.4 Notations for meta-learning

Notation |  Meaning
---|---
$\phi$ |  Parameters (weights and biases) of the neural network that is learned.
$\theta=\\{\omega,\lambda\\}$ |  Parameters of the learning algorithm, here the consolidated synaptic weights $\omega$, and eventually, the strength of attraction to those weights $\lambda$.
$f(\phi,\theta)$ |  Dynamics of the learning algorithm.
$\tau$ |  Index of the task considered.
$L^{\mathrm{learn}}_{\tau}(\phi)$ |  Data-driven loss that is combined with a prior-term to obtain the loss minimized by the learning algorithm.
$L^{\mathrm{eval}}_{\tau}(\phi)$ |  Loss measuring the performance of the output of the learning algorithm.
$\psi=u$ |  Controller on all the units (they are all output units in this context).

###  Appendix S2 Proofs for the least-control principle

The purpose of this section is to prove all the theoretical results behind the
least-control principle. Recall that our principle advocates to solve the
following constrained optimization problem

$\min_{\theta}\min_{\phi,\psi}\,\frac{1}{2}\lVert\psi\rVert^{2}\quad\mathrm{s.t.}\enspace
f(\phi,\theta)+\psi=0,\enspace\frac{\partial L}{\partial y}(h(\phi))=0,$ (19)

that is minimizing the amount of control needed to reach a loss-minimizing
state. Finding an optimal control $\psi_{\scalebox{0.42}{$\bigstar$}}$
consists in solving the least-control problem:

$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})=\operatorname*{arg\,min}_{\phi,\psi}\frac{1}{2}\lVert\psi\rVert^{2}\quad\mathrm{s.t.}\enspace
f(\phi,\theta)+\psi=0,\enspace\frac{\partial L}{\partial y}(h(\phi))=0.$ (20)

Most of the proofs that we present here rely on interpreting the problem above
as a constrained minimization problem, and using the associated first-order
stationarity condition to manipulate $\psi_{\scalebox{0.42}{$\bigstar$}}$ and
$\phi_{\scalebox{0.42}{$\bigstar$}}$. Compared to the main text, we introduced
the notation $y=h(\phi)$ in this formulation. It is here to select output
units from the rest and clarifies mathematical manipulations.

####  S2.1 Constrained optimization and Lagrange multipliers

Throughout the proofs for the least-control principle, we characterize the
optimal control $\psi_{\scalebox{0.42}{$\bigstar$}}$ and state
$\phi_{\scalebox{0.42}{$\bigstar$}}$ through the stationarity conditions that
they verify, that are known as the KKT conditions. In particular, this allows
considering
$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$ as
an implicit function of $\theta$, and computing derivatives through the
implicit function theorem dontchev_implicit_2009. We provide a quick review on
constrained optimization in Section S2.7 for the reader who is new to the
topic.

We now state what are the KKT conditions for the constrained optimization
problem (20). To do so, we introduce the corresponding Lagrangian, which we
denote as the LCP-Lagrangian:

$\mathcal{L}(\phi,\psi,\lambda,\mu,\theta):=\frac{1}{2}\lVert\psi\rVert^{2}+\lambda^{\top}\left(f(\phi,\theta)+\psi\right)+\frac{\partial
L}{\partial y}(h(\phi))\mu,$ (21)

where $\lambda$ and $\mu$ are the Lagrange multipliers associated to the
constraints $f(\phi,\theta)+\psi=0$ and $\nabla_{y}L(h(\phi))=0$. Then,
$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$
verifies the KKT conditions associated to Eq. 20 if there exists
$(\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}})$
such that

$\left\\{\begin{split}\frac{\partial\mathcal{L}}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)&={\lambda_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial
f}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial
y^{2}}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})=0\\\
\frac{\partial\mathcal{L}}{\partial\psi}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)&={\psi_{\scalebox{0.42}{$\bigstar$}}}^{\top}+{\lambda_{\scalebox{0.42}{$\bigstar$}}}^{\top}=0\\\
\frac{\partial\mathcal{L}}{\partial\lambda}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)&=f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}+{\psi_{\scalebox{0.42}{$\bigstar$}}}^{\top}=0\\\
\frac{\partial\mathcal{L}}{\partial\mu}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)&=\frac{\partial
L}{\partial y}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))=0.\end{split}\right.$
(22)

In the proofs that follow, we use equivalently that
$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$ is
optimal for Eq. 20 and that it verifies the KKT conditions of Eq. 22. We need
to invoke sufficient conditions, such as the positive definiteness of the
Hessian of the Lagrangian, to make it rigorous and to show that a state
satisfying the KKT conditions is a local minimizer; we discuss this matter in
more details in Section S2.7. However, we omit these considerations in the
proofs to keep the arguments as concise as possible.

####  S2.2 Theorem 1: first-order gradient

We first state the formal version of Theorem 1 and then prove it, by
differentiating through the KKT conditions (22). Note that we provide an
alternate proof that leverages the energy-based formulation of the principle
in Section S2.5.

######  Theorem S1 (Formal).

Let $(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$
an optimal control for the least-control problem (20) and
$(\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}})$
the Lagrange multipliers for which the KKT conditions are satisfied. Under the
assumption that the Hessian of the LCP-Lagrangian
$\partial^{2}_{\phi,\psi,\lambda,\mu}\mathcal{L}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)$
is invertible, the least-control principle yields the following gradient for
$\theta$:

$\displaystyle\left(\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)\right)^{\top}$
$\displaystyle=-\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}\psi_{\scalebox{0.42}{$\bigstar$}}=\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta).$
(23)

###### Proof.

The proof relies on two ingredients: rewriting $\mathcal{H}$ using the LCP-
Lagrangian and then differentiating through this Lagrangian.

We start by rewriting the least-control objective $\mathcal{H}(\theta)$ using
the LCP-Lagrangian. Recall that $\mathcal{H}$ is defined as

$\mathcal{H}(\theta)=\frac{1}{2}\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert^{2}.$
(24)

As $(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$
is an optimal control, there indeed exists Lagrange multipliers
$(\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}})$
such that
$\partial_{\phi,\psi,\lambda,\mu}\mathcal{L}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)=0$,
as assumed in the statement of the theorem. We use this equality condition to
implicitly define
$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}})$
as functions of $\theta$. The implicit function theorem along with the
assumption that the Hessian is invertible ensures that these functions are
well defined and differentiable. As all the constraints are satisfied for
$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$, we
have

$\begin{split}\mathcal{H}(\theta)&=\mathcal{L}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)\\\
&=\frac{1}{2}\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert^{2}+{\lambda_{\scalebox{0.42}{$\bigstar$}}}^{\top}\left(f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+\psi_{\scalebox{0.42}{$\bigstar$}}\right)+\frac{\partial
L}{\partial
y}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\mu_{\scalebox{0.42}{$\bigstar$}}\\\
&=\frac{1}{2}\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert^{2}\end{split}$
(25)

We can then calculate $\mathrm{d}_{\theta}\mathcal{H}(\theta)$:

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)&=\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{L}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)\\\
&=\frac{\partial\mathcal{L}}{\partial\theta}+\frac{\partial\mathcal{L}}{\partial\phi}\frac{\mathrm{d}\phi_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}+\frac{\partial\mathcal{L}}{\partial\psi}\frac{\mathrm{d}\psi_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}+\frac{\partial\mathcal{L}}{\partial\lambda}\frac{\mathrm{d}\lambda_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}+\frac{\partial\mathcal{L}}{\partial\mu}\frac{\mathrm{d}\mu_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}\\\
&=\frac{\partial\mathcal{L}}{\partial\theta}+0+0+0+0\\\
&={\lambda_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta),\end{split}$
(26)

where we used the chain rule for the second equation, and the stationarity
equations of the Lagrangian for the third equation. Through the stationarity
equation
$\partial_{\psi}\mathcal{L}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)=0$,
we have that
$\lambda_{\scalebox{0.42}{$\bigstar$}}=-\psi_{\scalebox{0.42}{$\bigstar$}}$.
The constraint on $\psi$ additionally implies that
$\psi_{\scalebox{0.42}{$\bigstar$}}=-f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$,
which finishes the proof. ∎

Extension to $\theta$-dependent losses. Throughout this work, we focus on loss
functions $L(\phi)$ that do not explicitly depend on $\theta$. The influence
of $\theta$ on the loss only appears through the equilibrium condition
$f(\phi,\theta)=0$. Still, we can generalize Theorem S1 further by also
considering loss functions that explicitly depend on $\theta$, on top of their
implicit dependence on $\theta$ through $\phi$. A simple example is when we
add a weight regularization term $\lVert\theta\rVert^{2}$ to the loss $L$ when
training neural networks, or in meta-learning when $\theta$ does not only
impact the learning algorithm of the system but also its prediction
zintgraf_fast_2019,lee_meta-learning_2019,zhao_meta-learning_2020.

The proof of Theorem S1 can be slightly adapted to hold for the more general
case, the only difference is that the equality
$\partial_{\theta}\mathcal{L}=\lambda_{*}^{\top}\partial_{\theta}f(\phi_{*},\theta)$
is no longer true. Instead, we have

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)&=\frac{\partial\mathcal{L}}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)\\\
&=-{\psi_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial\theta\partial
y}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta).\end{split}$ (27)

The second derivative
$\partial_{\theta}\partial_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}),\theta)$
captures mixed dependencies of the loss $L$ on $y$ and $\theta$. Note that
when the loss $L$ contains no terms mixing $h(\phi)$ and $\theta$, as in the
weight decay example we mentioned earlier, this second-order derivative is
zero. As for this weight decay example the resulting updates are the same as
for the loss without any weight decay, this indicates that some
$\theta$-dependent losses are not captured in the least-control principle we
introduced in the main text.

We now propose an extended formulation of the least-control principle to fully
incorporate $\theta$-dependent losses:

$\min_{\theta}\min_{\phi,\psi}\frac{1}{2}\lVert\psi\rVert^{2}+L(\phi,\theta)~{}~{}~{}\mathrm{s.t.}~{}~{}f(\phi,\theta)+\psi=0,~{}~{}\nabla_{\phi}L(\phi,\theta)=0.$
(28)

Intuitively, the least-control objective $\lVert\psi\rVert^{2}$ combined with
the constraint that the loss is minimized w.r.t. $\phi$ at the controlled
equilibrium captures all the ways $\theta$ influences the loss through $\phi$,
and the new term $L(\phi,\theta)$ takes care of the ways in which $\theta$
influences the loss without influencing $\phi$, while keeping the same
stationarity conditions for the LCP-Lagrangian (the extra term
$\nabla_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}),\theta)$ that now appears
in $\partial_{\phi}\mathcal{L}$ vanishes). Using the same proof strategy as in
Theorem 1, we obtain the following gradient:

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)&=\frac{\partial\mathcal{L}}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)\\\
&=-{\psi_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial\theta\partial
y}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+\frac{\partial
L}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta).\end{split}$
(29)

This generalized least-control principle now allows to fully consider terms in
the loss that depend on $\theta$, while keeping the nice properties of being
single-phased, having no need to invert any Jacobian, and using no
infinitesimal learning signals. However, now the learning rule is not always
first-order anymore due the mixed second-order derivative.

One special case of interest where the resulting learning rule is first-order
again is when the $\theta$-dependence of the loss is encapsulated in the
decoder $h(\phi,\theta)$, i.e. when we have the loss $L(h(\phi,\theta))$.
Applying Eq. 29 to this loss results in:

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)&=-{\psi_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial
y^{2}}(h(\phi_{\scalebox{0.42}{$\bigstar$}},\theta))\frac{\partial
h}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+\frac{\partial
L}{\partial y}(h(\phi_{\scalebox{0.42}{$\bigstar$}},\theta))\frac{\partial
h}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta).\\\
&=-{\psi_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial
y^{2}}(h(\phi_{\scalebox{0.42}{$\bigstar$}},\theta))\frac{\partial
h}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+0\end{split}$
(30)

where the last term vanishes as
$\partial_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}},\theta))=0$ is enforced by
the constraints of the least-control objective of Eq. 3. When we use the
inversion dynamics (8) or the energy-based dynamics (10), we can easily access
the Lagrange multiplier $\mu_{\scalebox{0.42}{$\bigstar$}}$. For this, let us
take a look at the equilibrium equation of the inversion dynamics (8), which
is equivalent to the equilibrium condition of the energy-based dynamics (10)
if $\partial_{\phi}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ is
invertible:

$0=\left(\frac{\partial
f(\phi_{*},\theta)}{\partial\phi}\right)^{\top}\psi_{*}+\frac{\partial
h(\phi_{*},\theta)^{\top}}{\partial\phi}u_{*}$ (31)

Comparing this equilibrium condition with the KKT conditions of Eq. 22, and
using the relation
$\lambda_{\scalebox{0.42}{$\bigstar$}}=-\psi_{\scalebox{0.42}{$\bigstar$}}$,
we have that

$\displaystyle\frac{\partial^{2}L(h(\phi_{*},\theta))}{\partial
y^{2}}\mu_{\scalebox{0.42}{$\bigstar$}}=-u_{*}$ (32)

when $\partial_{\phi}h(\phi_{*},\theta)$ is full rank and in the limit of
$\alpha\to 0$ such that
$(\phi_{*},\psi_{*})=(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$.
Taking everything together, we arrive at the following simple first-order
update rule for models with a $\theta$-dependent decoder $h$, in the limit of
$\alpha\to 0$:

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)&=-\psi_{*}^{\top}\frac{\partial
f}{\partial\theta}(\phi_{*},\theta)-u_{*}^{\top}\frac{\partial
h(\phi_{*},\theta)}{\partial\theta}\end{split}$ (33)

Note that this update only holds when the inversion dynamics (8) or energy-
based dynamics (10) are used. If other dynamics are used for finding the
optimal control $\psi_{\scalebox{0.42}{$\bigstar$}}$, one needs to investigate
case-by-case how to compute the Lagrange multiplier
$\mu_{\scalebox{0.42}{$\bigstar$}}$.

####  S2.3 Theorem 2: general controller dynamics

Theorem 2 shows that we can find optimal control
$\psi_{\scalebox{0.42}{$\bigstar$}}$ and state
$\phi_{\scalebox{0.42}{$\bigstar$}}$ for Eq. 20 through dynamics exhibiting
the following equilibrium equations

$0=f(\phi_{*}^{\alpha},\theta)+Q(\phi_{*}^{\alpha},\theta)u_{*}^{\alpha}~{}~{}\mathrm{and}~{}~{}0=-\nabla_{y}L(h(\phi_{*}^{\alpha}))-\alpha
u_{*}^{\alpha}$ (34)

in the limit $\alpha\rightarrow 0$, when $Q(\phi_{*}^{\alpha},\theta)$
satisfies the column space condition (35). We prove that by showing that the
limit satisfies the KKT conditions, and hence is optimal. We restate Theorem 2
below with the full technical specifications.

######  Theorem S2 (Formal).

Let $(\phi_{*}^{\alpha},u_{*}^{\alpha})$ be a steady state of the generalized
dynamics satisfying (34) such that $\phi_{*}^{\alpha}$ admits a finite limit
$\phi_{\scalebox{0.42}{$\bigstar$}}$ when $\alpha$ goes to 0. Assume that
$\partial_{\phi}{f}(\phi_{\scalebox{0.42}{$\bigstar$}})$ and
$\partial_{y}^{2}{L}(\phi_{\scalebox{0.42}{$\bigstar$}})$ are invertible,
$\partial_{\phi}h(\phi_{\scalebox{0.42}{$\bigstar$}})$ is full rank, $Q$ is
continuous, and that the following column space condition is satisfied:

$\displaystyle\mathrm{col}\left[Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\right]=\mathrm{row}\left[\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})\left(\frac{\partial
f}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\right)^{-1}\right]$
(35)

Then, $u_{*}^{\alpha}$ converges to a finite limit
$u_{\scalebox{0.42}{$\bigstar$}}$ and
$(\phi_{\scalebox{0.42}{$\bigstar$}},Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)u_{\scalebox{0.42}{$\bigstar$}})$
verifies the KKT conditions for the least-control problem (20).

###### Proof.

Note that there is a slight abuse of notation in the statement of the theorem
as we use the superscript ${}_{\scalebox{0.42}{$\bigstar$}}$ to denote the
limits of the different quantities without yet knowing if they correspond to
an optimal quantity for the least-control problem (20). Part of the proof is
to actually show that those limits are optimal.

First, we show that $u_{*}^{\alpha}$ has a finite limit when $\alpha$ goes to
0. The function $\phi^{\alpha}_{*}$ admits a finite limit
$\phi_{\scalebox{0.42}{$\bigstar$}}$ when $\alpha$ goes to zero by hypothesis.
Taking $\alpha\rightarrow 0$ in the equilibrium equations (34) results in

$\displaystyle
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)-\lim_{\alpha\rightarrow
0}\frac{1}{\alpha}Q(\phi_{*}^{\alpha},\theta)\nabla_{y}L(h(\phi_{*}^{\alpha}))=0,$
(36)

as $f$ is continuous. The matrices
$\partial_{\phi}h(\phi_{\scalebox{0.42}{$\bigstar$}})$ and
$\partial_{\phi}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{-1}$ are full
rank, so the column space condition implies that
$Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ is also full rank. We can
therefore extract $Q(\phi_{*}^{\alpha},\theta)$ out of the limit in the
previous equation:

$\lim_{\alpha\rightarrow
0}\frac{1}{\alpha}Q(\phi_{*}^{\alpha},\theta)\nabla_{y}L(h(\phi_{*}^{\alpha}))=Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\lim_{\alpha\rightarrow
0}\frac{1}{\alpha}\nabla_{y}L(h(\phi_{*}^{\alpha})),$ (37)

so that $u_{*}^{\alpha}=-\alpha^{-1}\nabla_{y}L(h(\phi_{*}^{\alpha}))$ has a
finite limit that we note $u_{\scalebox{0.42}{$\bigstar$}}$, using once again
the fact that $Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ is full rank and
(36). In particular, this implies that $\phi_{\scalebox{0.42}{$\bigstar$}}$ is
feasible as $\nabla_{y}L(h(\phi_{*}^{\alpha}))=O(\alpha)$ so
$\nabla_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))=0$.

We can now prove that the KKT conditions are satisfied at
$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}})$ with
$\psi_{\scalebox{0.42}{$\bigstar$}}:=Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)u_{\scalebox{0.42}{$\bigstar$}}$.
Note that we have
$f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+\psi_{\scalebox{0.42}{$\bigstar$}}=0$
by taking the limit $\alpha\rightarrow 0$ in the equilibrium equation

$f(\phi_{*}^{\alpha},\theta)-\frac{1}{\alpha}Q(\phi_{*}^{\alpha},\theta)\nabla_{y}L(h(\phi_{*}^{\alpha}))=0.$
(38)

As
$\mathrm{col}\left[Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\right]=\mathrm{row}\left[\partial_{\phi}h(\phi_{\scalebox{0.42}{$\bigstar$}})\partial_{\phi}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{-1}\right]$,
and $\partial^{2}_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))$ is invertible,
there exist a $\mu$ and $\mu_{\scalebox{0.42}{$\bigstar$}}$, such that

$\begin{split}\psi_{\scalebox{0.42}{$\bigstar$}}&=Q(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)u_{\scalebox{0.42}{$\bigstar$}}\\\
&=\frac{\partial
f}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{-\top}\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})^{\top}\mu\\\
&=\frac{\partial
f}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{-\top}\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})^{\top}\frac{\partial^{2}L}{\partial
y^{2}}(\phi_{\scalebox{0.42}{$\bigstar$}})\mu_{\scalebox{0.42}{$\bigstar$}}.\end{split}$
(39)

Multiplying from the left with
$\partial_{\phi}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}$ results
in

$-\frac{\partial
f}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}\psi_{\scalebox{0.42}{$\bigstar$}}+\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})^{\top}\frac{\partial^{2}L}{\partial
y^{2}}(\phi_{\scalebox{0.42}{$\bigstar$}})\mu_{\scalebox{0.42}{$\bigstar$}}=0.$
(40)

We now take
$\lambda_{\scalebox{0.42}{$\bigstar$}}:=-\psi_{\scalebox{0.42}{$\bigstar$}}$
and can easily check that
$(\phi_{\scalebox{0.42}{$\bigstar$}},\psi_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}})$
is a stationary point for the LCP-Lagrangian, i.e. it satisfies the KKT
conditions (20). It follows that $\psi_{\scalebox{0.42}{$\bigstar$}}$ is an
optimal control and $\phi_{\scalebox{0.42}{$\bigstar$}}$ an optimally-
controlled state. ∎

####  S2.4 Propositions 3, 4, 5: the least-control principle solves the
original learning problem

We here restate the results from Section 2.3 and prove them.

######  Proposition S3.

Assuming $L$ is convex on the system output $y$, we have that the optimal
control $\psi_{\scalebox{0.42}{$\bigstar$}}$ is equal to 0 iff. the free
equilibrium $\phi^{*}$ minimizes $L$.

###### Proof.

First remark that, as $L$ is convex, $\phi$ being a minimizer of $L(h(\phi))$
is equivalent to $\nabla_{y}L(h(\phi))=0$. Then, the two conditions are
equivalent to $\phi_{\scalebox{0.42}{$\bigstar$}}$ and $\phi^{*}$ satisfying
both $f(\phi,\theta)=0$ and $\nabla_{y}L(h(\phi))=0$. ∎

######  Proposition S4.

Assuming $L$ is convex on the system output $y$, a local minimizer $\theta$ of
the least-control objective $\mathcal{H}$ is a global minimizer of the
original learning problem (1), under the sufficient condition that
$\partial_{\theta}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ has full row
rank.

###### Proof.

As $\theta$ is a local minimizer of $\mathcal{H}(\theta)$, we have that

$\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)=f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)=0.$ (41)

The full row rank assumption of
$\partial_{\theta}{f}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ ensures that
$f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)=0$. We hence have that
$\phi_{\scalebox{0.42}{$\bigstar$}}$ is a global minimizer of $L(h(\phi))$, as
$\nabla_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))=0$, and $L$ is convex, and
satisfies $f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)=0$. It follows that
$\theta$ is a global minimizer for the original learning problem. ∎

######  Remark S1.

In the last two propositions, we assumed that the loss $L$ is convex on output
units. This is for example the case in our supervised learning experiments,
but not for meta-learning. If we relax this assumption, Propositions 3 and 4
still hold, with the difference that global minimizers become local ones.

Additionally, the result of those two propositions does not only hold in the
strong control limit but also in the $\alpha>0$ regime, under the strict
feedback condition of Proposition S9. The proof is a direct combination of the
proof of the two propositions above with the stationarity result of
Proposition S9.

######  Proposition S5.

If $\frac{1}{2}\lVert f\rVert^{2}$ is $\mu$-strongly convex as a function of
$\phi$, $L\circ h$ is $M$-Lipschitz continuous and the minimum of $L$ is 0,
then

$L(h(\phi^{*}))\leq\frac{\sqrt{\mu}}{\sqrt{2}M}\lVert\psi_{\scalebox{0.42}{$\bigstar$}}\rVert\leq\frac{\sqrt{\mu}}{\sqrt{2}M}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert.$ (42)

###### Proof.

As $h(\phi_{*})$ minimizes $L$ and $L$ is Lipschitz-continuous,

$L(h(\phi^{*}))=L(h(\phi^{*}))-L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\leq
M\lVert\phi^{*}-\phi_{\scalebox{0.42}{$\bigstar$}}\rVert.$ (43)

The strong convexity of $\frac{1}{2}\lVert f\rVert^{2}$ yields

$\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert=\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}-\frac{1}{2}\lVert
f(\phi^{*},\theta)\rVert^{2}\geq\frac{\mu}{2}\lVert\phi_{\scalebox{0.42}{$\bigstar$}}-\phi^{*}\rVert^{2},$
(44)

as $\phi^{*}$ is a global minimizer of $\frac{1}{2}\lVert f\rVert^{2}$.
Gathering the last two inequalities gives the desired result. ∎

We state and prove the remaining informal statements on the impact of
approximate equilibria appearing in Section 2.3 in Section S2.6, as they
require the energy-based view of the least control principle to be proved.

####  S2.5 The least-control principle as constrained energy minimization

In Section 2.4, we show that substituting the constraint
$f(\phi,\theta)+\psi=0$ back into the least-control problem implies that
$\phi_{\scalebox{0.42}{$\bigstar$}}$ is a minimizer of

$\min_{\phi}\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}\quad\mathrm{s.t.}\enspace\nabla_{y}L(h(\phi))=0.$
(45)

The same objective can be obtained from an energy-based perspective, by
minimizing the augmented energy $F(\phi,\theta,\beta)=\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}+\beta L(h(\phi))$ and taking the limit
$\beta\rightarrow+\infty$. We formalize this result by first showing that the
limit of stationary points of $F$ when $\beta\rightarrow\infty$ satisfies the
KKT conditions (Proposition S6), and that the global minimizers coincide,
under more restrictive assumption (Proposition S7). Note that the KKT
conditions associated to (45) are different from the ones in Eq. 22: the
corresponding Lagrangian is

$\mathcal{L}(\phi,\mu,\theta):=\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}+\mu^{\top}\left(\nabla_{y}L(h(\phi))\right)$ (46)

and KKT conditions are

$\left\\{\begin{split}&\frac{\partial\mathcal{L}}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)=\frac{\partial}{\partial\phi}\left[\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}\right]+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial
y^{2}}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})=0\\\
&\frac{\partial\mathcal{L}}{\partial\mu}(\phi_{\scalebox{0.42}{$\bigstar$}},\mu_{\scalebox{0.42}{$\bigstar$}},\theta)=\nabla_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))=0\\\
\end{split}\right..$ (47)

######  Proposition S6.

Let $\phi_{*}^{\beta}$ be a function of $\beta$ that admits a finite limit
$\phi_{\scalebox{0.42}{$\bigstar$}}$ when $\beta$ goes to infinity and
verifies

$\frac{\partial F}{\partial\phi}(\phi_{*}^{\beta},\beta,\theta)=0$ (48)

for every $\beta$. If we further assume that the loss Hessian
$\partial^{2}_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))$ is invertible and
$\partial_{\phi}h(\phi_{\scalebox{0.42}{$\bigstar$}})$ is full rank, then
$\phi_{\scalebox{0.42}{$\bigstar$}}$ satisfies the KKT conditions associated
to Eq. 45.

###### Proof.

We use the shorthand $E:=\frac{1}{2}\lVert f\rVert^{2}$ for conciseness. For
every $\beta$, $\phi_{*}^{\beta}$ verifies

$\frac{\partial F}{\partial\phi}(\phi_{*}^{\beta},\theta,\beta)=\frac{\partial
E}{\partial\phi}(\phi_{*}^{\beta},\theta)+\beta\frac{\partial L}{\partial
y}(h(\phi_{*}^{\beta}))\frac{\partial h}{\partial\phi}(\phi_{*}^{\beta})=0$
(49)

By taking $\beta$ to infinity in the equation above and using continuity of
$\partial_{\phi}E$, we obtain

$\frac{\partial
E}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})=-\lim_{\beta\rightarrow\infty}\beta\frac{\partial
L}{\partial y}(h(\phi_{*}^{\beta}))\frac{\partial
h}{\partial\phi}(\phi_{*}^{\beta}).$ (50)

This first implies that

$\frac{\partial L}{\partial
y}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})=\lim_{\beta\rightarrow\infty}\frac{\partial
L}{\partial y}(h(\phi_{*}^{\beta}))\frac{\partial
h}{\partial\phi}(\phi_{*}^{\beta})=0$ (51)

as $\partial_{\phi}h$, $\partial_{y}L$ and $h$ are continuous, and then that
$\nabla_{y}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))=0$ as
$\partial_{\phi}h(\phi_{\scalebox{0.42}{$\bigstar$}})$ is full rank.

Eq. 50, along with $\partial_{\phi}h(\phi_{\scalebox{0.42}{$\bigstar$}})$
being full rank, implies that $\beta\partial_{y}L(h(\phi_{*}^{\beta}))$ admits
a finite limit, note it $\mu$. As
$\partial_{y}^{2}L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))$ is invertible,
there exists $\mu_{\scalebox{0.42}{$\bigstar$}}$ such that
$\partial_{y}^{2}L(\phi_{\scalebox{0.42}{$\bigstar$}})\mu_{\scalebox{0.42}{$\bigstar$}}=\mu$.
Put back into (50) it yields

$\frac{\partial
E}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial
y^{2}}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})=0,$ (52)

hence $\phi_{\scalebox{0.42}{$\bigstar$}}$ satisfies the KKT conditions
associated to (45). ∎

######  Proposition S7.

Assume $L$ to positive and equal to 0 if and only if $\nabla_{y}L=0$. Let
$\phi_{*}^{\beta}$ be a global minimizer of the augmented energy $F$ for every
$\beta$, such that $\phi_{*}^{\beta}$ admits a finite limit
$\phi_{\scalebox{0.42}{$\bigstar$}}$ when $\beta$ goes to infinity. Then the
limit $\phi_{\scalebox{0.42}{$\bigstar$}}$ is a global minimizer of the
constrained optimization problem (45).

###### Proof.

First, remark that the condition on the minimizers of $L$ implies that solving

$\min_{\phi}\,\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}\quad\mathrm{s.t.}\enspace\nabla_{y}L(h(\phi))=0$ (53)

is equivalent to solving

$\min_{\phi}\,\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}\quad\mathrm{s.t.}\enspace L(h(\phi))=0.$ (54)

We use the second characterization in the rest of the proof.

We introduce the short hand

$\begin{split}f_{\scalebox{0.42}{$\bigstar$}}&:=\min_{\phi}\,\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}\quad\mathrm{s.t.}\enspace L(h(\phi))=0.\end{split}$
(55)

For any $\phi$ we have that

$F(\phi_{*}^{\beta},\theta,\beta)\leq F(\phi,\theta,\beta)$ (56)

by definition of $\phi_{*}^{\beta}$, in particular for all the $\phi$
verifying the constraint $L(h(\phi))=0$. It follows that

$F(\phi_{*}^{\beta},\theta,\beta)\leq\min_{L(h(\phi))=0}F(\phi,\theta,\beta)=\min_{L(h(\phi))=0}\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}=f_{\scalebox{0.42}{$\bigstar$}}.$ (57)

The last inequality can be rewritten as

$L(h(\phi_{*}^{\beta}))\leq\frac{f_{\scalebox{0.42}{$\bigstar$}}-\lVert
f(\phi_{*}^{\beta},\theta)\rVert^{2}/2}{\beta}.$

The upper bound converges to 0 as the numerator converges to a finite value by
continuity of $\lVert f\rVert^{2}$. Since $L$ is positive and continuous and
$h$ is continuous we obtain $L(h(\phi_{\scalebox{0.42}{$\bigstar$}}))=0$,
i.e., $\phi_{\scalebox{0.42}{$\bigstar$}}$ is feasible.

We now show that $\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}/2=f_{\scalebox{0.42}{$\bigstar$}}$.
As $\phi_{\scalebox{0.42}{$\bigstar$}}$ is feasible we have

$f_{\scalebox{0.42}{$\bigstar$}}=\min_{L(h(\phi))=0}\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}\leq\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}.$ (58)

This implies that

$\limsup_{\beta\rightarrow\infty}F(\phi_{*}^{\beta},\theta,\beta)=\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}+\limsup_{\beta\rightarrow\infty}\beta
L(h(\phi_{*}^{\beta}))\geq f_{\scalebox{0.42}{$\bigstar$}}$ (59)

where we used the positivity of $L$ for the last inequality. Because of (57),
we also have

$\limsup_{\beta\rightarrow\infty}F(\phi_{*}^{\beta},\beta)\leq
f_{\scalebox{0.42}{$\bigstar$}}$ (60)

so that the combination of the last two equations yield

$\limsup_{\beta\rightarrow\infty}\beta L(h(\phi_{*}^{\beta}))=0$ (61)

and

$\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}=f_{\scalebox{0.42}{$\bigstar$}}.$
(62)

This finishes the proof. ∎

######  Remark S2.

Under the same assumptions, we can actually show that, for any $\beta$,
$\beta^{\prime}\in\mathbb{R}_{+}$, we have

$0\leq F(\phi_{*}^{\beta},\theta,\beta)\leq
F(\phi_{*}^{\beta^{\prime}},\theta,\beta^{\prime})\leq\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}.$ (63)

This can be obtained by combining Proposition S7 and the fact that
$\beta\mapsto F(\phi_{*}^{\beta},\theta,\beta)$ is a non decreasing function
as

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\beta}F(\phi_{*}^{\beta},\theta,\beta)&=\frac{\partial
F}{\partial\beta}(\phi_{*}^{\beta},\theta,\beta)+\frac{\partial
F}{\partial\phi}(\phi_{*}^{\beta},\theta,\beta)\frac{\mathrm{d}\phi_{*}^{\beta}}{\mathrm{d}\beta}\\\
&=L(\phi_{*}^{\beta})+0\\\ &\geq 0.\end{split}$ (64)

The function $\beta\mapsto F(\phi_{*}^{\beta},\theta,\beta)$ therefore
converges to the least-control objective $\mathcal{H}(\theta)$ from below.

In the energy-based view of the least-control principle, the least-control
objective becomes

$\mathcal{H}(\theta)=\frac{1}{2}\lVert
f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\rVert^{2}.$ (65)

We use this formulation to provide an alternative proof to Theorem S1 that
might be insightful for the reader.

###### Alternative proof of Theorem S1.

Theorem S1 states that

$\left(\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)\right)^{\top}=\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta).$
(66)

We use the shorthand notation $E:=\frac{1}{2}\lVert f(\phi,\theta)\rVert^{2}$.
The chain rule yields

$\frac{\mathrm{d}}{\mathrm{d}\theta}E(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)=\frac{\mathrm{d}}{\mathrm{d}\theta}E(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)=\frac{\partial
E}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+\frac{\partial
E}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\frac{\mathrm{d}\phi_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}.$
(67)

As $\nabla_{\phi}L$ is independent of $\theta$, we have that
$\mathrm{d}_{\theta}h(\phi_{\scalebox{0.42}{$\bigstar$}})=0$. This rewrites
$\partial_{\phi}h(\phi_{\scalebox{0.42}{$\bigstar$}})\mathrm{d}_{\theta}\phi_{\scalebox{0.42}{$\bigstar$}}=0$.
This can be combined to the KKT conditions to show that the indirect term is
equal to $0$: right multiplying the condition

$0=\frac{\partial
E}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial
y}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})$ (68)

by $\mathrm{d}_{\theta}\phi_{\scalebox{0.42}{$\bigstar$}}$ gives

$\begin{split}0&=\frac{\partial
E}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\frac{\mathrm{d}\phi_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}+{\mu_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial^{2}L}{\partial
y}(h(\phi_{\scalebox{0.42}{$\bigstar$}}))\frac{\partial
h}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}})\frac{\mathrm{d}\phi_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}\\\
&=\frac{\partial
E}{\partial\phi}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\frac{\mathrm{d}\phi_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}\theta}.\end{split}$
(69)

This finishes the proof. ∎

####  S2.6 The parameter update is robust to approximate optimal control

The purpose of this section is to formally prove that the first-order update
prescribed by Theorem 1 is robust to inaccurate approximations of the
optimally-controlled state $\phi_{\scalebox{0.42}{$\bigstar$}}$. More
precisely, we show that the quality of the gradient can be linked to the
distance between the approximate and the true optimally-controlled state
(Proposition S8) and that the update is minimizing a closely related objective
when the perfect control limit is not exactly reached (Proposition S9).

######  Proposition S8.

Let $\phi_{*}$ be an optimally-controlled state and $\hat{\phi}$ an
approximation of it that is used to update the parameters $\theta$. Then, if
the conditions of Theorem S1 apply and
$\partial_{\theta}f(\phi,\theta)^{\top}f(\phi,\theta)$ is $M$-Lipschitz
continuous we have that

$\left\lVert f(\hat{\phi},\theta)^{\top}\frac{\partial
f}{\partial\theta}(\hat{\phi},\theta)-\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)\right\rVert\leq
M\lVert\hat{\phi}-\phi_{*}\rVert.$ (70)

###### Proof.

We use Theorem S1 and then the Lipschitz continuity assumption:

$\begin{split}\left\lVert f(\hat{\phi},\theta)^{\top}\frac{\partial
f}{\partial\theta}(\hat{\phi},\theta)-\frac{\mathrm{d}}{\mathrm{d}\theta}\mathcal{H}(\theta)\right\rVert&=\left\lVert
f(\hat{\phi},\theta)^{\top}\frac{\partial
f}{\partial\theta}(\hat{\phi},\theta)-f(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)^{\top}\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)\right\rVert\\\
&\leq M\lVert\hat{\phi}-\phi_{\scalebox{0.42}{$\bigstar$}}\rVert.\end{split}$
(71)

∎

We empirically verify that the behavior predicted by Proposition S8 holds in
practice. To do so, we compare the estimated gradient
$f(\hat{\phi},\theta)^{\top}\partial_{\theta}f(\hat{\phi},\theta)$ for several
approximations $\hat{\phi}$ of the optimally-controlled state $\phi_{*}$ to
the true least-control gradient $\mathrm{d}_{\theta}\mathcal{H}(\theta)$ on a
feedforward neural network learning problem. Figure S1 shows the gradient
estimation error as a function of the steady state approximation
$\lVert\hat{\phi}-\phi_{*}\rVert$. The evolution is almost linear, suggesting
that the inequality in Eq. 70 is close to being an equality for some constant
$M$.

Figure S1: Empirical verification of Proposition S8 on a randomly initalized
feedforward neural network with 3 hidden layers of size 300. The different
gradients are estimated using 100 data samples randomly drawn from the MNIST
dataset. (A) We run energy-based dynamics for $\alpha=0$ with a time constant
of 200 for 200 steps, compute the gradient estimation error at each step, and
plot it as a function of the steady state approximation error
$\lVert\hat{\phi}-\phi_{*}\rVert$. The optimally-controlled $\phi_{*}$ is
computed by running the same dynamics until convergence. (B) After computing
$\phi_{*}$, we perturb it with random noise drawn from
$\mathcal{U}([0,1])\times\mathcal{N}(0,0.5)$, and plot the same relationship
as in panel A.

######  Proposition S9.

Let $(\phi_{*},u_{*})$ verifying the steady-state equations

$f(\phi_{*},\theta)+Q(\phi_{*},\theta)u_{*}=0,\quad\mathrm{and}\quad-\alpha
u_{*}-\nabla_{y}L(h(\phi_{*}))=0.$ (72)

If

$Q(\phi_{*},\beta)=\partial_{\phi}f(\phi_{*},\theta)^{-\top}\partial_{\phi}h(\phi_{*})^{\top},$
(73)

then $\phi_{*}$ is a stationary point of the augmented energy
$F(\phi,\theta,\alpha)=\frac{1}{2}\lVert
f(\phi,\theta)\rVert^{2}+\alpha^{-1}L(h(\phi))$.

###### Proof.

We first use the formula for $Q$ and use the steady-state equation on $u_{*}$:

$\begin{split}0&=f(\phi_{*},\theta)+Q(\phi_{*},\theta)u_{*}\\\
&=f(\phi_{*},\theta)-\partial_{\phi}f(\phi_{*},\theta)^{-\top}\partial_{\phi}h(\phi_{*})^{\top}u_{*}\\\
&=f(\phi_{*},\theta)+\alpha^{-1}\partial_{\phi}f(\phi_{*},\theta)^{-\top}\partial_{\phi}h(\phi_{*})^{\top}\nabla_{y}L(h(\phi_{*}))\end{split}$
(74)

Left-multiplying by $\partial_{\phi}f(\phi_{*},\theta)$ yields

$\partial_{\phi}f(\phi_{*},\theta)^{\top}f(\phi_{*},\theta)+\alpha^{-1}\partial_{\phi}h(\phi_{*})^{\top}\nabla_{y}L(h(\phi_{*}))=0,$
(75)

which is exactly $\nabla_{\phi}F(\phi,\theta,\alpha)=0$. ∎

Proposition S9 allows understanding what the update
$\partial_{\theta}f(\phi_{*},\theta)^{\top}f(\phi_{*},\theta)$ is doing when
the controlled dynamics reach an equilibrium, but $\alpha$ is non zero. In
this case, we cannot guarantee that $Q(\phi_{*},\theta)u_{*}$ is an optimal
control. Still, when the strict condition on $Q$ of Eq. 73 is satisfied (it is
strict compared to the more general condition (35) of Theorem S2), the update
prescribed by the least-control principle is minimizing
$F(\phi_{*},\theta,\alpha)$ as

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\theta}F(\phi_{*},\theta,\alpha)&=\frac{\partial
F}{\partial\theta}(\phi_{*},\theta,\alpha)+\frac{\partial
F}{\partial\phi}(\phi_{*},\theta,\alpha)\frac{\mathrm{d}\phi_{*}}{\mathrm{d}\theta}\\\
&=\frac{\partial F}{\partial\theta}(\phi_{*},\theta,\alpha)+0\\\
&=\frac{\partial
f}{\partial\theta}(\phi_{*},\theta)^{\top}f(\phi_{*},\theta).\end{split}$ (76)

We showed in Section S2.5 that this objective is closely related to the least-
control objective, and gets closer when $\alpha\rightarrow 0_{+}$. Note that
the same conclusions apply if we use a proportional controller with strength
$\beta$, as it verifies the steady-state equation (72) for
$\alpha=\beta^{-1}$.

####  S2.7 Quick review of constrained optimization

We heavily rely on the Karush–Kuhn–Tucker (KKT) conditions and Lagrange
multiplier method for characterizing constrained optimum. We state here some
important properties that are throughout the proofs, and we refer the
interested reader to nocedal_numerical_2006, bertsekas_constrained_2014 for a
comprehensive overview of the relevant theory.

KKT conditions. Consider the following equality-constrained optimization
problem

$\displaystyle\min_{x}f(x)~{}~{}\mathrm{s.t.}~{}g(x)=0,$ (77)

with $f$ some scalar function to be optimized and $g$ a vector-valued function
defining equality constraints on the set of feasible states. The corresponding
Lagrangian is defined as

$\displaystyle\mathcal{L}(x,\lambda):=f(x)+\lambda^{\top}g(x),$ (78)

with $\lambda$ the so-called Lagrange multipliers. It can be shown, e.g.
[Theorem 12.1]nocedal_numerical_2006, that local solutions
$x_{\scalebox{0.42}{$\bigstar$}}$ of the constrained optimization problem
defined above satisfy the KKT conditions: there exists a Lagrange multiplier
$\lambda_{\scalebox{0.42}{$\bigstar$}}$ such that

$\left\\{\begin{split}0&=\frac{\partial f}{\partial
x}(x_{\scalebox{0.42}{$\bigstar$}})+\frac{\partial g}{\partial
x}(x_{\scalebox{0.42}{$\bigstar$}})\lambda_{\scalebox{0.42}{$\bigstar$}}\\\
0&=g(x)\end{split}\right.$ (79)

or equivalently such that
$(x_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}})$ is a
stationary point of the Lagrangian, i.e.
$\partial_{x_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}}}\mathcal{L}(x_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}})=0$.

The KKT conditions are first-order stationary conditions for constrained
optimization, in the same way $\partial_{x}f=0$ is a stationary condition for
unconstrained minimization. On the other side, there exists sufficient
conditions under which those stationary conditions give a local, constrained
or unconstrained, minimum. For unconstrained optimization, it can be that the
Hessian $\nabla^{2}_{x}f$ is positive definite. For constrained optimization
there are several variants, but those conditions can more or less be
understood as the Hessian
$\partial^{2}_{x}\mathcal{L}(x_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}})$
of the Lagrangian w.r.t. $x$ is positive definite (see e.g. [Theorem
12.5]nocedal_numerical_2006).

Differentiating through minima. We now consider that both the objective $f$
and the constraints $g$ are dependent on some parameter $y$, and aim to
calculate how $f(x_{\scalebox{0.42}{$\bigstar$}}(y),y)$ will react to a change
in $y$, that is we aim to compute the gradient associated to

$\displaystyle\min_{y}f(x_{\scalebox{0.42}{$\bigstar$}}(y),y)=\min_{y}\min_{x}f(x,y)~{}~{}\mathrm{s.t.}~{}g(x,y)=0.$
(80)

For unconstrained bilevel optimization (i.e. without the equality constraints
$g(x,y)=0$), we differentiate through the inner minima
$x_{\scalebox{0.42}{$\bigstar$}}(y)$ using the necessary stationary condition
$\partial_{x}f(x_{\scalebox{0.42}{$\bigstar$}}(y),y)=0$, combined with the
implicit function theorem dontchev_implicit_2009 to ensure that the implicit
function $x_{\scalebox{0.42}{$\bigstar$}}(y)$ is well defined. This gives

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}y}f(x_{\scalebox{0.42}{$\bigstar$}}(y),y)&=\frac{\partial
f}{\partial y}(x_{\scalebox{0.42}{$\bigstar$}}(y),y)+\frac{\partial
f}{\partial
x}(x_{\scalebox{0.42}{$\bigstar$}}(y),y)\frac{\mathrm{d}x_{\scalebox{0.42}{$\bigstar$}}(y)}{\mathrm{d}y}\\\
&=\frac{\partial f}{\partial
y}(x_{\scalebox{0.42}{$\bigstar$}}(y),y)+0.\end{split}$ (81)

Note that $\mathrm{d}_{y}x_{\scalebox{0.42}{$\bigstar$}}(y)$ can be obtained
via the implicit function theorem by using the first-order condition
$\partial_{x}f(x_{\scalebox{0.42}{$\bigstar$}}(y),y)=0$ but we do not need to
compute it.

A similar technique can be applied to differentiate through a constrained
minimum. We first remark that $x_{\scalebox{0.42}{$\bigstar$}}$ satisfies the
KKT conditions so there exists a Lagrange multiplier
$\lambda_{\scalebox{0.42}{$\bigstar$}}(y)$ such that

$f(x_{\scalebox{0.42}{$\bigstar$}}(y),y)=\mathcal{L}(x_{\scalebox{0.42}{$\bigstar$}}(y),\lambda_{\scalebox{0.42}{$\bigstar$}}(y),y).$
(82)

We then use the implicit function theorem on the KKT condition
$\partial_{x,\lambda}\mathcal{L}(x_{\scalebox{0.42}{$\bigstar$}},\lambda_{\scalebox{0.42}{$\bigstar$}},y)=0$
to show that the functions $x_{\scalebox{0.42}{$\bigstar$}}(y)$ and
$\lambda_{\scalebox{0.42}{$\bigstar$}}(y)$ are well defined and
differentiable, under the assumption that the Hessian of the Lagrangian w.r.t.
$x$ and $\lambda$ is invertible. Finally, we have:

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}y}f(x_{\scalebox{0.42}{$\bigstar$}}(y),y)&=\frac{\mathrm{d}}{\mathrm{d}y}\mathcal{L}(x_{\scalebox{0.42}{$\bigstar$}}(y),\lambda_{\scalebox{0.42}{$\bigstar$}}(y),y)\\\
&=\frac{\partial\mathcal{L}}{\partial y}+\frac{\partial\mathcal{L}}{\partial
x}\frac{\mathrm{d}x_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}y}+\frac{\partial\mathcal{L}}{\partial\lambda}\frac{\mathrm{d}\lambda_{\scalebox{0.42}{$\bigstar$}}}{\mathrm{d}y}\\\
&=\frac{\partial\mathcal{L}}{\partial y}+0+0\\\ \end{split}$ (83)

where the last inequality holds as $x_{\scalebox{0.42}{$\bigstar$}}(y)$ and
$\lambda_{\scalebox{0.42}{$\bigstar$}}(y)$ are stationary points of the
Lagrangian $\mathcal{L}$.

###  Appendix S3 Implicit gradient, recurrent backpropagation and the link to
the least-control principle

The least-control principle aims at solving the learning problem

$\min_{\theta}L(h(\phi^{*}))\quad\mathrm{s.t.}\enspace f(\phi^{*},\theta)=0$
(84)

indirectly, by minimizing the least-control objective. This leads to first-
order updates, contrary to the direct minimization of the loss $L(\phi^{*})$.
In this section, we derive the implicit gradient associated to direct loss
minimization, show how the recurrent backpropagation computes it, and finally
highlight the similarities behind the least control principle and recurrent
backpropagation.

Implicit gradient. The implicit gradient $\mathrm{d}_{\theta}L(\phi^{*})$ can
be calculated analytically using the implicit function theorem
dontchev_implicit_2009, we here calculate it. The quantity $\phi^{*}$ can be
characterized as an implicit function of $\theta$, which verifies

$f(\phi^{*},\theta)=0$ (85)

for all $\theta$. The implicit function theorem guarantees that $\phi^{*}$ is
properly defined as an implicit function of $\theta$ if
$\partial_{\phi}f(\phi^{*},\theta)$ is invertible, and then

$\frac{\mathrm{d}\phi^{*}}{\mathrm{d}\theta}=-\left(\frac{\partial
f}{\partial\phi}(\phi^{*},\theta)\right)^{-1}\frac{\partial
f}{\partial\theta}(\phi^{*},\theta).$

We can then use the chain rule to obtain the implicit gradient:

$\begin{split}\frac{\mathrm{d}}{\mathrm{d}\theta}L(h(\phi^{*}))&=\frac{\partial
L}{\partial y}(h(\phi^{*}))\frac{\partial
h}{\partial\phi}(\phi^{*})\frac{\mathrm{d}\phi^{*}}{\mathrm{d}\theta}\\\
&=-\frac{\partial L}{\partial y}(\phi^{*})\frac{\partial
h}{\partial\phi}(h(\phi^{*}))\left(\frac{\partial
f}{\partial\phi}(\phi^{*},\theta)\right)^{-1}\frac{\partial
f}{\partial\theta}(\phi^{*},\theta).\end{split}$ (86)

Recurrent backpropagation. The implicit gradient in computationally
intractable for most of practical applications due to the matrix inversion.
The key idea behind the recurrent backpropagation algorithm is that the column
vector

$\delta^{*}:=\left(\frac{\partial
f}{\partial\phi}(\phi^{*},\theta)\right)\\!^{-\top}\frac{\partial
h}{\partial\phi}(\phi^{*})^{\top}\frac{\partial L}{\partial
y}(h(\phi^{*}))^{\top}$ (87)

is the steady state of the dynamics

$\dot{\delta}=\left(\frac{\partial
f}{\partial\phi}(\phi^{*},\theta)\right)^{\top}\delta-\frac{\partial
h}{\partial\phi}(\phi^{*})^{\top}\frac{\partial L}{\partial
y}(h(\phi^{*}))^{\top}$ (88)

or alternatively the fixed point of the iterative procedure

$\delta_{\mathrm{next}}=\delta+\left(\frac{\partial
f}{\partial\phi}(\phi^{*},\theta)\right)^{\top}\delta-\frac{\partial
h}{\partial\phi}(\phi^{*})^{\top}\frac{\partial L}{\partial
y}(h(\phi^{*}))^{\top}.$ (89)

Once $\delta^{*}$, or its estimation, is obtained through this dynamical
procedure, it can then be used to estimate the implicit gradient with

$\frac{\mathrm{d}}{\mathrm{d}\theta}L(\phi^{*})=-{\delta^{*}}^{\top}\frac{\partial
f}{\partial\theta}(\phi^{*},\theta)$ (90)

Backpropagation. When the computational graph encoded in $f$ is acyclic, that
is we can find a permutation of the state such that $f(\phi,\theta)_{i}$ only
depends on the states $\phi_{j}$ for $j\leq i$, the computation of
$\delta^{*}$ can be simplified. Indeed, instead of running the dynamics of Eq.
88, one can directly compute $\delta^{*}$ through the fixed-point equation

$\left(\frac{\partial
f}{\partial\phi}(\phi^{*},\theta)\right)^{\top}\delta=\frac{\partial
h}{\partial\phi}(\phi^{*})^{\top}\frac{\partial L}{\partial
y}(h(\phi^{*}))^{\top}.$ (91)

as the Jacobian $\partial_{\phi}f(\phi^{*},\theta)$ is a lower triangular
matrix. This is for example how $\delta^{*}$ is calculated for feedforward
neural networks. In such settings, we can therefore interpret the dynamics
(88) to be the continuous time version of the backpropagation algorithm.

Comparison with the least-control principle. We now compare the implicit
gradient, as computed with recurrent backpropagation, with the gradient of the
least-control objective, which is equal to

$\frac{\mathrm{d}\mathcal{H}}{\mathrm{d}\theta}(\theta)=-{\psi_{\scalebox{0.42}{$\bigstar$}}}^{\top}\frac{\partial
f}{\partial\theta}(\phi_{\scalebox{0.42}{$\bigstar$}},\theta)$ (92)

as given by Theorem 1. In addition to the close resemblance between the
gradients (90) and (92), the dynamical equations on $\delta$ and $\psi$ share
some interesting similarities. Let us take the example of the inversion
dynamics

$\begin{split}\dot{\phi}&=f(\phi,\theta)+\psi\\\
\dot{\psi}&=\left(\frac{\partial
f}{\partial\phi}(\phi,\theta)\right)^{\top}\psi+\frac{\partial
h}{\partial\phi}(\phi)^{\top}u\\\ \end{split}$ (93)

of Eq. 8. The $\psi$ dynamics are very similar to the one of $\delta$: they
can be understood as multiplying the inverse of the Jacobian of $f$ with an
error minimization signal, that is either $-\partial_{y}L$ or $u$, to yield
$\delta^{*}$ or $\psi_{\scalebox{0.42}{$\bigstar$}}$. The important conceptual
difference is that the controlled dynamics of the least-control principle
simultaneously compute both the steady state $\phi_{*}$ and optimal control
$\psi_{*}$ in a single phase, whereas recurrent backpropagation uses two
separate phases for computing $\phi^{*}$ and $\delta^{*}$. This unique feature
of the least-control principle is made possible by letting the control $\psi$
influence the dynamics and hence the steady state of $\phi$.

###  Appendix S4 On the relation between the least-control principle, free
energy and predictive coding

Predictive coding [36, 37, 56, 58]friston_free_2006 emerged as an important
framework to study perception and sensory processing in the brain. In this
framework, the brain is assumed to maintain a probabilistic model of the
environment, which is constantly used to perform inference on its sensory
inputs. Here, we expand in more detail on the relation between our least-
control principle and predictive coding. More specifically, we discuss how the
constrained energy minimization of Eq. 9 relates to the free energy used in
recent work on supervised predictive coding [36], and how the new insights of
the least-control principle, such as more flexible dynamics, translate to the
predictive coding framework.

####  S4.1 Supervised predictive coding

The starting point of standard predictive coding theories is the specification
of a probabilistic generative model from latent causes towards sensory inputs;
this model is then used to infer the most likely causes underlying incoming
sensory stimuli. Recent work [36] investigated supervised predictive coding, a
form of predictive coding where the direction of causality is flipped, i.e.,
where one uses a discriminative model from sensory inputs towards latent
causes, and trains this model by clamping its outputs to the correct teacher
value and performs inference to propagate teaching signals throughout the
network. Whittington and Bogacz [36] related the parameter updates resulting
from supervised predictive coding network to error backpropagation [31, 32]
when the output is weakly nudged towards the teacher value instead of being
clamped. In Section S4.3 we show that the least-control principle relates the