expression levels are observed. Thus the average in Eq (85) can be written as
an average over positions, so that

$\displaystyle I(\\{g_{\rm i}\\};x)$ $\displaystyle=$ $\displaystyle-\int
dx\,P_{X}(x)\ln P_{X}(x)-{1\over 2}\int dx\,P_{X}(x)\ln\left[2\pi
e\sigma_{x}^{2}(x)\right]$ (87) $\displaystyle=$ $\displaystyle-\int
dx\,P_{X}(x)\ln\left[P_{X}(x)\sqrt{2\pi e}\sigma_{x}(x)\right]$ (88)

which brings us back exactly to Eq (54). Although there are many dimensions to
the output of this system, the fact that there is only one dimension at the
input means that, at least in the low noise limit, we can get back to a
picture very much like the one input, one output system that we started with
in §3.1.

In particular, if we imagine that the embryo could adjust the distribution of
cell positions, then the optimal information transmission would occur when

$P_{X}(x)\propto{1\over{\sigma_{x}(x)}}.$ (89)

Since the real $P_{X}(x)$ is essentially uniform, optimal information
transmission predicts that $\sigma_{x}(x)$ should be uniform, and this is what
we see. Again, it requires considerable conspiracy among the four gap genes to
achieve uniform positional error, so this seems like a significant success for
the theory of optimal information transmission. There are two problems.

The first problem is that $\sigma_{x}(x)$ is not exactly uniform, but theory
actually gives us a way to measure the significance of these deviations. With
uniform $P_{X}(x)=1/L$ the actual information transmitted is, substituting
into Eq (88),

$I={1\over L}\int_{0}^{L}dx\,\ln\left[{L\over{\sqrt{2\pi
e}\sigma_{x}(x)}}\right].$ (90)

On the other hand, if we are free to optimize $P_{X}(x)$ over the range
$0<x<L$ then form Eq (61) the maximum information possible given the measured
$\sigma_{x}(x)$ becomes

$I_{\rm max}=\ln\left[{1\over{\sqrt{2\pi
e}}}\int_{0}^{L}{{dx}\over{\sigma_{x}(x)}}\right].$ (91)

If we plug in the results from our analysis of the experimental data, we
find313131Estimating information theoretic quantities from real data can be
challenging, and there are interesting theoretical questions about how best to
do this, especially if you need control over the error bars (as in this case).
See, for example, Appendix A.8 of Ref [57].

${I\over{I_{\rm max}}}=0.984\pm 0.003.$ (92)

Thus optimization of information transmission predicts a match between the
positional noise levels and the distribution of cell positions, and this match
is sufficiently good that it brings the embryo within $2\%$ of the optimum
[120].

The second problem is that it seems a bit weird to talk about redistributing
cells along the embryo’s axis. We don’t really need to do this. The gap gene
network output can be thought of not as providing information about position
but rather about the concentrations of the maternal input molecules. The
mapping between position and input concentration is something that could be
different with different parameters of the relevant dynamics, different
anchoring of the mRNA molecules, etc.. Thus we can imagine a family of embryos
with different possible distributions of inputs to the gap gene network. But
if the mapping between position and input concentration is deterministic and
invertible, then information about concentration is the same as information
about position, and all of the arguments here about matching go through.

To get a rough estimate of the total positional information conveyed by the
gap genes we can use Eq (90) but with $\ln\rightarrow\log_{2}$ so the units
are bits, and the approximate $\sigma_{x}/L\sim 0.01$, which gives
$I\sim\log_{2}(100/\sqrt{2\pi e})\sim 4.6\,{\rm bits}$. We can’t see all of
this because we are measuring only in the central $80\%$ of the
anterior–posterior axis ($0.1<x/L<0.9$); outside this region imaging becomes
prone to systematic errors from the curvature of the embryo. But if we correct
for this then the integral gives a result very close to the rough answer, and
also very close to a more brute force integration over the four dimensional
space $\\{g_{\rm i}\\}$, which does not require any small noise or Gaussian
approximations [120, 165].

The actual number of bits is not that much more than four, and there are four
gap genes … maybe it’s just one bit per gene after all? The way to test this
is to imagine that the readout mechanisms can only resolve two levels of
expression, high and low; formally this means transforming

$g_{\rm i}\rightarrow\sigma_{\rm i}=H(g_{\rm i}-\theta_{\rm i}),$ (93)

where $H$ is the step function and $\theta$ is a threshold that divides the
two levels. If we know the thresholds then we can compute
$I_{\mathbf{\theta}}(\\{\sigma_{\rm i}\\};x)$, where the notation reminds us
that the answer depends on the thresholds ${\mathbf{\theta}}=\\{\theta_{\rm
i}\\}$. To be generous, we can optimize the thresholds, maximizing how much
information this on/off description of gene expression can capture. The answer
is that $I(\\{\sigma_{\rm i}\\};x)<3\,{\rm bits}$ [120]. Further, this
information is distributed very inhomogenously, so that some “binary words”
$\\{\sigma_{\rm i}\\}$ point to $x$ values that span $\sim 10\%$ of the
anterior–posterior axis. Thus, to extract $\sim 4\,{\rm bits}$ of positional
information from four genes requires mechanisms that have much more than this
capacity to read out the expression levels.

Recently we returned to the issue of readout precision [169, 170]. We know, as
just explained, that reading each concentration with one bit of precision is
not enough to extract all the available information. On the other hand, our
discussion of decoding positional information from the gap genes (§2) assumed
that cells had access to the true measured concentrations of each molecule,
which surely is unrealistic. How can the embryo best trade bits of precision
in the readout against the bits of relevant positional information that are
preserved?

Formally we can imagine that the expression levels of the gap genes are mapped
into some intermediate variable, such as the occupancy of binding sites along
the DNA; let’s call this intermediate variable $C$. Inevitably this mapping is
noisy, and this means that the information which $C$ can carry about
$\\{g_{\rm i}\\}$, $I(C;\\{g_{\rm i}\\})$, is limited. Given this limitation,
what mapping $\\{g_{\rm i}\\}\rightarrow C$ will maximize the information
about position $I(C;x)$? This defines a new optimization problem

$\max_{P(C|\\{g_{\rm i}\\})}\left[I(C;x)-TI(C;\\{g_{\rm i}\\})\right],$ (94)

where $T$ is a Lagrange multiplier and we are explicit that the mapping
$\\{g_{\rm i}\\}\rightarrow C$ is probabilistic; this is an example of the
information bottleneck problem [171]. To extract all the available
information, that is to have $I(C;x)\approx I(\\{g_{\rm i}\\};x)$, requires
readout mechanisms with a capacity $I(C;\\{g_{\rm i}\\})$ of at least $8\,{\rm
bits}$ [169]. Further, the most efficient mechanisms involve $C$ being
sensitive to combinations of the different expression levels; if we have
separate sensors $g_{\rm i}\rightarrow C_{\rm i}$ they need to have vastly
more capacity. Finally, the embryo is in a regime where the trading of
$I(C;x)$ vs $I(C;\\{g_{\rm i}\\})$ has a universal form [170]. If we take
seriously the idea that $C$ is something like the occupancy of binding sites,
or the collective states of the “enhancers” into which these sites are
grouped, then these arguments about trading bits provide a path to predict the
molecular mechanisms that instantiate the optimal decoding strategies of §2.3.

Before leaving this topic I want to emphasize that Figs 18, 19, and 20 all are
testing the same optimization principle. Thus we are making predictions about
neurons that generate graded voltages, neurons that generate spikes, and
networks of genes, all with the same theoretical idea and all connecting to
experiment with no free parameters.

##  4 Network architecture

In the flow of positional information from the maternal inputs to the gap
genes to the pair rule genes, we have seen evidence for optimization in the
distribution of inputs (§3.3) and in the processing of the outputs (§2.3). We
now have to ask if we can apply optimization principles to the network itself.
Doing this involves returning to the challenge laid out in the first lecture:
realistic descriptions of the gap gene network require 50+ parameters, and are
in some obvious sense very complicated. To approach optimization of such a
complex network it will be useful to break off smaller pieces of the problem
and gain intuition.

The idea that we can derive the functional behavior of a network from the
optimization of information transmission is quite old, having its origins in
the context of sensory information processing by the brain. Just a decade
after Shannon’s original work on information theory the National Physical
Laboratory in the UK hosted a remarkable Symposium on the Mechanization of
Thought Processes. Among other presentations, Horace Barlow spoke about
“Sensory mechanisms, the reduction of redundancy, and intelligence” [172].
This was, I think, the first place where optimization principles for neural
information processing were articulated.323232Barlow revisited these ideas at
another conference a few years later, and this is the more widely cited
version of his ideas about “efficient coding” [173]. Thinking about our modern
publication habits, it seems worth noting that this work had impact on
generations of scientists even though there is no regular journal article to
cite. It also seems worth emphasizing the ambition that one sees in these
titles, both of Barlow’s paper and for the symposium as a whole.

###  4.1 Linear filtering in neural networks

Figure 21: Transformation from inputs $\\{x_{\rm i}\\}$ to outputs $\\{y_{\rm
i}\\}$, as in Eq (95). Only a fraction of possible connections $M_{\rm ij}$
are illustrated; fewer still are labelled. Not shown are the independent noise
sources added to each output element.

Let’s dive in and think about a network in which a layer of input variables
$\\{x_{\rm i}\\}$ drives a layer of output variables $\\{y_{\rm i}\\}$, as in
Fig 21. The simplest possibility is that the transformation is a noisy linear
mapping, so that

$y_{\rm i}=\sum_{\rm j}M_{\rm ij}x_{\rm j}+\zeta_{\rm i};$ (95)

again to keep things simple we imagine the number of inputs and outputs are
the same so ${\rm i}=1,\,2,\,\cdots,\,N$ and similarly for $\rm j$. Let the
noise $\zeta_{\rm i}$ be Gaussian and independent at every site $\rm i$. In
this case optimizing the network means choosing the transformation matrix
$\hat{M}$ so maximize the mutual information between inputs and outputs. If
the variance of the noise is fixed, then one can always increase the mutual
information by increasing the magnitude of $\hat{M}$, which seems like
cheating. To have a well defined problem we should bound the magnitudes, so
that the scale of the noise has meaning in relation to the inputs; a
conventional choice is to constrain the summed variances of the outputs. This
leaves us with the optimization problem

$\max_{\hat{M}}\left[I\left(\\{y_{\rm i}\\};\\{x_{\rm
i}\\}\right)-\mu\sum_{{\rm i}=1}^{N}\langle y_{\rm i}^{2}\rangle\right].$ (96)

The structure of this problem depends on the distribution $P(\\{x_{\rm i}\\})$
from which inputs are drawn.

The general version of Eq (96) is challenging. It is much easier if we can
approximate $P(\\{x_{\rm i}\\})$ as Gaussian. Then we have

$\displaystyle P(\\{x_{\rm i}\\})$ $\displaystyle=$
$\displaystyle{1\over\sqrt{(2\pi)^{N}}}\exp\left[-{1\over
2}\ln\det\hat{C}_{x}-{1\over 2}\sum_{{\rm i},{\rm j}}x_{\rm
i}\left(\hat{C}_{x}^{-1}\right)_{\rm ij}x_{\rm j}\right]$ (97) $\displaystyle
P(\\{y_{\rm i}\\}|\\{x_{\rm i}\\})$ $\displaystyle=$
$\displaystyle{1\over\sqrt{(2\pi)^{N}}}\exp\left[-{N\over
2}\ln\langle\zeta^{2}\rangle-{1\over{2\langle\zeta^{2}\rangle}}\sum_{{\rm
i}}\left(y_{\rm i}-\sum_{\rm j}M_{\rm ij}x_{\rm j}\right)^{2}\right]$ (98)
$\displaystyle P(\\{y_{\rm i}\\})$ $\displaystyle=$
$\displaystyle{1\over\sqrt{(2\pi)^{N}}}\exp\left[-{1\over
2}\ln\det\hat{C}_{y}-{1\over 2}\sum_{{\rm i},{\rm j}}y_{\rm
i}\left(\hat{C}_{y}^{-1}\right)_{\rm ij}y_{\rm j}\right];$ (99)

the covariance matrices are

$\displaystyle\left(\hat{C}_{x}\right)_{\rm ij}$ $\displaystyle=$
$\displaystyle\langle x_{\rm i}x_{\rm j}\rangle$ (100)
$\displaystyle\left(\hat{C}_{y}\right)_{\rm ij}$ $\displaystyle=$
$\displaystyle\langle y_{\rm i}y_{\rm j}\rangle=M_{\rm
ik}\left(\hat{C}_{x}\right)_{\rm kl}M_{\rm
jl}+\langle\zeta^{2}\rangle\delta_{\rm ij}$ (101)
$\displaystyle\Rightarrow\hat{C}_{y}$ $\displaystyle=$
$\displaystyle\hat{M}\hat{C}\hat{M}^{T}+\langle\zeta^{2}\rangle\mathbb{1}.$
(102)

We can substitute into the mutual information

$\displaystyle I\left(\\{y_{\rm i}\\};\\{x_{\rm i}\\}\right)$
$\displaystyle\equiv$ $\displaystyle\int d^{N}x\int d^{N}y\,P(\\{y_{\rm
i}\\}|\\{x_{\rm i}\\})P(\\{x_{\rm i}\\})\log\left[{{P(\\{y_{\rm
i}\\}|\\{x_{\rm i}\\})}\over{P(\\{y_{\rm i}\\})}}\right]$ (103)
$\displaystyle=$ $\displaystyle{\bigg{\langle}}\log\left[{{P(\\{y_{\rm
i}\\}|\\{x_{\rm i}\\})}\over{P(\\{y_{\rm i}\\})}}\right]{\bigg{\rangle}}$
(105) $\displaystyle=$ $\displaystyle-{N\over
2}\ln\langle\zeta^{2}\rangle-{1\over{2\langle\zeta^{2}\rangle}}\sum_{{\rm
i}}{\bigg{\langle}}\left(y_{\rm i}-\sum_{\rm j}M_{\rm ij}x_{\rm
j}\right)^{2}{\bigg{\rangle}}$
$\displaystyle\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,+{1\over
2}\ln\det\hat{C}_{y}+{1\over 2}\sum_{{\rm i},{\rm j}}\langle y_{\rm i}y_{\rm
j}\rangle\left(\hat{C}_{y}^{-1}\right)_{\rm ij}.$

Notice that, from Eq (95), we have

$y_{\rm i}-\sum_{\rm j}M_{\rm ij}x_{\rm j}=\zeta_{\rm
i}\Rightarrow{1\over{2\langle\zeta^{2}\rangle}}\sum_{\rm
i}{\bigg{\langle}}\left(y_{\rm i}-\sum_{\rm j}M_{\rm ij}x_{\rm
j}\right)^{2}{\bigg{\rangle}}={1\over{2\langle\zeta^{2}\rangle}}\sum_{\rm
i}\langle\zeta^{2}\rangle={N\over 2}.$ (106)

Similarly

$\langle y_{\rm i}y_{\rm j}\rangle=\left(\hat{C}_{y}\right)_{\rm
ij}\Rightarrow{1\over 2}\sum_{{\rm i},{\rm j}}\langle y_{\rm i}y_{\rm
j}\rangle\left(\hat{C}_{y}^{-1}\right)_{\rm ij}={1\over 2}\sum_{{\rm i},{\rm
j}}\left(\hat{C}_{y}\right)_{\rm ij}\left(\hat{C}_{y}^{-1}\right)_{\rm
ij}={N\over 2}.$ (107)

Thus the complicated looking summations cancel and we have

$I\left(\\{y_{\rm i}\\};\\{x_{\rm i}\\}\right)=-{N\over
2}\ln\langle\zeta^{2}\rangle+{1\over 2}\ln\det\hat{C}_{y}={1\over 2}{\rm
Tr}\ln\left[\mathbb{1}+{1\over{\langle\zeta^{2}\rangle}}\hat{M}\hat{C}\hat{M}^{T}\right]$
(108)

If the eigenvalues of $\hat{M}\hat{C}\hat{M}^{T}$ are $\lambda_{\rm n}$, then
the optimization problem in Eq (96) becomes

$\max\left[{1\over 2}\sum_{\rm n}\ln\left(1+\lambda_{\rm
n}/\langle\zeta^{2}\rangle\right)-\mu\sum_{\rm n}\lambda_{\rm n}\right].$
(109)

The optimum is reached where all the $\lambda_{\rm n}$ are equal.

The correlation structure of the inputs usually is non–trivial, so the
eigenvalues of the covariance matrix $\hat{C}_{x}$ are spread over some
spectrum. Optimizing information transmission in the class of problems defined
by Eq (95) means rearranging these inputs to “whiten” this spectrum.333333The
name comes from the fact that in truly white light all components of the
spectrum have equal weight. Notice that if all eigenvalues of
$\hat{M}\hat{C}_{x}\hat{M}^{T}$ are equal then $\hat{C}_{y}$ is proportional
to the unit matrix and hence different signals $y_{\rm i}$ and $y_{{\rm
j}\neq{\rm i}}$ are independent of one another. Thus the optimal $\hat{M}$
diagonalizes the covariance matrix of the input signals, transforming to
principal components, and then rescales these so that they have equal
variances.

A simple but important example of these ideas is color vision. Recall that in
daylight our vision is based on three kinds of cones, each tuned to a
different range of photon energies. We can think of these as sensitive
approximately to red, green, and blue, though you should never say this in
front of someone who actually studies color vision—“red” is a percept, not the
label of a cell type; the convention is to describe the different cones as
sensitive to long, medium, and short wavelengths. It is an old idea that the
three cone signals are processed in well defined combinations corresponding
roughly to the percepts of luminance, red vs green, and blue vs yellow. The
history of these ideas is complicated, with all sorts of heroic figures
shouting at one another. Thing were confusing in part because the “opponent
process” theory seemed to involve four color axes (red, green, blue, and
yellow) while the alternative “trichromatic” theory involved only three (which
we now attach to the three types of cones). It seems to have been Schrödinger
who sorted this out in 1925 [174]. We now know that the three cone signals
indeed are grouped together in the three combinations already in the retina,
and it is these combined signals that are transmitted to the brain [175].

In 1983, Buchsbaum and Gottschalk suggested that the transformation into
luminance, red vs green, and blue vs yellow might be the solution to the
optimization problem we are discussing here [176]. In order to see if this
makes sense, we need to know the covariance matrix of the three cone signals.
This depends strongly on the absorption spectra of the cone pigments, with
large positive correlations arising simply because these spectra overlap. But
the cone signal statistics also depend on the spectral composition of the
images that the eye is looking at. To get at these, Ruderman, Cronin, and
Chiao used a hyperspectral camera take pictures in natural
environments—woodlands, forests and rainforests, and a mangrove swamp [177].
In effect this camera does spectroscopy in each pixel of a digital image, and
then these spectrally resolved images can be projected onto the known
sensitivities of the individual cone pigments to estimate the photon capture
in each of the three cones looking at the same point. In keeping with the
discussion of adaptation above (§3.2) the cone signals $x_{\rm i}$ were taken
as the log of the number of photon counts.343434We use our cones in bright
light, so there is no worry about zero counts.

The results obtained by Ruderman et al are remarkably simple and crisp. In our
notation, the optimal matrix $\hat{M}$ takes the form

$\hat{M}=\left[\begin{array}[]{ccc}{1\over{\sqrt{3}\sigma_{\ell}}}&0&0\\\
0&{1\over{\sqrt{6}\sigma_{by}}}&0\\\
0&0&{1\over{\sqrt{2}\sigma_{rg}}}\end{array}\right]\left[\begin{array}[]{rrr}1.004&1.005&0.991\\\
1.014&0.968&-2.009\\\ 0.993&-1.007&0.016\end{array}\right].$ (110)

We see that $y_{1}$ is the sum of the signals in the three cones, and the
weights are equal in the second decimal place or better. Similarly $y_{3}$ is
the difference between the signals in long and medium length cones, with
almost no contribution from the short wavelength cones; this is the “red minus
green” opponent channel. Finally $y_{2}$ combines the long and medium cone
signals and subtracts the short wavelength signal, corresponding to “yellow
minus blue.” As with the luminance channel $y_{1}$, the combinations of cone
signals in the two opponent channels are in integer ratios, with one percent
accuracy. The signals along the different channels have standard deviations in
the ratio $\sigma_{\ell}:\sigma_{by}:\sigma_{rg}\sim 1:0.2:0.02$. While we
have a great appreciation for the subtleties of color, most of the variance in
the images that we is in the luminance channel.353535In fact this might be a
cautionary tale about dimensionality reduction. We can capture $\sim 95\%$ of
the variance in what we see with just one dimension, corresponding to a
completely greyscale world. Embarrassingly, I don’t know if the combination of
cone signals into neural signals has been measured precisely enough to test
these predictions of integer ratios.

Staying with the visual system, we can think of the $\\{x_{\rm i}\\}$ as
signals from cones at different positions (now neglecting color). If the cones
form a regular lattice, and the input visual signals are translationally
invariant, then everything will be diagonal after a Fourier transform. Let’s
call $\mathbf{k}$ the spatial Fourier variable, and then

$\tilde{y}({\mathbf{k}})=\tilde{M}({\mathbf{k}})\tilde{x}({\mathbf{k}})+\tilde{\zeta}({\mathbf{k}}).$
(111)

Further, the transformation $\tilde{M}({\mathbf{k}})$ that maximizes
information transmission will obey

$|\tilde{M}({\mathbf{k}})|\propto{1\over{\sqrt{S_{x}({\mathbf{k}})}}},$ (112)

where $S_{x}({\mathbf{k}})$ is the power spectrum of the signals $\\{x_{\rm
i}\\}$ [178, 179]. We know that the spatial power spectrum of natural images
is scale invariant [180],

$S({\mathbf{k}})={A\over{|{\mathbf{k}}|^{2-\eta}}},$ (113)

so the prediction is
$|\tilde{M}({\mathbf{k}})|\propto|{\mathbf{k}}|^{1-\eta/2}$. The growth with
$|{\mathbf{k}}|$ is cut off by the effects of noise in the inputs $x_{\rm i}$,
e.g. the random arrival of photons.

Given the statistical structure of natural images, optimization of information
flow predicts that there will be zero gain for zero spatial frequency,
$|\tilde{M}({\mathbf{k}}=0)|=0$. The transformation $M_{\rm ij}$ thus serves
as roughly as a spatial differentiator, so that the output $y_{\rm i}$ is
large when the pattern $\\{x_{\rm j}\\}$ includes a sharp edge, and is small
when the $\\{x_{\rm j}\\}$ are nearly uniform. Qualitatively these predictions
are correct for neurons in the retina. The output cells, which carry
information from eye to brain, have long been known to have “center–surround”
receptive fields [181, 182]: the response of these neurons is driven by the
difference between the average light intensity in a small region and the
average over a larger surrounding region, and in many cases the two regions
have equal weight so that the response is differentiating. Barlow understood
that this sort of spatial differencing, also called “lateral inhibition”
[183], would remove the redundancy between signals in neighboring
photoreceptors, enhancing the transmission of information by limited numbers
of action potentials along limited numbers of neurons [172, 173]; the analysis
here translates this qualitative observation into equations.

These arguments about whitening apply also in the time domain. In the simplest
example

$y(t)=\int d\tau M(\tau)x(t-\tau)+\eta(t).$ (114)

If $\eta$ is white noise, $\langle\eta(t)\eta(t^{\prime})\rangle={\cal
N}_{0}\delta(t-t^{\prime})$, then the rate at which information about $x(t)$
is conveyed by $y(t)$ is [57, 147, 184]

$R_{\rm info}={1\over 2}\int{{d\omega}\over{2\pi}}\ln\left[1+{1\over{{\cal
N}_{0}}}|\tilde{M}(\omega)|^{2}S_{x}(\omega)\right],$ (115)

where $S_{x}(\omega)$ is the power spectrum of the input signal

$\langle
x(t)x(t^{\prime})\rangle=\int{{d\omega}\over{2\pi}}e^{-i\omega(t-t^{\prime})}S_{x}(\omega).$
(116)

If we optimize $R_{\rm info}$ while holding fixed the output dynamic range
$\langle y^{2}\rangle$, then we find an analog of Eq (112),

$|\tilde{M}(\omega)|\propto{1\over\sqrt{S_{x}(\omega)}}.$ (117)

If input signals have scale invariant dynamics, $S_{x}(\omega)\sim
1/|\omega|$, then we predict $|\tilde{M}(\omega)|\sim\sqrt{|\omega|}$. This is
weird, since it corresponds to the filter $M$ taking half of a derivative.

Direct measurements of the transformation $M$ between neurons are not so easy.
Our linear models make more sense if the cells have graded voltage responses,
as in the first stages of vision. In the fly retina, for example, both the
photoreceptor cells and the next cells in line (large monopolar cells, LMCs
from §3.2) have a large regime in which the average voltage $V(t)$ responds
linear to time variations in light intensity $I(t)$ around a background
$I_{0}$,

$\langle V(t)\rangle-V_{0}={1\over{I_{0}}}\int
d\tau\,T(\tau)[I(t-\tau)-I_{0}]$ (118)

defining a transfer function $T(\tau)$; it is more convenient to think about
the Fourier transform $\tilde{T}(\omega)$. These transfer functions can be
measure in both the receptors and LMCs, with results in Fig 22A. Then we can
take the ratio to estimate the filter across the synapse,

$\tilde{M}(\omega)={{\tilde{T}_{\rm LMC}(\omega)}\over{T_{\rm
receptor}(\omega)}},$ (119)

with results in Fig 22B. Strikingly, we really do see
$|\tilde{M}(\omega)|\sim\sqrt{|\omega|}$ across a wide range of frequencies.
Such fractional differentiation is much more widespread, perhaps serving to
optimize the transmission of information about scale invariant signals
throughout the brain [186, 187]. These ideas have distant but fascinating
precursors [188].

Figure 22: Filtering at the first synapse in fly vision. (A) Linear response
of voltage to temporal variations in light intensity, as in Eq (118), for
photoreceptors and large monopolar cells (LMC). (B) Effective transfer
function between photoreceptors and LMCs, from Eq (119). Redrawn from Ref
[185].

###  4.2 Ingredients of a genetic network

The simplest example of a genetic network was shown schematically in Fig 3. To
be more consistent with the notation below, let’s describe this by saying that
a single transcription factor (TF) at concentration $c$ controls the
expression level $g$ of a target gene. We’ll assume for simplicity that the
system is in steady state.

The information that $g$ provides about the input concentration $c$ is

$I(g;c)=\int dg\int dcP(g|c)P_{\rm in}(c)\log\left[{{P(g|c)}\over{P_{\rm
out}(g)}}\right],$ (120)

where the distribution of outputs

$P_{\rm out}(g)=\int dc\,P(g|c)P_{\rm in}(c).$ (121)

Consistent with the analysis of decoding in §2.3, we’ll assume that the noise
in the response of $g$ to the input $c$ is Gaussian, so that

$P(g|c)={1\over\sqrt{2\pi\sigma_{g}^{2}(c)}}\exp\left[-{{(g-\bar{g}(c))^{2}}\over{2\sigma_{g}^{2}(c)}}\right].$
(122)

We expect that the mean expression level $\bar{g}(c)$ has a sigmoidal
dependence on the TF concentration, as in Fig 3, which we can write as

$\bar{g}(c)={{c^{h}}\over{K^{h}+c^{h}}};$ (123)

$h>0$ means that the TF activates expression and $K$ is the concentration at
which this effects is half–maximal. More quantitatively $h$ is a measure of
sensitivity. We can rationalize this form by imagining the $h$ molecules of
the TF bind cooperatively to sites along the DNA, and this binding influences
transcription. To complete our description, and the calculation of the
information transmission, we need to know the noise level $\sigma_{g}(c)$.

The molecules whose concentration is measured by $g$ ultimately are made one
at a time, and these are random events. So we expect $\sqrt{N}$ fluctuations
in making $N$ molecules. We have to be a bit careful, first because we have
chosen units in which the maximum $\bar{g}$ is unity, and second because many
proteins can be translated from a single mRNA molecule before it is degraded.
Thus we can write the contribution of this counting noise as

$\sigma_{g,{\rm count}}^{2}(c)={1\over{N_{\rm max}}}\bar{g}(c),$ (124)

where $N_{\rm max}$ is the maximum number of independent molecules being made.
Probably this is the number of mRNAs, but we can be flexible.

In order to regulate gene expression, the TF molecules have to arrive at a
small target along the DNA, where they bind to specific sequences. We can thus
think of the regulatory mechanism as a small sensor of transcription factor
concentration, with gene expression as the readout. The physical limits to
sensing concentrations were first discussed by Berg and Purcell in the context
of bacterial chemotaxis [56]. This remains one of the foundational papers of
our field, so it is worth taking a detour to review their arguments, and some
of the subsequent developments, leading up to Eq (136).

Figure 23: Schematic of concentration measurements [56, 57]. A receptor of
linear dimension $a$ samples a volume $\sim a^{3}$ and thus counts a mean
number of molecules $\bar{n}\sim ca^{3}$, where $c$ is the concentration.
Molecules random walk in and out of the sensitive volume with a diffusion
constant $D$, corresponding to an exchange or correlation time $\tau_{c}\sim
a^{2}/D$.

A sensor of linear dimension $a$ sitting in a solution of molecules with
concentration $c$ will count $\bar{n}\sim ca^{3}$ molecules on average, as
shown in Fig 23. But as molecules diffuse in and out of the sensitive volume,
this molecule count will fluctuate by $\delta n\sim\sqrt{\bar{n}}$. Since
concentration is proportional to molecule number, this means that estimates of
concentration will have a fractional fluctuation

${{\delta c}\over c}{\bigg{|}}_{1}\sim{{\delta
n}\over{\bar{n}}}\sim{1\over{\sqrt{ca^{3}}}},$ (125)

where the subscript reminds us that this is based on one snapshot of molecule
counts. We should be able to reduce the noise by averaging over time, but this
works only if we make multiple independent measurements; just counting the
same molecules again doesn’t give a better estimate of the surrounding
concentration. It takes a time $\tau_{c}\sim a^{2}/D$ for molecules in the
sensitive volume to exchange with the bulk solution via diffusion, so if we
average over time $\tau_{\rm avg}$ we can make $\tau_{\rm avg}/\tau_{c}$
independent measurements, and thus reduce the noise

${{\delta c}\over c}\rightarrow{{\delta c}\over
c}{\bigg{|}}_{1}\cdot{1\over\sqrt{\tau_{\rm
avg}/\tau_{c}}}\sim{1\over\sqrt{Dac\tau_{\rm avg}}}.$ (126)

We see that the different parameters of the problem combine in a very simple
way. This is the “Berg–Purcell limit” to concentration sensing.

It’s always good to check that the units work out:

$Dac\tau_{\rm
avg}=\left[\ell^{2}/t\right]\left[\ell\right]\left[1/\ell^{3}\right]\left[t\right]=\left[\right],$
(127)

so this is a dimensionless combination. Note that the $3$ in $[1/\ell^{3}]$ is
from three dimensions, so the answer must be different if we are trying to
sense the concentration of molecules diffusing in a membrane, or if there is a
significant contribution from proteins sliding along the length of the
(one–dimensional) DNA molecule [189].

It is useful to note that $\sim Dac$ is the mean rate at which molecules
arrive at a (three–dimensional) target of size $a$ via diffusion. For example,
if we have the chemical reaction $A+B\rightarrow AB$, we write the dynamics of
the concentration as

${{d[AB]}\over{dt}}=k_{2}[A][B],$ (128)

where $k_{2}$ is a “second order rate constant.” This rate constant is bounded
by the rate at which $A$ and $B$ can find each other, and this is $k_{2}\sim
Da$, where $a$ is the size of the molecules. One can make this precise by
solving the diffusion equation with appropriate boundary conditions,363636See
Problem 54 in Ref [57]. and develop more precise models in which the molecules
have to approach one another at the correct orientation. Thus one can think of
$Dac$ as the rate at which molecules arrive at the sensor, and $Dac\tau_{\rm
avg}$ as the mean number of molecules that our sensor counts. The Berg–Purcell
limit then is the statement that there are $\sqrt{n}$ fluctuations in counting
molecules, just as with counting photons from a conventional light source.

In this discussion the linear dimensions of the sensor $a$ is a rough concept.
In their original discussion, Berg and Purcell were thinking about a bacterium
sensing the concentration of attractive or repulsive molecules in its
environment, and the initial guess is that $a$ is just the size of the
bacterium itself. But this is weird, since what really happens is that
molecules bind to particular receptors on the cell surface. These receptors
are themselves of molecular dimensions—nanometers rather than the micron size
of the whole cell—and receptors for any particular molecules cover only a tiny
fraction of the cell’s surface.

If there is just one receptor of linear dimension $a_{\rm r}$ then the
Berg–Purcell argument should still work. If there are $N_{\rm r}$ of these
receptors then it is plausible that noise is reduced by a factor
$1/\sqrt{N_{\rm r}}$, which is equivalent to saying they act together as a
single receptor with effective size $a_{\rm eff}\sim N_{\rm r}a_{\rm r}$. But
if the number of receptor becomes large enough that the area occupied by the
receptors becomes comparable to the area of the cell surface, $N_{\rm r}a_{\rm
r}^{2}\sim a_{\rm cell}^{2}$, then it seems like the whole cell should act as
one big receptor and $a_{\rm eff}\rightarrow a_{\rm cell}$. How does this
work?

In a fabulous bit of hand waving plus analogies, Berg and Purcell argued that
the tendency of random walk trajectories to bounce along a surface leads to
correlations in the encounters of individual molecules with nearby receptors.
With $N_{\rm r}$ receptors scattered over the surface of the cell, this lead
to an effective size

$a_{\rm eff}\sim a_{\rm cell}{{N_{r}a_{\rm r}}\over{N_{r}a_{\rm r}+a_{\rm
cell}}}.$ (129)

Notice that this saturates when $N_{r}a_{\rm r}\sim a_{\rm cell}$, at which
the fractional coverage of the surface is $N_{r}a_{\rm r}^{2}/a_{\rm
cell}^{2}\sim a_{\rm r}/a_{\rm cell}\sim 10^{-3}$. This is a dramatic effect,
and matters enormously in the life of the cell, which can act as one big
sensor even though only a small fraction of its surface is covered by any
single receptor type.

The goal of Berg and Purcell’s work was (in part) to compare the physical
limits to concentration sensing with the performance of real bacteria as they
navigate through chemical gradients. The result was that it would be
physically impossible for cells to make decisions with the observed
reliability if they were making spatial measurements, comparing concentrations
at head vs. tail to see if they are moving in the right direction. Instead
they must measure changing concentrations in time along the path taken as they
swim. They need to average these time derivatives, effectively comparing the
recent past with a longer term average, but the duration of this averaging is
limited by the rotational Brownian motion of the cell itself. The result is a
semi-quantitative theory of what bacteria must do in order to achieve their
observed chemotactic performance, and all of these predictions proved to be
correct.

One question is whether the Berg–Purcell arguments could be sharpened to give
a more quantitative theory of chemotactic strategies, and this continues to be
an interesting direction. More relevant to our discussion is whether these
limits to concentration sensing are sufficiently general that they can be
applied, for example, to the problem of transcriptional control. This matters
because of a simple order–of–magnitude argument. Transcription factors
function at concentrations of tens of nanoMolar,

$c\sim 10\,{\rm nM}=10\times 10^{-9}\times(6\times
10^{23}){1\over{10^{3}\,({\rm cm})^{3}}}\times\left({{10^{-4}\,{\rm
cm}}\over{\mu{\rm m}}}\right)^{3}\sim 6\,(\mu{\rm m})^{-3}.$ (130)

Diffusion constants for proteins in the cytoplasm are $D\sim 1\,\mu{\rm
m}^{2}/{\rm s}$, and the target to which these molecules bind has dimensions
$a\sim 3\,{\rm nm}$. The result is that

$Dac\sim 2\times 10^{-2}\,s^{-1}\Rightarrow{{\delta c}\over
c}\sim{1\over{\sqrt{Dac\tau_{\rm
avg}}}}\sim\left({{1\,\rm{min}}\over{\tau_{\rm avg}}}\right)^{1/2}.$ (131)

This suggests that reliable responses to ten percent differences in
transcription factor concentration will require more than an hour of temporal
integration. Neighboring cells in the fly embryo experience maternal inputs
that differ by $\sim 10\%$ in concentration, and they reliably express
different combinations of gap genes; this certainly takes less than one hour
[125].

The conclusion from Eq (131) is not that transcriptional regulation definitely
reaches the physical limits to concentration sensing, but rather that it
operates in a regime where these limits are relevant. Even this more modest
conclusion hinges on applying the Berg–Purcell ideas far from their origins;
this is made more uncertain by the beautifully intuitive but non–rigorous
nature of the original arguments.373737I never had the chance to discuss this
with Purcell, but Berg was clear that their arguments were rough and that it
would be nice to have something more rigorous. Some years ago my colleagues
and I started to worry about this [190, 191, 189], and by now the literature
has grown substantially [192, 193, 194, 195, 196, 197, 198, 199]. Here is what
I think we know:

  * •

One way to make the Berg–Purcell argument rigorous is to analyze the
fluctuations in occupancy of a receptor binding site as it comes to
equilibrium with molecules diffusing in the surrounding solution. These
fluctuations are a form of thermal noise, subject to the
fluctuation–dissipation theorem.

  * •

Using the fluctuation–dissipation theorem we can see explicitly how diffusion
among nearby receptors generates correlated noise and results with the flavor
of Eq (129).

  * •

Cooperativity among multiple binding events enhances sensitivity and reduces
excess noise, but never below the bound set by Berg and Purcell.

  * •

Analytic results from the fluctuation dissipation theorem can be reproduced by
careful numerical simulation, though small discrepancies remain to be
understood.

  * •

Cells can push below the Berg–Purcell limit by factors $\sim 2$ with signal
processing strategies that are more sophisticated than just averaging receptor
occupancy.

  * •

These strategies only work away from equilibrium, providing a path to connect
energy dissipation and signaling accuracy.

  * •

There are interesting generalizations to sensing time dependent
concentrations, or the concentrations of multiple species by multiple
receptors; results can be counterintuitive.

From all this work, the general conclusion is that Berg and Purcell got it
right: Eq (126) defines a minimal noise level for sensing concentrations, up
to factors of order unity that could also be seen as ambiguity in defining the
size $a$ of the detector.

What the Berg–Purcell limit tells us is that the random arrival of molecules
at their target site generate an effective concentration noise with variance

$\sigma_{c}^{2}=\left({{\delta c}\over
c}\right)^{2}c^{2}\sim{c\over{Da\tau_{\rm avg}}}.$ (132)

This effective concentration noise will propagate through the genetic
regulatory element, contributing to the variance in the output as

$\sigma_{g,{\rm
BP}}^{2}(c)\sim{\bigg{|}}{{d\bar{g}(c)}\over{dc}}{\bigg{|}}^{2}\sigma_{c}^{2}.$
(133)

Putting this together with the counting noise from Eq (124) we have

$\displaystyle\sigma_{g}^{2}(c)$ $\displaystyle=$ $\displaystyle\sigma_{g,{\rm
count}}^{2}(c)+\sigma_{g,{\rm BP}}^{2}(c)={1\over{N_{\rm
max}}}\bar{g}(c)+{\bigg{|}}{{d\bar{g}(c)}\over{dc}}{\bigg{|}}^{2}{c\over{Da\tau_{\rm
avg}}}$ (134) $\displaystyle=$ $\displaystyle{1\over{N_{\rm
max}}}\left[\bar{g}(c)+(c/c_{0}){\bigg{|}}{{d\bar{g}(c)}\over{d(c/c_{0})}}{\bigg{|}}^{2}\right],$
(135)

where the natural units of concentration are $c_{0}=N_{\rm max}/Da\tau_{\rm
avg}$. From the numbers above, we see that real transcription factor
concentrations are comparable to $c_{0}$, which is part of why the
optimization of information transmission leads to interesting results.

Now we can make use of results from the previous lecture, §3.1. First we
propagate the output noise $\sigma_{g}$ back through the input/output
relation, as in Fig 17, to obtain an effective input noise

$\sigma_{c}^{\rm
eff}={\bigg{|}}{{d\bar{g}(c)}\over{dc}}{\bigg{|}}^{-1}\sigma_{g}(c)={{c_{0}}\over{h\sqrt{N_{\rm
max}}}}{{(c/c_{0})}\over{\bar{g}(c)[1-\bar{g}(c)]}}\left[\bar{g}(c)+{{h^{2}}\over{(c/c_{0})}}\bar{g}(c)[1-\bar{g}(c)]\right]^{1/2}.$
(136)

Then we can work in the small noise limit to estimate the maximum information,
from Eqs (60) and (61),

$I_{\rm max}=\log_{2}\left[{1\over\sqrt{2\pi e}}\int_{0}^{c_{\rm
max}}{{dc}\over{\sigma_{c}^{\rm eff}}}\right]\,\,{\rm bits},$ (137)

where we go back to conventional units and note explicitly that the
transcription factor has some maximum concentration $c_{\rm max}$. Playing
with this a bit one can see that

$I_{\rm max}={1\over 2}\log_{2}\left[{{N_{\rm max}}\over\sqrt{2\pi
e}}\right]+F(h,K/c_{0};c_{\rm max}/c_{0}).$ (138)

Thus we can optimize the two parameters $h$ and $K$, and the answer will
depend on the maximum concentration $c_{\rm max}$; again it is natural to
express concentrations in units of $c_{0}$.

Figure 24: Optimizing information about the concentration of a single
transcription factor [200]. (A) A map of $F$ from Eq (138) as a function of
$K$ and $h$, showing a single clear optimum at reasonable parameters ($c_{\rm
max}/c_{0}=1$). (B–F) Optimal input/output relations $\bar{g}_{\rm i}(c)$ with
five target genes, with increasing values of $c_{\rm max}/c_{0}$ indicated by
dashed vertical lines. At smaller $c_{\rm max}/c_{0}$ the optimal solutions
are redundant, so fewer distinct input/output relations are visible. (G)
Optimal values of $K/c_{0}$, showing a sequence of bifurcations as $c_{\rm
max}/c_{0}$ increases and responses become distinct.

Figure 24A shows the results of this optimization at $c_{\rm max}/c_{0}=1$
[200]. Perhaps the most important conclusion is that the optimal parameters
are perfectly sensible, not driven off to extreme values. This happens because
of the interplay between the two components of the noise, counting at the
output and Berg–Purcell at the input.

A natural generalization is to the case where the single transcription factor
controls multiple genes, at expression levels
$\\{g_{1},\,g_{2},\,\cdots,\,g_{K}\\}$, but to keep things simple these target
genes do not interact. In Figure 24B–G we see the results when there are $K=5$
targets and $c_{\rm max}/c_{0}$ changes across a dynamic range of $30\times$.
At small values of $c_{\rm max}/c_{0}$ the optimal solution is for all $K$
genes to have the same values of $K$ and $h$, so that they are completely
redundant copies of one another (Fig 24B). I have always found this result
fascinating because redundancy often is taken as prima facie evidence against
any information theoretic optimization principle. Here we see that redundancy
actually is the result of such a principle, in the right regime. I don’t know
whether this is a path to understanding the appearance of redundancy in
biological signaling more broadly, but it surely is an object lesson.

As the maximum allowed concentration of the input transcription factor
increases, the optimal strategy for information transmission changes and more
of the target genes come to have distinguishable responses (Fig 24B–F). This
happens through a series of bifurcations that we can visualize in a plot of
$K/c_{0}$ vs $c_{\rm max}/c_{0}$ (Fig 24G). Successive bifurcations add
distinct target gene responses at higher concentration (larger $K$) and with
steeper responses (larger $h$), until the set of input/output relations “tile”
the full dynamic range of inputs.

The results of Fig 24 are just in the case where a single transcription factor
activates a set of non–interacting target genes. I should admit that when we
started thinking about these problems my colleagues and I thought that the
path from these simple examples to something realistic would be quick, but we
were wrong. We did learn about some pieces of the problem.

To begin, the pattern of responses from multiple targets in Fig 24 is
redundant, because activation of the genes with large $K$ allows us to infer
that the genes with smaller $K$ also are active. This redundancy can be
reduced, and information transmission enhanced, by repressive interactions
among the targets, as shown in Fig 25 [201]. This parallels the discussion of
lateral inhibition in the retina, outlined in §4.1, and produces profiles of
(mean) expression level vs input concentration that remind us of those shown
by the gap genes. These benefits of repression in enhancing efficiency are
seen even when there are no feedback loops.

Figure 25: Redundancy is reduced in a system where a single transcription
factor (concentration $c$) activates multiple genes $\\{g_{1},g_{2},g_{3}\\}$
that repress one another, as shown at left. At right are the mean expression
levels that optimize information transmission, with $c_{\rm max}/c_{0}=10$.
Redrawn from Ref [201].

The prototypical example of feedback is when a single target gene also can
activate itself [202]. This system has a bistable regime at sufficiently
strong self–activation, but the optimal parameter values are on the monostable
side of this bifurcation. As the bifurcation is approached, however, there is
critical slowing down. This emergent long time scale serves to reduce noise,
but near criticality there is also a strong path for noise in the output to be
injected back into the system. The closeness of the optimum to the critical
point thus depends on the maximal concentration of output molecules $N_{\rm
max}$.

In many systems, including the fly embryo, multiple cells or nuclei can
exchange proteins or mRNA through diffusion. This generates a spatial
averaging that can reduce noise, adding to the effects of temporal
averaging.383838This combination of spatial and temporal averaging seems to be
necessary to understand how high levels of noise in the initiation of
transcription yield low levels of noise in the concentration of the gap gene
proteins [203]. Importantly, because diffusion itself is noisy, this effect
can’t push the final noise level below the Poisson level of counting noise. In
the embryo, meaningful variations in input and output are laid out in space,
and of course sufficiently strong diffusion will degrade these patterns. The
result is that there is an optimal diffusion constant that maximizes
information transmission [204].

Figure 26: Schematic of direct transcriptional regulation (DTR) and indirect
translational regulation (ITR) [205]. (A) In DTR, as above, activator TFs
(green squares at concentration $c$) interact with binding sites along the DNA
to activate expression of the regulated gene $g$. (B) In ITR scenario, input
molecules (green squares) bind to mRNAs $m$ of protein $y$ (red chain) to make
the mRNA inaccessible for translation (gray oval). Translation proceeds from
unbound mRNA molecules, giving rise to proteins $y$ (red stars). These
proteins act as repressors for gene $g$; the overall mapping from
$c\rightarrow g$ is thus activating in both scenarios.

In Figures 24B–F we see that even as the different target genes acquire
distinct input/output relations, the optimal solutions still do not make much
use of the lowest accessible concentrations; this regime is simply too noisy
to allow reliable information transmission. But suppose that the cell makes
many mRNA molecules of some intermediate protein $y$, and that the protein $c$
binds to these mRNA and represses translation, as in Fig 26; $y$ can then act
indirectly as a transcriptional repressor so that the whole path $c\rightarrow
g$ is activating. Then the “measurement” of low concentrations is made in
parallel at many sites, rather than at just one site along the DNA, and the
response is largest and most reliable at the lowest concentrations. This
suggests that a molecule which functions both as a transcriptional activator
and a translational repressor could make better use of its full dynamic range.
It requires a careful calculation to show that information transmission really
can be larger even when we constrain the total number of molecules, but it
does [205]. One of the primary maternal morphogens in the fly embryo belongs
to a whole class of proteins that function as such dual
transcription/translation regulators [206, 207, 208, 209], and our best
estimates put this system in the regime where dual regulation enhances
information transmission. I found it remarkable that this almost baroque level
of biological complexity can be derived as part of the solution to a
fundamental physics problem faced by the cell.

Each of these pieces—repression to reduce redundancy, feedback to average over
time, diffusion to average over space, and dual regulations—is an interesting
problem by itself, and it was possible to make progress with a combination of
analytic and (modest) numerical approaches. But all of these things are
happening at once in the fly embryo, even just in the gap gene network. To put
(most of) these pieces together requires a more sophisticated approach [210].

###  4.3 The gap genes, once more

Let’s plunge right in. We have a collection of nuclei labeled ${\rm
n}=1,\,2,\,\cdots,\,N$ along the anterior–posterior axis of the embryo, with a
distance $\Delta$ from one to the next. Associated with each nucleus are the
expression levels of the four gap genes $\\{g_{\rm i}({\rm n})\\}$, and these
molecules can diffuse between neighbors with an effective diffusion constant
$D$. As before we will keep track of one concentration for each species, not
worrying about the separate dynamics of mRNA and proteins. We will assume that
each gap gene product has the same maximal synthesis rate $r_{\rm max}$ and
the same decay time $\tau$; these are simplifications but also supported by
experiment. The gap genes are driven by maternal inputs $\\{c_{\alpha}({\rm
n})\\}$ with $\alpha=1,\,2,\,3$, and we will assume that the maternal inputs
are constant in time, which we know to be correct over the relevant time
window for at least one of them [64]. With these assumptions, the dynamics of
the gap gene expression levels obey a generalization of the first equation
that we wrote down in these lectures [Eq (1)]:

$\displaystyle{{dg_{\rm i}({\rm n},t)}\over{dt}}$ $\displaystyle=$
$\displaystyle r_{\rm max}f_{\rm i}\left(\\{g_{\rm j}({\rm
n},t)\\};\\{c_{\alpha}({\rm n})\\}\right)-{1\over\tau}g_{\rm i}({\rm n},t)$
(139) $\displaystyle\,\,\,\,\,\,\,\,\,\,+{D\over{\Delta^{2}}}\left[g_{\rm
i}({\rm n}+1,t)-2g_{\rm i}({\rm n},t)+g_{\rm i}({\rm n}-1,t)\right]+\eta_{\rm
i}({\rm n},t),$

where $f_{\rm i}$ is a separate regulation function for each gap gene, and
$\eta_{\rm i}({\rm n},t)$ are noise terms. Three static inputs are driving
four interacting genes, as in Fig 27, so there are $(3\times 4)+(4\times
4)=28$ arrows, as noted in §1.1.

Figure 27: The network of four gap genes $\\{g_{\rm i}\\}$ driven by three
maternal inputs $\\{c_{\alpha}\\}$. Regulatory interactions
$c_{\alpha}\rightarrow g_{\rm i}$ are strictly feedforward (blue arrows). In
contrast, interactions $g_{\rm i}\rightarrow g_{\rm j}$ can form feeback
loops. It is thought that $c_{\alpha}\rightarrow g_{\rm i}$ are largely
activating while $g_{\rm i}\rightarrow g_{{\rm j}\neq{\rm i}}$ are largely
repressive (black blunt arrows), although there is evidence for
self–activation $g_{\rm i}\rightarrow g_{\rm i}$ (curved black arrows). The
schematic includes this consensus, which proves to be a feature of the
optimized network, though it is not imposed.

We take the maternal inputs $\\{c_{\alpha}({\rm n})\\}$ as known, and fixed as
we try to optimize the gap gene network itself. This certainly is fair for the
input which is large at the anterior end, where we have accurate measurements
in both live and fixed embryos establishing the peak absolute concentration
and the approximate exponential decay with distance into the egg. We will
assume that the input which is large at the posterior end is just a mirror
image, and that the terminal inputs are symmetric with the same peak
concentration but more rapid decays.

In order to proceed we need a model for the regulation functions. In
particular we need to describe what happens as multiple regulatory arrows
converge on a single target gene. We have taken a simple view inspired by
allostery in proteins [211, 212, 213, 214]. In a single large protein
molecule, binding of a small molecule at one point on the surface can
influence the binding of molecules far away; a classic example is the
cooperative binding of four oxygen molecules to hemoglobin in our blood.
Monod, Wyman, and Changeux (MWC) proposed that this happens because the
protein can exist in two structures, and the small molecules have different
binding energies to these two structures. In this picture there is no direct
interaction between the binding events; all interactions are mediated through
the protein [215]. In the simplest formulation these events occur at thermal
equilibrium, even though one often is describing the activity of enzymes, ion
channels, and other systems that evidently are not in equilibrium. The idea is
that the rate of the events that we are interested in is proportional to the
occupancy of some state(s), and this occupancy is well approximated by
estimates from equilibrium statistical mechanics. There is a tradition of
using such equilibrium arguments for transcriptional control as well [216,
217], although the validity of this quasi–equilibrium view remains an open
question [218].

Figure 28: The Monod–Wyman–Changeux (MWC) model for regulation. Here there are
two binding sites for the (blue) transcription factor (TF) protein, at
concentration $c$. Binding at the two sites are independent events, but the
binding constants $K_{\rm on}$ and $K_{\rm off}$ are different in the two
states. If binding is stronger in the on state ($K_{\rm on}<K_{\rm off}$) then
the free energy difference between off and on states increases as TFs bind.

Figure 28 schematizes the MWC model for regulation of a gene by a single
transcription factor (TF). The system can be in “on” or “off” states, with
transcription occurring in the on state, and there are two binding sites for
the TF, which is at concentration $c$. Assuming the system is in equilibrium
we can write the probability of being in the on state in terms of the free
energy differences as shown, to give

$\displaystyle P_{\rm on}$ $\displaystyle=$
$\displaystyle{{e^{-F_{0}/k_{B}T}(1+c/K_{\rm on})^{2}}\over{(1+c/K_{\rm
off})^{2}+e^{-F_{0}/k_{B}T}(1+c/K_{\rm on})^{2}}}$ (140) $\displaystyle=$
$\displaystyle{1\over{1+\exp[-{\cal F}(c)]}}$ (141) $\displaystyle{\cal F}(c)$
$\displaystyle=$ $\displaystyle-{{F_{0}}\over{k_{B}T}}+2\ln\left({{1+c/K_{\rm
on}}\over{1+c/K_{\rm off}}}\right).$ (142)

Notice that if binding is very tight in the on state and very weak in the off
state, we should have $K_{\rm on}\ll c\ll K_{\rm off}$ and we can rewrite

$P_{\rm on}={{c^{2}}\over{c_{1/2}^{2}+c^{2}}},$ (143)

with $c_{1/2}=K_{\rm on}e^{F_{0}/2k_{B}T}$. We identify the probability of
being in the on state with the regulation function $f$. Notice that factor of
$2$ in Eq (142) is counting the two binding sites.

One attractive feature of the MWC model is that it generalizes to having
multiple TFs converge to regulate a single gene. Consider that gene $\rm i$
has $H_{\rm ij}$ binding sites for the protein encoded by gene $\rm j$ and
$H_{{\rm i}\alpha}$ sites for the maternal input $\alpha$, Then if we follow
the same equilibrium statistical mechanics arguments as above, we will find

$\displaystyle f_{\rm i}\left(\\{g_{\rm j}\\};\\{c_{\alpha}\\}\right)$
$\displaystyle=$ $\displaystyle{1\over{1+\exp\left[-{\cal
F}_{i}\left(\\{g_{\rm j}\\};\\{c_{\alpha}\\}\right)\right]}}$ (144)
$\displaystyle{\cal F}_{\rm i}\left(\\{g_{\rm j}\\};\\{c_{\alpha}\\}\right)$
$\displaystyle=$ $\displaystyle-{{F_{{\rm i}0}}\over{k_{B}T}}+\sum_{\rm
j}H_{\rm ij}\ln\left(1+{{g_{\rm j}}\over{K_{\rm
ij}}}\right)+\sum_{\alpha}H_{{\rm
i}\alpha}\ln\left(1+{{c_{\alpha}}\over{K_{{\rm i}\alpha}}}\right),$ (145)

where we work in the approximation that one of the two binding constants is
large (very weak binding). Note that by changing the sign of $H$ we can have
TFs act as both activators or repressors. This gives us a parameterization for
the regulation functions in Eq (139), and as promised in the first lecture we
have two parameters $K_{\rm ij}$ and $H_{\rm ij}$ for each pair of species
that can interact.

The noise in Eq (139) has contributions from the input (Berg–Purcell) and
output (counting) noise as in Eq (135).393939To be fully realistic we can add
a small extra noise with constant fractional variance, which seems to be
necessary to match the data [210, 219]. There is also noise attached to the
diffusion terms, and one has to be careful that there is independent noise in
the fluxes ${\rm n}\rightarrow{\rm n}\pm 1$, not independent noise added to
each site; this insures that diffusion noise does not violate conservation of
molecules. All of these noise sources are white, as can be seen for example
from the fact that variances in Eq (135) are inversely proportional to
averaging times. Spectral densities are set by the absolute numbers or
concentrations of molecules, as above, and these are known quite well from
experiment; for details see Ref [210].

Now that we have all the ingredients in Eq (139), we could just simulate. But
our goal is to optimize the parameters so as to maximize the information that
expression levels $\\{g_{\rm i}\\}$ provide about position or cellular
identity $\rm n$. It would not be enough to have a single solution of these
stochastic differential equations, we need to know about the whole
distribution of solutions. This quickly becomes intractable. Fortunately we
know that in the real system noise levels are small and fluctuations are
approximately Gaussian. This means that we can linearize Eq (139) in the small
fluctuations around the mean, and find closed equations for the time evolution
of the means $\langle g_{\rm i}({\rm n},t)\rangle$ and the covariance matrix
$\langle\delta g_{\rm i}({\rm n},t)\delta g_{\rm j}({\rm n},t)\rangle$. Thus
at a single setting of all the parameters $({\bf H},{\bf K})$ we can do one
(large) integration forward in time and find everything we need in order to
evaluate the positional information at $t\sim 45\,{\rm min}$ into nuclear
cycle 14. We explore the 50+ dimensional parameter space by a version of
simulated annealing [210].

The essential results of the optimization process are shown in Fig 29. We
start with some random setting of all the adjustable parameters, and typically
this will leave some of the gap genes fully on and some fully off, uniformly
across the entire embryo, so that there is zero positional information (point
1 in Fig 29). As we explore and anneal we find parameter settings that allow
first one then two, three, and all four gaps genes to be driven on and off by
the full dynamic range of the maternal inputs (points 2, 3, and 4). Much of
the time spent in optimization is required to converge from these patterns
onto something richer and more informative, finally arriving at an optimum
(point 5). This optimal network has spatial patterns of gene expression very
similar to those of the real network, and the absolute magnitude of the
positional information is close as well.

Figure 29: Maximizing positional information in the gap gene network [210].
Random initial parameter settings produce zero positional information, with
all gap genes fully on or off across the length of the embryo. Across $\sim
20,000$ steps of optimization we find parameter settings that allow first one
then two, three, and finally all four gap genes to be modulated by the dynamic
range of maternal inputs ($1\rightarrow 2\rightarrow 3\rightarrow 4$). The
majority of the optimization run then tunes the parameters to maximize the
positional information, arriving at a setting ($5$) that behaves very
similarly to the real network.

I want to emphasize that in deriving the patterns of gap gene expression that
we see in the upper right of Fig 29, there are no free parameters—all are
determined by the optimization principle. In fact if we set this principle
aside and try to fit the model to the mean profiles, we don’t do any better in
matching the data. More subtly, these best fits convey substantially less than
the maximum information because they correspond to parameter settings with
excess noise. Thus in some way the optimization principle is getting us closer
to the real network than standard model fitting. Also, the small differences
between the optimum and the real network must reflect a limitation in the
class of models that we are considering, not a failure of optimization. There
are a few more interesting features of the solution(s):

  * •

The positional information seen in real networks really is at the edge of what
this class of models can support, as seen by random sampling of the parameter
space.

  * •

Almost all of the possible interactions are realized in the optimal solution,
and their signs agree with available data.

  * •

We have constrained the maximum molecule counts and concentrations; other
local optima differ in the mean utilization of these resources.

  * •

We read out positional information at a moment in nuclear cycle 14 where the
real expression levels are maximally informative, and do not constrain the
dynamics, but the optimal networks have very slowly changing patterns, as with
the real system.

We can also think of this as a “laboratory” in which to test alternative
scenarios, making a change and then re–optimizing all the remaining
parameters:

  * •

We have taken the diffusion constant $D$ in Eq (139) as known, but there is a
broad maximum of information near this measured value.

  * •

If we eliminate feedback loops we can still reach similar levels of positional
information but only with unphysically large value of the $H$ parameters;
within realistic settings feedback is essential.

  * •

If the same maximum number of molecules were spread across only three gap
genes it would be impossible to achieve the same level of positional
information; benefits of a fifth gap gene are minimal.

  * •

Although cross–regulation within the gap gene network provides some
resilience, eliminating any one of the maternal inputs results in
significantly less positional information.

I suspect we are just scratching the surface. Perhaps most intriguing
observation is that different local optima have broadly consistent gap gene
expression patterns, but with shifts and swaps similar to those seen in
related species [220, 221, 222].

##  5 Conclusion

This Summer School celebrates a special moment in the long history of
interactions between physics and biology. Just one generation back, physicists
and biologists couldn’t agree on much, but they could agree that searching for
a theoretical physics of life was a waste of time. Physicists saw biology as
too messy, and biologists saw the physicists’ search for simplicity and
universality as a poor match to the evident complexity and diversity of life.
Much has changed.

I have emphasized that enormous progress in experiment has created new
opportunities for theory. In particular, experiments have revealed that many
living systems exhibit behaviors with a precision and reproducibility far
beyond what once was imagined. In some cases this precision corresponds to
functional behavior close to the limits of what is allowed by the laws of
physics. Thus, aspects of early embryonic development can take their place
alongside classical examples such as photon counting in vision and molecule
counting in bacterial chemotaxis.

It is an old idea that evidence for near–optimal performance could be turned
around and formulated as a theoretical principle from which aspects of
mechanism and function can be derived. This idea has a checkered history. Many
people have used optimization principles in the absence of any direct evidence
for optimization, which is fine if the next experiments provide those direct
tests. More subtly, when we formulate an optimization principle we often must
search among a class of possible mechanisms in order to find the optimum, and
too often this class has been woefully oversimplified. Again these
simplifications are fine as starting points, but sometimes the more realistic
calculations never come. The message is that seemingly simple and clear
principles of physical optimization become challenging when we try to use them
in the realistic context of life’s complexity.

I don’t know if what I have described here will survive the next rounds of
experiments. But I am excited that we are implementing physical notions of
optimization in realistic settings. We have asked how the embryo can best
decode positional information contained in the actual expression levels of gap
genes, and this makes detailed predictions for the distortion of the body plan
in mutants; theory and experiment agree quantitatively, with no adjustable
parameters (§2.3). We have asked how the embryo can match the statistics of
maternal inputs to the measured signal and noise characteristics of the gap
gene network, and the surprising uniformity of positional information along
the anterior–posterior axis emerges as a prediction of this optimization; the
real embryo is within two percent of the optimum (§3.3). Finally, we have
asked how the architecture and parameters of a reasonably realistic model for
the gap gene network can be tuned to maximize information transmission, and we
find networks very much like the real network emerging as a result, with no
free parameters; this optimization principle predicts many features of the
network that we did not constrain, and provides a framework for exploring the
interplay of chance and necessity (§4.3).

I want to end by saying a few words about what we are not doing. Much of the
progress described here, and in other lectures at the Summer School, takes a
phenomenological view, hoping to say something crisp about system level
functions without digging into too much microscopic detail. There is another
community that is much more focused on molecules, and the divide between these
communities can be quite stark. To give one example, we probably know more
about the functional dynamics of individual ion channels than about any other
class of protein molecules, and we have a precise theoretical framework for
how these molecular dynamics shape the electrical dynamics of single neurons.
There are $100+$ different kinds of channels encoded in the genome, and single
neurons express a cocktail of perhaps seven different kinds. But all of this
molecular richness disappears rather abruptly once we start to talk about
coding and computation in networks of neurons.

As physicists we are used to the idea that macroscopic phenomena are described
by coarse–grained models, and that in these models many microscopic details
have disappeared (see Approach 3 in §1.2). But in the examples we understand,
from the inanimate world, we can do this coarse–graining explicitly to see
which (relevant) elements of the model survive to influence macroscopic
dynamics and which (irrelevant) details are erased. As far as I know nobody
knows how to start with a realistic description of ion channels and
coarse–grain to arrive at a model of neural networks. Further, we do know that
some molecular details must matter, because particular classes of cells in the
brain will switch from using one kind of channel to another at crucial moments
in development, and of course different types of cells use different
combinations of channels. We also know that not all macroscopic quantities are
universal, and there is the danger that what is universal might not be
relevant for the organism.

My concern about the gap between molecular and system level descriptions is
not just the vague feeling that we are missing something. In many cases,
including the fly embryo, some of the most powerful new experimental tools
allow direct and reliable manipulation at the molecular level. The parameters
of our network description, attached to the arrows in Fig 27, are themselves
encoded (in part) in the DNA sequence of transcription factor binding sites.
It now is possible to edit these sequences with single base pair precision,
but we don’t know how these manipulations relate to the circles and arrows, so
it is hard to see how our current theories connect to an exploding set of new
experiments. In truth we don’t even know if we can start with sequences and do
a systematic calculation to arrive at something like Eqs (139, 144, 145). A
more complete physics of life will build these bridges.

## Acknowledgements

Teaching in summer schools is a special pleasure, a stimulus to putting one’s
thoughts in order. My thanks to the organizers—Anne–Florence Bitbol, Thierry
Mora, Ilya Nemenman, and Aleksandra Walczak—for this opportunity, and for
years of friendship. Thanks also to the students, who provided a warm and
enthusiastic welcome. The ideas presented here owe much to many wonderful
collaborators and friends, as may be seen from the references. But this is a
school, and acknowledging these debts also is an opportunity to provide advice
to the students.

The idea that living systems approach fundamental physical limits to their
performance has fascinated me from the first time I heard about photon
counting in vision. On arriving in Groningen as a postdoctoral fellow forty
years ago (!), I had the good fortune to meet Rob de Ruyter van Steveninck,
who was in the office next door. The experiments he was doing on the fly
visual system, combining measurements in the retina with measurements on
motion–sensitive neurons deep in the brain, made it seem possible to connect
general theoretical ideas about physical limits and optimization with
detailed, quantitative experiments. We could not have predicted that those
early discussions would turn into decades of continuing collaboration and
friendship. We also have had the pleasure of being joined by many colleagues.
Lesson for theorists: talk to the experimentalist next door, it literally can
change your life.

When I moved to Princeton I wanted to take the ideas of signals, noise, and
information that Rob and I had explored in neurons down to the molecular
level; Sima Setayeshgar and I made some first theoretical efforts in this
direction. David Tank had moved at the same time, and was excited that new
fluorescence techniques might make it possible to measure signals on this
scale with enough precision to say something meaningful. Eric Wieschaus joined
our conversation, and while he was unclear on exactly what David and I wanted
to do (we were unclear as well), he explained why we should do it in flies.
Eric ended by saying that if some physics student wanted to try, we should
send the student to him to learn the relevant techniques. Thomas Gregor was
working with me on a theory project, but decided to take Eric up on his offer,
and this led to spectacular new experiments. In parallel, Curt Callan and I
were working with another student, Gašper Tkačik, on purely theoretical
questions about the optimization of information flow in transcriptional
control. Luckily, Gašper and Thomas sat in the same office, and they made many
of the crucial theory/experiment connections on their own. Again, we could not
have predicted that these early discussions would form the basis for two
decades of work, for collaborations with generations of new colleagues, and
for valued friendships. Lesson for theorists: be ready for experimental
developments which allow speculative theoretical ideas to become concrete.

Lesson from both stories: one experiment can test a theory or raise a specific
theoretical question, but sometimes theory/experiment collaboration is a long
term investment.

In addition to the handful of friends who were at the origins of these ideas,
the specific things I discussed in these lectures involved collaborations with
Naama Brenner, Curtis Callan, Julien Dubuis, Adrienne Fairhall, Dmitry Krotov,
Geoff Lewen, Geoff Owen, Mariela Petkova, Marc Potters, Fred Rieke, Shiva
Sinha, Thomas Sokolowski, and Aleksandra Walczak. This bland listing doesn’t
do justice to all the fun we have had together.

#### Funding information.

This work was supported in part by the US National Science Foundation through
the Center for the Physics of Biological Function (PHY–1734030), and by
Fellowships from the Simons Foundation and the John Simon Guggenheim Memorial
Foundation.

## References

  * [1] ER Harrison, Darkness at Night: A Riddle of the Universe (Harvard University Press, Cambridge MA, 1987).
  * [2] PJE Peebles, Principles of Physical Cosmology (Princeton University Press, Princeton NJ, 1993).
  * [3] PW Anderson, Basic Notions of Condensed Matter Physics (Benjamin/Cummings, Menlo Park CA, 1984).
  * [4] NW Timoféff–Ressovsky, KG Zimmer, and M Delbrück, Über die Natur der Genmutation und der Genstruktur Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen. Neue band 1, 189 (1935).
  * [5] PR Sloan and B Fogel, eds, Creating a Physical Biology. The Three–Man Paper and Early Molecular Biology (The University of Chicago Press, Chicago IL, 2011).
  * [6] E Schrödinger, What is Life? (Cambridge University Press, Cambridge UK, 1944).
  * [7] JD Watson, TA Baker, SP Bell, A Gann, M Levine, and R Losick, Molecular Biology of the Gene, 7th Edition (Pearson, Boston MA, 2014).
  * [8] B Alberts, A Johnson, J Lewis, M Raff, K Roberts, and P Walter, Molecular Biology of the Cell, 4th Edition (Garland Science, New York NY, 2002).
  * [9] C N usslein–Volhard and E Wieschaus, Mutations affecting segment number and polarity in Drosophila. Nature 287, 795 (1980). https://doi.org/10.1038/287795a0
  * [10] MD Petkova, G Tkačik, W Bialek, EF Wieschaus, and T Gregor, Optimal decoding of cellular identities in a genetic network. Cell 176, 844 (2019). https://doi.org/10.1016/j.cell.2019.01.007
  * [11] E Wieschaus and C Nüsslein–Volhard, The Heidelberg screen for pattern mutants of Drosophila: A personal account. Annu Rev Cell Dev Biol 32, 1 (2016). https://doi.org/10.1146/annurev-cellbio-113015-023138
  * [12] PA Lawrence, The Making of a Fly: The Genetics of Animal Design. (Blackwell, Oxford, 1992).
  * [13] RN Cahn, The eighteen parameters of the standard model in your everyday life. Rev Mod Phys 68, 952 (1996). https://doi.org/10.1103/RevModPhys.68.951
  * [14] J Bardeen, LN Cooper, and JR Schrieffer, Theory of superconductivity. Phys Rev 108, 1175 (1957). https://doi.org/10.1103/PhysRev.108.1175
  * [15] KG Wilson, The renormalization group: Critical phenomena and the Kondo problem. Rev Mod Phys 47, 773 (1975). https://doi.org/10.1103/RevModPhys.47.773
  * [16] G Parisi, Statistical Field Theory (Addison–Wesley, Redwood City CA, 1988).
  * [17] J Zinn–Justin, Quantum Field Theory and Critical Phenomena (Oxford University Press, Oxford UK and New York NY, 1989).
  * [18] J Cardy, Scaling and Renormalization in Statistical Physics (Cambridge University Press, Cambridge, 1996).
  * [19] RB Laughlin, Anomalous quantum Hall effect: An incompressible quantum fluid with fractionally charged excitations. Phys Rev Lett 50, 1395 (1983). https://doi.org/10.1103/PhysRevLett.50.1395
  * [20] DJ Gross and F Wilczek, Ultraviolet behavior of non–Abelian gauge theories. Phys Rev Lett 30, 1343 (1973). https://doi.org/10.1103/PhysRevLett.30.1343
  * [21] HD Politzer, Reliable perturbative results for strong interactions? Phys Rev Lett 30, 1346 (1973). https://doi.org/10.1103/PhysRevLett.30.1346
  * [22] CG Callan, Jr and DJ Gross, Bjorken scaling in quantum field theory. Phys Rev D 8, 4383 (1973). https://doi.org/10.1103/PhysRevD.8.4383
  * [23] P Newman, Deep inelastic lepton–nucleon scattering at HERA. Int J Mod Phys A 19, 1061 (2004). https://doi.org/10.1142/S0217751X0401897X
  * [24] Y LeCun, Y Bengio, and G Hinton, Deep learning. Nature 521, 436 (2015). https://doi.org/10.1038/nature14539
  * [25] A Vaswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, AN Gomez, Ł Kaiser, I Polosukhin, Attention is all you need. In Advances in Neural Information Processing Systems 30, I Guyon et al, eds (Curran Associates, 2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [26] TB Brown, J Kaplan, G Sastry, B Mann, N Ryder, P Dhariwal, A Neelakantan, A Askell, S Agarwal, M Subbiah, P Shyam, A Herbert–Voss, G Krueger, DM Ziegler, T Henighan, R Child, J Wu, A Ramesh, C Winter, C Hesse, M Chen, B Chess, E Sigler, J Clark, M Litwin, S Gray, C Berner, S McCandlish, A Radford, I Sutskever, and D Amodei, Language models are few–shot learners. Advances in Neural Information Processing Systems 33, 1877 (2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * [27] HD Block, The perceptron: A model for brain functioning. I. Rev Mod Phys 34, 123 (1962). https://doi.org/10.1103/RevModPhys.34.123
  * [28] HD Block, BW Knight, Jr, and F Rosenblatt, Analysis of a four-layer series-coupled perceptron. II. Rev Mod Phys 34, 135 (1962). https://doi.org/10.1103/RevModPhys.34.135
  * [29] JJ Hopfield, Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci (USA) 79, 2554 (1982). https://doi.org/10.1073/pnas.79.8.2554
  * [30] DJ Amit, Modeling Brain Function: The World of Attractor Neural Networks (Cambridge University Press, Cambridge UK, 1989).
  * [31] J Hertz, A Krogh, and RG Palmer, Introduction to the Theory of Neural Computation (Addison Wesley, Redwood City CA, 1991).
  * [32] P Mehta and DJ Schwab, An exact mapping between the variational renormalization group and deep learning. arXiv:1410.3831 [stat.ML] (2014). https://doi.org/10.48550/arXiv.1410.3831
  * [33] P Mehta, M Bukov, C–H Wang, AGR Day, C Richardson, CK Fisher, and DJ Schwab, A high-bias, low-variance introduction to machine learning for physicists. Phys Repts 810, 1–124 (2019). https://doi.org/10.1016/j.physrep.2019.03.001
  * [34] G Carleo, I Cirac, K Cranmer, L Daudet, M Schuld, N Tishby, L Vogt–Maranto, and L Zdeborová, Machine learning and the physical sciences. Rev Mod Phys 91, 045002 (2019). https://doi.org/10.1103/RevModPhys.91.045002
  * [35] J Kaplan, S McCandlish, T Henighan, TB Brown, B Chess, R Child, S Gray, A Radford, J Wu, and D Amodei, Scaling laws for neural language models. arXiv:2001.08361 [cs.LG] (2020). https://doi.org/10.48550/arXiv.2001.08361
  * [36] DA Roberts and S Yaida, with B Hanin, The Principles of Deep Learning Theory (Cambridge University Press, Cambridge UK, 2022). See also arXiv:2106.10165 [cs.LG] (2021). https://doi.org/10.1017/9781009023405
  * [37] L Peliti, ed, Biologically Inspired Physics: Proceedings of a NATO Workshop, Cargèse 1990 (Plenum Press, New York, 1991).
  * [38] D Nelson, T Piran, and S Weinberg, eds, Statistical Mechanics of Membranes and Surfaces (World Scientific, Singapore, 1989).
  * [39] S Ramaswamy, Active matter. J Stat Mech (2017) 054002. https://doi.org/10.1088/1742-5468/aa6bc5
  * [40] MM Desai, DS Fisher, and AW Murray, The speed of evolution and maintenance of variation in asexual populations. Curr Biol 17, 385 (2007). https://doi.org/10.1016/j.cub.2007.01.072
  * [41] DS Fisher, Leading the dog of selection by its mutational nose. Proc Natal Acad Sci (USA) 108, 2633 (2011). https://doi.org/10.1073/pnas.1100339108
  * [42] RA Neher and O Hallatschek, Genealogies of rapidly adapting populations. Proc Natal Acad Sci (USA) 110, 437 (2012). https://doi.org/10.1073/pnas.1213113110
  * [43] MM Desai, AM Walczak, and DS Fisher, Genetic diversity and the structure of Ggenealogies in rapidly adapting populations. Genetics 193, 565 (2013). https://doi.org/10.1534/genetics.112.147157
  * [44] DS Fisher, Asexual evolution waves: Fluctuations and universality. J Stat Mech P01011 (2013). https://doi.org/10.1088/1742-5468/2013/01/P01011
  * [45] S Levy, J Blundell, S Venkataram, DA Petrov, DS Fisher, and G Sherlock, Quantitative evolutionary dynamics using high-resolution lineage tracking. Nature 519, 181 (2015). https://doi.org/10.1038/nature14279
  * [46] BH Good, MJ McDonald, JE Barrick, RE Lenski, and MM Desai, The dynamics of molecular evolution over 60,000 generations. Nature 551, 45 (2017). https://doi.org/10.1038/nature24287
  * [47] AN Nguyen Ba, I Cvijović, JI Rojas Echenique, KR Lawrence, A Rego–Costa, X Liu, SF Levy, and MM Desai, High–resolution lineage tracking reveals travelling wave of adaptation in laboratory yeast. Nature 575, 494 (2019). https://doi.org/10.1038/s41586-019-1749-3
  * [48] M Łuksza and M L assig, A predictive fitness model for influenza. Nature 507, 57 (2014). https://doi.org/10.1038/nature13087
  * [49] RA Neher, T Bedford, RS Daniels, CA Russell, and BI Shraiman, Prediction, dynamics, and visualization of antigenic phenotypes of seasonal influenza viruses. Proc Natl Acad Sci (USA) 113, E1701 (2016). https://doi.org/10.1073/pnas.1525578113
  * [50] F Zanini, J Brodin, L Thebo, C Lanz, G Bratt, J Albert, and RA Neher, Population genomics of intrapatient HIV-1 evolution. eLife 4, e11282 (2016). http://dx.doi.org/10.7554/eLife.11282.001
  * [51] MK Transtrum, BB Machta, KS Brown, BC Daniels, CR Myers, and JP Sethna, Perspective: Sloppiness and emergent theories in physics, biology, and beyond. J Chem Phys 143, 010901 (2015). https://doi.org/10.1063/1.4923066
  * [52] KS Brown, CC Hill, GA Calero, CR Myers, KH Less, JP Sethna, and RA Crione, The statistical mechanics of complex signaling networks: nerve growth factor signaling. Phys Biol 1, 184 (2004). https://doi.org/10.1088/1478-3967/1/3/006
  * [53] DA Rand, A Raju, M Sáez, F Corson, and ED Siggia, Geometry of gene regulatory dynamics. Proc Natl Acad Sci (USA) 118, e2109729118 (2021). https://doi.org/10.1073/pnas.2109729118
  * [54] S Hecht, S Shlaer, and MH Pirenne, Energy, quanta and vision. J Gen Physiol 25, 819 (1942). https://doi.org/10.1085/jgp.25.6.819
  * [55] HB Barlow, The size of ommatidia in apposition eyes. J Exp Biol 29, 667 (1952). https://doi.org/10.1242/jeb.29.4.667
  * [56] HC Berg and EM Purcell, Physics of chemoreception. Biohys J 20, 193 (1977). https://doi.org/10.1016/S0006-3495(77)85544-6
  * [57] W Bialek, Biophysics: Searching for Principles (Princeton University Press, Princeton, 2012).
  * [58] M Nikolić, V Antonetti, F Liu, G Muhaxheri, MD Petkova, M Scheeler, EM Smith, W Bialek, and T Gregor. Scale invariance in early embryonic development. arXiv:2312.17684 [q–bio.MN] (2023). https://doi.org/10.48550/arXiv.2312.17684
  * [59] O Shimomura, FH Johnson, and Y Saiga, Extraction, purification and properties of aequorin, a bioluminescent protein from the luminous hydromedusan, Aequorea. J Cell Comp Physiol 59, 223 (1962). https://doi.org/10.1002/jcp.1030590302
  * [60] FH Johnson, O Shimomura, Y Saiga, LC Gershman, GT Reynolds, and JR Waters Jr, Quantum efficiency of Cypridina luminescence, with a note on that of Aequorea. J Cell Comp Physiol 60, 85 (1962). https://doi.org/10.1002/jcp.1030600111
  * [61] DC Prasher, VK Eckenrode, WW Ward, FG Prendergast, and MJ Cormier, Primary structure of the Aequorea victoria green fluorescent protein. Gene 111, 229 (1992). https://doi.org/10.1016/0378-1119(92)90691-H
  * [62] M Chalfie, Y Tu, G Euskirchen, WW Ward, and DC Prasher, Green fluorescent protein as a marker for gene expression. Science 263, 802 (1994). https://doi.org/10.1126/science.8303295
  * [63] RY Tsien, Constructing and exploiting the fluorescent protein paintbox. Angew Chem Int Ed 48, 5612 (2009). https://doi.org/10.1002/anie.200901916
  * [64] T Gregor, EF Wieschaus, AP McGregor, W Bialek, and DW Tank, Stability and nuclear dynamics of the Bicoid morphogen gradient. Cell 130, 141 (2007). https://doi.org/10.1016/j.cell.2007.05.026
  * [65] JO Dubuis, R Samanta, and T Gregor, Accurate measurements of dynamics and reproducibility in small genetic networks. Mol Sys Bio 9, 639 (2013). https://doi.org/10.1038/msb.2012.72
  * [66] SC Little and T Gregor, Single mRNA molecule detection in Drosophila. In RNA Detection. Methods in Molecular Biology, vol 1649, I Gaspar, ed, pp 127–142 (Humana Press, New York, NY, 2018)
  * [67] E Lubeck and L Cai, Single–cell systems biology by super–resolution imaging and combinatorial labeling. Nat Methods 9, 743 (2012). https://doi.org/10.1038/nmeth.2069
  * [68] KH Chen, AN Boettiger, JR Moffitt, SS Wang, and X Zhuang, Spatially resolved, highly multiplexed RNA profiling in single cells. Science 348, aaa6090 (2015). https://doi.org/10.1126/science.aaa6090
  * [69] B Zoller, SC Little, and T Gregor, Diverse spatial expression patterns emerge from unified kinetics of transcriptional bursting. Cell 175, 835 (2018). https://doi.org/10.1016/j.cell.2018.09.056
  * [70] DR Larson, D Zenklusen, B Wu, JA Chao, and RH Singer, Real-time observation of transcription initiation and elongation on an endogenous yeast gene. Science 332, 475 (2011). https://doi.org/10.1126/science.1202142
  * [71] T Lucas, T Ferraro, B Roelens, JDLH Chanes, AM Walczak, M Coppey, and N Dostatni, Live imaging of Bicoid-dependent transcription in Drosophila embryos. Curr Biol 23, 2135 (2013). http://dx.doi.org/10.1016/j.cub.2013.08.053
  * [72] HG Garcia, M Tikhonov, A Lin, and T Gregor, Quantitative imaging of transcription in living Drosophila embryos links polymerase activity to patterning. Curr Biol 23, 2140 (2013). https://doi.org/10.1016/j.cub.2013.08.054
  * [73] P–T Chen, B Zoller, M Levo, and T Gregor, Gene activity as the predictive indicator for transcriptional bursting dynamics. arXiv:2304.08770 [q–bio.MN] (2023). https://doi.org/10.48550/arXiv.2304.08770
  * [74] H Chen, M Levo, L Barinov, M Fujioka, JB Jaynes, and T Gregor, Dynamic interplay between enhancer–promoter topology and gene activity. Nat Genet 50, 1296 (2018). https://doi.org/10.1038/s41588-018-0175-z
  * [75] L Barinov, S Ryabichko, W Bialek, and T Gregor, Transcription-dependent spatial organization of a gene locus. arXiv:2012.15819 [q–bio.MN] (2020). https://doi.org/10.48550/arXiv.2012.15819
  * [76] ED Adrian, The Basis of Sensation: The Action of the Sense Organs. (WW Norton, New York, 1928).
  * [77] AL Hodgkin and AF Huxley, A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol (Lond) 117, 500 (1952). https://doi.org/10.1113/jphysiol.1952.sp004764
  * [78] DJ Aidley, The Physiology of Excitable Cells, Fourth Edition. (Cambridge University Press, Cambridge, 1998).
  * [79] F Rieke, D Warland, R de Ruyter van Steveninck, and W Bialek, Spikes: Exploring the Neural Code. (MIT Press, Cambridge, 1997).
  * [80] R Segev, J Goodhouse, J Puchalla, and MJ Berry II, Recording spikes from a large fraction of the ganglion cells in a retinal patch. Nat Neurosci 7, 1154 (2004). https://doi.org/10.1038/nn1323
  * [81] AM Litke, N Bezayiff, EJ Chichilnisky, W Cunningham, W Dabrowski, AA Grillo, M Grivich, P Grybos, P Hottowy, S Kachiguine, RS Kalmar, K Mathieson, D Petrusca, M Rahman, and A Sher, What does the eye tell the brain?: Development of a system for the large-scale recording of retinal output activity. IEEE Trans Nucl Sci 51, 1434 (2004). https://doi.org/10.1109/TNS.2004.832706
  * [82] O Marre, D Amodei, N Deshmukh, K Sadeghi, F Soo, TE Holy, and MJ Berry II, Mapping a complete neural population in the retina. J Neurosci 32, 14859 (2012). https://doi.org/10.1523/JNEUROSCI.0723-12.2012
  * [83] PK Campbell, KE Jones, RJ Huber, KW Horch, and RA Normann, A silicon-based, three-dimensional neural interface: Manufacturing processes for an intracortical electrode array. IEEE Trans Biomed Eng 38, 758 (1991). https://doi.org/10.1109/10.83588
  * [84] JJ Jun, NA Steinmetz, JH Siegle, DJ Denman, M Bauza, B Barbarits, AK Lee, CA Anastassiou, A Andrei, Ç Aydin, M Barbic, TJ Blanche, V Bonin, J Couto, B Dutta, SL Gratiy, DA Gutnisky, M Häusser, B Karsh, P Ledochowitsch, CM Lopez, C Mitelut, S Musa, M Okun, M Pachitariu, J Putzeys, PD Rich, C Rossant, W Sun, K Svoboda, M Carandini, KD Harris, C Koch, J O’Keefe, and TD Harris, Fully integrated silicon probes for high–density recording of neural activity. Nature 551, 232 (2017). https://doi.org/10.1038/nature24636
  * [85] JE Chung, HR Joo, JL Fan, DF Liu, AH Barnett, S Chen, C Geaghan–Breiner, MP Karlsson, M Karlsson, KY Lee, H Liang, JF Magland, JA Pebbles, AC Tooker, LF Greengard, VM Tolosa, and LM Frank, High–density, long–lasting, and multi–region electrophysiological recordings using polymer electrode arrays. Neuron 101, 21 (2019). https://doi.org/10.1016/j.neuron.2018.11.002
  * [86] Y Gong, The evolving capabilities of rhodopsin-based genetically encoded voltage indicators. Curr Opin Chem Biol 27, 84 (2015). https://doi.org/10.1016/j.cbpa.2015.05.006
  * [87] HH Yang and F St–Pierre, Genetically encoded voltage indicators: Opportunities and challenges. J Neurosci 36, 9977 (2016). https://doi.org/10.1523/JNEUROSCI.1095-16.2016
  * [88] J Platisa, H Zeng, L Madisen, LB Cohen, VA Pieribone, and DA Storace, Voltage imaging in the olfactory bulb using transgenic mouse lines expressing the genetically encoded voltage indicator ArcLight. Sci Rep 12, 1875 (2022). https://doi.org/10.1038/s41598-021-04482-3
  * [89] SW Evans, D–Q Shi, M Chavarha, MH Pitt, J Taxidis, B Madruga, JL Fan, F–J Hwang, SC van Keulen, C–M Suomivuori, MM Pang, S Su, S Lee, YA Hao, G Zhang, D Jiang, L Pradhan, RH Roth, Y Liu, CC Dorian, AL Reese, A Negrean, A Losonczy, CD Makinson, S Wang, TR Clandinin, RO Dror, JB Ding, N Ji, P Golshani, LM Giocomo, G–Q Bi, and MZ Lin, A positively tuned voltage indicator for extended electrical recordings in the brain. Nat Methods 20, 1104 (2023). https://doi.org/10.1038/s41592-023-01913-z
  * [90] L Tian, SA Hires, T Mao, D Huber, ME Chiappe, SH Chalasani, L Petreanu, J Akerboom, SA McKinney, ER Schreiter, CI Bargmann, V Jayaraman, K Svoboda, and LL Looger, Imaging neural activity in worms, flies and mice with improved GCaMP calcium indicators. Nat Methods 6, 875 (2009). https://doi.org/10.1038/nmeth.1398
  * [91] L Tian, SA Hires, and LL Looger, Neural activity imaging with genetically encoded calcium indicators. In Imaging in Neuroscience, F Helmchen and A Konnerth, eds, pp 647–656 (Cold Spring Harbor Laboratory Press, 2012). https://doi.org/10.1101/pdb.top069609
  * [92] YZhang, M Rózsa, Y Liang, D Bushey, Z Wei, J Zheng, D Reep, GJ Broussard, A Tsang, G Tsegaye, S Narayan, CJ Obara, J–X Lim, R Patel, R Zhang, MB Ahrens, GC Turner, SS–H. Wang, WL Korff, ER Schreiter, K Svoboda, JP Hasseman, I Kolb, and LL Looger, Fast and sensitive GCaMP calcium indicators for imaging neural populations. Nature 615, 884 (2023). https://doi.org/10.1038/s41586-023-05828-9
  * [93] W Denk, JH Strickler, and WW Webb, Two–photon laser–scanning fluorescence microscopy. Science 248, 73 (1990). https://doi.org/10.1126/science.23210
  * [94] DA Dombeck, AN Khabbaz, F Collman, TL Adelman, and DW Tank, Imaging large–scale neural activity with cellular resolution in awake, mobile mice. Neuron 56, 43 (2007). https://doi.org/10.1016/j.neuron.2007.08.003
  * [95] A Song, AS Charles, SA Koay, JL Gauthier, SY Thiberge, JW Pillow, and DW Tank, Volumetric two-photon imaging of neurons using stereoscopy (vTwINS). Nat Methods 14, 420 (2017). https://doi.org/10.1038/nmeth.4226
  * [96] S Weisenburger, F Tejera, J Demas, B Chen, J Manley, FT Sparks, FM Traub, T Daigle, H Zeng, A Losonczy, and A Vaziri, Volumetric Ca2+ imaging in the mouse brain using hybrid multiplexed sculpted light microscopy. Cell 177, 1050 (2019). https://doi.org/10.1016/j.cell.2019.03.011
  * [97] J Demas, J Manley, F Tejera, K Barber, H Kim, FM Traub, B Chen, and A Vaziri, High-speed, cortex-wide volumetric recording of neuroactivity at cellular resolution using light beads microscopy. Nat Methods 18, 1103 (2021). https://doi.org/10.1038/s41592-021-01239-8
  * [98] CD Harvey, F Collman, DA Dombeck, and DW Tank, Intracellular dynamics of hippocampal place cells during virtual navigation. Nature 461, 941 (2009). https://doi.org/10.1038/nature08499
  * [99] MB Ahrens, MB Orger, DN Robson, JM Li, and PJ Keller, Whole–brain functional imaging at cellular resolution using light–sheet microscopy. Nat Methods 10, 413 (2013). https://doi.org/10.1038/nmeth.2434
  * [100] JP Nguyen, FB Shipley, AN Linder, GS Plummer, JW Shaevitz, and AM Leifer, Whole–brain calcium imaging with cellular resolution in freely behaving Caenorhabditis elegans Proc Natl Acad Sci (USA) 113, E1074 (2016). https://doi.org/10.1073/pnas.1507110112
  * [101] G Nagel, T Szellas, W Huhn, S Kateriya, N Adeishvili, P Berthold, D Ollig, P Hegemann, and E Bamberg, Channelrhodopsin–2, a directly light-gated cation–selective membrane channel. Proc Natl Acad Sci (USA) 100, 13940 (2003). https://doi.org/10.1073/pnas.1936192100
  * [102] X Han amd ES Boyden, Multiple–color optical activation, silencing, and desynchronization of neural activity, with single–spike temporal resolution. PLoS One 2, e299 (2007). https://doi.org/10.1371/journal.pone.0000299
  * [103] F Zhang, M Prigge, F Beyrière, SP Tsunoda, J Mattis, O Yizhar, P Hegemann, and K Deisseroth, Red–shifted optogenetic excitation: A tool for fast neural control derived from Volvox carteri. Nat Neurosci 11, 631 (2008). https://doi.org/10.1038/nn.2120
  * [104] F Randi, AK Sharma, S Dvali, and AM Leifer, Neural signal propagation atlas of Caenorhabditis elegans. Nature 623, 406 2023). https://doi.org/10.1038/s41586-023-06683-4
  * [105] L Meshulam, JL Gauthier, CD Brody, DW Tank, and W Bialek, Collective behavior of place and non–place neurons in the hippocampal network. Neuron 96, 1178 (2017). https://doi.org/10.1016/j.neuron.2017.10.027
  * [106] L Meshulam, JL Gauthier, CD Brody, DW Tank, and W Bialek, Coarse–graining, fixed points, and scaling in a large population of neurons. Phys Rev Lett 123, 178103 (2019). https://doi.org/10.1103/PhysRevLett.123.178103
  * [107] K Svoboda, CF Schmidt, BJ Schnapp, and SM Block, Direct observation of kinesin stepping by optical trapping interferometry. Nature 365, 721 (1993). https://doi.org/10.1038/365721a0
  * [108] EA Abbondanzieri, WJ Greenleaf, JW Shaevitz, R Landick, and SM Block, Direct observation of base-pair stepping by RNA polymerase. Nature 438, 460 (2005). https://doi.org/10.1038/nature04268
  * [109] H Ueno, T Suzuki, K Kinosita Jr, and M Yoshida, ATP–driven stepwise rotation of F0F1–ATP synthase. Proc Natl Acad Sci (USA) 102, 1333 (2005). https://doi.org/10.1073/pnas.0407857102
  * [110] A Cavagna, D Conti, C Creato, L Del Castello, I Giardina, TS Grigera, S Melillo, L Parisi, and M Viale, Dynamic scaling in natural swarms. Nat Phys 13, 914 (2017). https://doi.org/10.1038/nphys4153
  * [111] A Cavagna, I Giardina, and T Grigera, The physics of flocking: Correlation as a compass from experiments to theory. Phys Repts 728, 1 (2018). https://doi.org/10.1016/j.physrep.2017.11.003
  * [112] B Qin, C Fei, AA Bridges, AA Mashruwala, HA Stone, NS Wingreen, and BL Bassler, Cell position fates and collective fountain flow in bacterial biofilms revealed by light–sheet microscopy. Science 369, 71 (2020). https://doi.org/10.1126/science.abb850
  * [113] K Copenhagen, R Alert, NS Wingreen, and JW Shaevitz, Topological defects promote layer formation in Myxococcus xanthus colonies. Nat Phys 17, 211 (2021). https://doi.org/10.1038/s41567-020-01056-4
  * [114] A Warmflash, B Sorre, F Etoc, ED Siggia, and AH Brivanlou, A method to recapitulate early embryonic spatial patterning in human embryonic stem cells. Nat Methods 11, 847 (2014). https://doi.org/10.1038/nmeth.3016
  * [115] MN Shahbazi, ED Siggia, and M Zernicka–Goetz, Self–organization of stem cells into embryos: A window on early mammalian development. Science 364, 948 (2019). https://doi.org/10.1126/science.aax0164
  * [116] W Bialek, B Carragher, I Cissé, MM Desai, OK Dudko, DI Goldman, J Kondev, PB Littlewood, AJ Liu, ME Moxon, JN Onuchic, MJ Schnitzer, and CM Waterman, Physics of Life. (National Academies Press, Washington DC, 2022). https://nap.edu/physicsoflife
  * [117] J Shlens, A tutorial on principal component analysis. arXiv:1404.1100 [cs.LG] (2014). https://doi.org/10.48550/arXiv.1404.1100
  * [118] ST Roweis and LK Saul, Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323 (2000). DOI:10.1126/science.290.5500.2323
  * [119] JP Gergen, D Coulter, and EF Wieschaus, Segmental pattern and blastoderm cell identities. In Gametogenesis and The Early Embryo, JG Gall, ed, pp 195–220 (Liss, New York, 1986).
  * [120] JO Dubuis, G Tkačik, EF Wieschaus, T Gregor, and W Bialek, Positional information, in bits. Proc Natl Acad Sci (USA) 110, 16301 (2013). https://doi.org/10.1073/pnas.1315642110
  * [121] L McGough, H Casademunt, M Nikolić, MD Petkova, T Gregor, and W Bialek, Finding the last bits of positional information. arXiv:2312.05963 [q–bio.MN] (2023). https://doi.org/10.48550/arXiv.2312.05963
  * [122] M Ptashne, A Genetic Switch: Phage and Higher Organisms (Blackwell, Cambridge MA, 1992).
  * [123] PV Pedone, R Ghirlando, GM Clore, AM Gronenbron, G Felsenfeld, and JG Omchinski, The single Cys2–His2 zinc finger domain of the GAGA protein flanked by basic residues is sufficient for high–affinity specific DNA binding. Proc Natl Acad Sci USA 93, 2822 (1996). https://doi.org/10.1073/pnas.93.7.2822
  * [124] RL Winston, DP Millar, JM Gottesfeld, and SB Kent, Characterization of the DNA binding properties of the bHLH domain of Deadpan to single and tandem sites. Biochemistry 38, 5138 (1999). https://doi.org/10.1021/bi982856a
  * [125] T Gregor, DW Tank, EF Wieschaus, and W Bialek, Probing the limits to positional information. Cell 130, 153 (2007). https://doi.org/10.1016/j.cell.2007.05.025
  * [126] W Bialek, Thinking about the brain. In Physics of Biomolecules and Cells: Les Houches Session LXXV, H Flyvbjerg, F Jülicher, P Ormos, and F David, eds, pp 485–577 (EDP Sciences, Les Ulis; Springer–Verlag, Berlin, 2002); arXiv:physics/0205030 (2002). https://doi.org/10.48550/arXiv.physics/0205030
  * [127] GD Field and F Rieke, Nonlinear signal transfer from mouse rods to bipolar cells and implications for visual sensitivity. Neuron 34, 773 (2002). https://doi.org/10.1016/S0896-6273(02)00700-6
  * [128] F Rieke, WG Owen, and W Bialek, Optimal filtering in the salamander retina. Advances in Neural Information Processing 3, 377 (1991). https://proceedings.neurips.cc/paper_files/paper/1990/file/019d385eb67632a7e958e23f24bd07d7-Paper.pdf
  * [129] S Hassenstein and W Reichardt, Systemtheoretische Analyse der Zeit-, Reihenfolgen- und Vorzeichenauswertung bei der Bewegungsperzeption des Rüsselkäfers Chlorophanus. Z Naturforsch 11B, 513 (1956). https://doi.org/10.1515/znb-1956-9-1004
  * [130] W Reichardt and T Poggio, Visual control of orientation behaviour in the fly: Part I. A quantitative analysis, Q Rev Biophys 9, 311 (1976). https://doi.org/10.1017/S0033583500002523
  * [131] EH Adelson and JR Bergen, Spatiotemporal energy models for the perception of motion. J Opt Soc Am A 2, 284 (1985). https://doi.org/10.1364/JOSAA.2.000284
  * [132] JPH van Santen and G Sperling, Elaborated Reichardt detectors. J Opt Soc Am A 2, 300 (1985). https://doi.org/10.1364/JOSAA.2.000300
  * [133] NJ Strausfeld, Atlas of an Insect Brain (Springer–Verlag, Berlin, 1976).
  * [134] K Hausen, Motion sensitive interneurons in the optomotor system of the fly. I. The horizontal cells: Structure and signals. Biol Cybern 45, 143 (1982). https://doi.org/10.1007/BF00335241
  * [135] DG Stavenga and RC Hardie, eds, Facets of Vision. (Springer–Verlag, Berlin, 1989).
  * [136] RR de Ruyter van Steveninck and SB Laughlin, The rate of information transfer at graded-potential synapses. Nature 379, 642\. (1996). https://doi.org/10.1038/379642a0
  * [137] RR de Ruyter van Steveninck and SB Laughlin, Light adaptation and reliability in blowfly photoreceptors. Int J Neural Syst 7, 437 (1996). https://doi.org/10.1142/S0129065796000415
  * [138] W Bialek, F Rieke, RR de Ruyter van Steveninck, and D Warland, Reading a neural code. Science 252, 1854 (1991). https://doi.org/10.1126/science.2063199
  * [139] R de Ruyter van Steveninck and W Bialek, Reliability and statistical efficiency of a blowfly movement–sensitive neuron. Phil Trans R Soc Lond 348, 321 (1995). https://doi.org/10.1098/rstb.1995.0071
  * [140] M Potters and W Bialek, Statistical mechanics and visual signal processing. J Phys I France 4, 1755 (1994). https://doi.org/10.1051/jp1:1994219
  * [141] SR Sinha, W Bialek, and RR de Ruyter van Steveninck, Optimal local estimates of visual motion in a natural environment. Phys Rev Lett 126, 018101 (2021). https://doi.org/10.1103/PhysRevLett.126.018101
  * [142] JE Fitzgerald, AY Katsov, TR Clandinin, and MJ Schnitzer, Symmetries in stimulus statistics shape the form of visual motion estimators. Proc Natl Acad Sci (USA) 108, 12909 (2011). https://doi.org/10.1073/pnas.1015680108
  * [143] R Behnia, DA Clark, AG Carter, TR Clandinin, and C Desplan, Processing properties of ON and OFF pathways for Drosophila motion detection. Nature 512, 427 (2014). https://doi.org/10.1038/nature13427
  * [144] L Wolpert, Positional information and the spatial pattern of cellular differentiation. J Theor Biol 25, 1 (1969). https://doi.org/10.1016/S0022-5193(69)80016-0
  * [145] D Krotov, JO Dubuis, T Gregor, and W Bialek, Morphogenesis at criticality? Proc Natl Acad Sci (USA) 111, 3683 (2014). https://doi.org/10.1073/pnas.1324186111
  * [146] CE Shannon, A mathematical theory of communication. Bell Sys Tech J 27, 379 and 623 (1948). https://doi.org/10.1002/j.1538-7305.1948.tb01338.x Reprinted in CE Shannon and W Weaver, The Mathematical Theory of Communication (University of Illinois Press, Urbana, 1949).
  * [147] TM Cover and JA Thomas, Elements of Information Theory (Wiley, New York, 1991).
  * [148] M Mézard and A Montanari, Information, Physics, and Computation. (Oxford University Press, Oxford and New York, 2009).
  * [149] SB Laughlin, A simple coding procedure enhances a neuron’s information capacity. Z Naturforsch 36c, 910–912 (1981). https://doi.org/10.1515/znc-1981-9-1040
  * [150] S Smirnakis, MJ Berry II, DK Warland, W Bialek, and M Meister, Adaptation of retinal processing to image contrast and spatial scale. Nature 386, 69 (1997). https://doi.org/10.1038/386069a0
  * [151] F Rieke, D Warland, R de Ruyter van Steveninck, and W Bialek Spikes: Exploring the Neural Code (MIT Press, Cambridge, 1997).
  * [152] LF Abbott and P Dayan, Theoretical Neuroscience. Computational and Mathematical Modeling of Neural Systems (MIT Press, Cambridge MA, 2001).
  * [153] E de Boer and P Kuyper, Triggered correlation. IEEE Trans Biomed Eng 15, 169 (1968). https://doi.org/10.1109/TBME.1968.4502561
  * [154] R de Ruyter van Steveninck and W Bialek, Real–time performance of a movement sensitive neuron in the blowfly visual system: Coding and information transfer in short spike sequences. Proc R Soc Lond B 234, 379–414 (1988). https://doi.org/10.1098/rspb.1988.0055
  * [155] N Brenner, W Bialek, and R de Ruyter van Steveninck, Adaptive rescaling optimizes information transmission. Neuron 26, 695 (2000). https://doi.org/10.1016/S0896-6273(00)81205-2
  * [156] W Bialek and R de Ruyter van Steveninck, Features and dimensions: Motion estimation in fly vision. arXiv:q–bio/0505003 (2005). https://doi.org/10.48550/arXiv.q-bio/0505003
  * [157] NC Rust, O Schwartz, JA Movshon, and EP Simoncelli, Spatiotemporal elements of Macaque V1 receptive fields. Neuron 46, 945 (2005). https://doi.org/10.1016/j.neuron.2005.05.021
  * [158] AL Fairhall, GD Lewen, W Bialek, and RR de Ruyter van Steveninck, Efficiency and ambiguity in an adaptive neural code. Nature 412, 787 (2001). https://doi.org/10.1038/35090500
  * [159] KI Nagel and AJ Doupe, Temporal processing and adaptation in the songbird auditory forebrain. Neuron 51, 845 (2006). https://doi.org/10.1016/j.neuron.2006.08.030
  * [160] M Maravall, RS Petersen, AL Fairhall, E Arabzadeh, and ME Diamond, Shifts in coding properties and maintenance of information transmission during adaptation in barrel cortex. PLoS Biol 5, e19 (2007). https://doi.org/10.1371/journal.pbio.0050019
  * [161] I Dean, N Harper, and D McAlpine, Neural population coding of sound level adapts to stimulus statistics. Nat Neurosci 8, (2005). https://doi.org/10.1038/nn1541
  * [162] B Wark, BN Lundstrom, and AL Fairhall, Sensory adaptation. Curr Opin Neurobiol 17, 423 (2007). https://doi.org/10.1016/j.conb.2007.07.001
  * [163] KJ Kim and F Rieke, Temporal contrast adaptation in the input and output signals of salamander retinal ganglion cells. J Neurosci 21, 287 (2001). https://doi.org/10.1523/JNEUROSCI.21-01-00287.2001
  * [164] KJ Kim and F Rieke, Slow Na+ inactivation and variance adaptation in salamander retinal ganglion cells. J Neurosci 23, 1506 (2003). https://doi.org/10.1523/JNEUROSCI.23-04-01506.2003
  * [165] G Tkačik, JO Dubuis, MD Petkova, and T Gregor, Positional information, positional error, and read–out precision in morphogenesis: a mathematical framework. Genetics 199, 39 (2015). https://doi.org/10.1534/genetics.114.171850
  * [166] F Liu, AH Morrison, and T Gregor, Dynamic interpretation of maternal inputs by the Drosophila segmentation gene network. Proc Natl Acad Sci (USA) 110, 6724 (2013). https://doi.org/10.1073/pnas.1220912110
  * [167] AM Arias and P Hayward, Filtering transcriptional noise during development: Concepts and mechanisms. Nat Rev Genet 7, 34–44 (2006). https://doi.org/10.1038/nrg1750
  * [168] TC Lacalli, Patterning, from conifers to consciousness: Turing’s theory and order from fluctuations. Front Cell Dev Biol 10, 871950 (2022). https://doi.org/10.3389/fcell.2022.871950
  * [169] M Bauer, MD Petkova, T Gregor, EF Wieschaus, and W Bialek, Trading bits in the readout from a genetic network. Proc Natl Acad Sci (USA) 118, e2109011118 (2021). https://doi.org/10.1073/pnas.210901111
  * [170] M Bauer and W Bialek, Information bottleneck in molecular sensing. PRX Life 1, 023005 (2023). https://doi.org/10.1103/PRXLife.1.023005
  * [171] N Tishby, FC Pereira, and W Bialek, The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, B Hajek and RS Sreenivas, eds, pp 368–377 (University of Illinois, 1999); arXiv:physics/0004057 (2000). https://doi.org/10.48550/arXiv.physics/0004057
  * [172] HB Barlow, Sensory mechanisms, the reduction of redundancy, and intelligence. In Proceedings of the Symposium on the Mechanization of Thought Processes, Volume II, DV Blake and AM Uttley, eds, pp 537–574 (HM Stationery Office, London, 1959).
  * [173] HB Barlow, Possible principles underlying the transformation of sensory messages. In Sensory Communication, W Rosenblith, ed, pp 217–234 (MIT Press, Cambridge, 1961).
  * [174] E Schrödinger, Über das Verhältnis der Vierfarben- zur Dreifarbentheorie. Sitzungsberichte der Akademie der Wissenschaften in Wien Ila, mathematischnaturwissenschaftliche Klasse 134, 417 (1925). Translated with commentary by KK Niall, On the trichromatic and opponent-process theories: An article by E. Schrödinger. Spat Vis 3, 79 (1988). https://doi.org/10.1163/156856888X00050
  * [175] KR Gegenfurtner and LT Sharpe, eds, Color Vision: From Genes to Perception (Cambridge University Press, Cambridge, 2001).
  * [176] G Buchsbaum and A Gottschalk, Trichromacy, opponent colours coding and optimum colour information transmission in the retina. Proc R Soc Lond B 220, 89 (1983). https://doi.org/10.1098/rspb.1983.0090
  * [177] DL Ruderman, TW Cronin, and C–C Chiao, Statistics of cone responses to natural images: Implications for visual coding. J Opt Soc Am A 15, 2036 (1998). https://doi.org/10.1364/JOSAA.15.002036
  * [178] JJ Atick and AN Redlich, Toward a theory of early visual processing. Neural Comput 2, 308 (1990). https://doi.org/10.1162/neco.1990.2.3.308
  * [179] JH van Hateren, Real and optimal neural images in early vision. Nature 360, 68 (1992). https://doi.org/10.1038/360068a0
  * [180] DL Ruderman and W Bialek, Statistics of natural images: Scaling in the woods. Phys Rev Lett 73, 814 (1994). https://doi.org/10.1103/PhysRevLett.73.814
  * [181] HB Barlow, Summation and inhibition in the frog’s retina. J Physiol (Lond) 119, 69 (1953). https://doi.org/10.1113/jphysiol.1953.sp004829
  * [182] SW Kuffler, Discharge patterns and functional organization of the mammalian retina. J Neurophysiol 16, 37 (1953). https://doi.org/10.1152/jn.1953.16.1.37
  * [183] HK Hartline, Visual receptors and retinal interaction. In Nobel Lectures, Physiology or Medicine 1963–1970. (Elsevier Publishing, Amsterdam, 1972). https://www.nobelprize.org/prizes/medicine/1967/hartline/lecture/
  * [184] CE Shannon, Communication in the presence of noise. Proc IRE 37, 10 (1949). https://doi.org/10.1109/JRPROC.1949.232969
  * [185] SB Laughlin and RR de Ruyter van Steveninck, Measurements of signal transfer and noise suggest a new model for graded transmission at an adapting retinal synapse. J Physiol (Lond) 494, P19 (1996).
  * [186] BN Lundstrom. MH Higgs, WJ Spain, and AL Fairhall, Fractional differentiation by neocortical pyramidal neurons. Nat Neurosci 11, 1335 (2008). https://doi.org/10.1038/nn.2212
  * [187] BL Lundstriom, AL Fairhall, and M Maravall, Multiple timescale encoding of slowly varying whisker stimulus envelope in cortical and thalamic neurons in vivo. J Neurosci 30, 5071 (2010). https://doi.org/10.1523/JNEUROSCI.2193-09.2010
  * [188] J Thorson and M Biederman–Thorson, Distributed relaxation processes in sensory adaptation. Science 183, 161 (1974). https://doi.org/10.1126/science.183.4121.161
  * [189] G Tkačik and W Bialek, Diffusion, dimensionality and noise in transcriptional regulation. Phys Rev E 79, 051901 (2009). https://doi.org/10.1103/PhysRevE.79.051901
  * [190] W Bialek and S Setayeshgar, Physical limits to biochemical signaling. Proc Natl Acad Sci (USA) 102, 10040 (2005). https://doi.org/10.1073/pnas.0504321102
  * [191] W Bialek and S Setayeshgar, Cooperativity, sensitivity and noise in biochemical signaling. Phys Rev Lett 100, 258101 (2008). https://doi.org/10.1103/PhysRevLett.100.258101
  * [192] JS van Zon, MJ Morelli, S Tanase–Nicola, and PR ten Wolde, Diffusion of transcription factors can drastically enhance the noise in gene expresssion. Biophys J 91, 4350 (2006). https://doi.org/10.1529/biophysj.106.086157
  * [193] RG Endres and NS Wingreen, Maximum likelihood and the single receptor. Phys Rev Lett 103, 158101 (2009). https://doi.org/10.1103/PhysRevLett.103.158101
  * [194] T Mora and NS Wingreen, Limits of sensing temporal concentration changes by single cells. Phys Rev Lett 104, 248101 (2010). https://doi.org/10.1103/PhysRevLett.104.248101
  * [195] K Kaizu, W de Ronde, J Paijmans, K Takahashi, F Tostevin, and PR ten Wolde, The Berg–Purcell limit revisited. Biophys J 106, 976 (2014). https://doi.org/10.1016/j.bpj.2013.12.030
  * [196] T Mora, Physical limit to concentration sensing amid spurious ligands Phys Rev Lett 115, 038102 (2015). https://doi.org/10.1103/PhysRevLett.115.038102
  * [197] M Carballo–Pacheco, J Desponds, T Gavrilchenko, A Mayer, R Prizak, G Reddy, I Nemenman, and T Mora, Receptor crosstalk improves concentration sensing of multiple ligands. Phys Rev E 99, 022423 (2019). https://doi.org/10.1103/PhysRevE.99.022423
  * [198] T Mora and I Nemenman, Physical limit to concentration sensing in a changing environment. Phys Rev Lett 123, 198101 (2019). https://doi.org/10.1103/PhysRevLett.123.198101
  * [199] V Ngampruetikorn, DJ Schwab, and GJ Stephens, Energy consumption and cooperation for optimal sensing. Nat Commun 11, 975 (2020). https://doi.org/10.1038/s41467-020-14806-y
  * [200] G Tkačik, AM Walczak, and W Bialek, Optimizing information flow in small genetic networks. Phys Rev E 80, 031920 (2009). https://doi.org/10.1103/PhysRevE.80.031920
  * [201] AM Walczak, G Tkačik, and W Bialek, Optimizing information flow in small genetic networks. II: Feed–forward interaction. Phys Rev E 81, 041905 (2010). https://doi.org/10.1103/PhysRevE.81.041905
  * [202] G Tkačik, AM Walczak, and W Bialek, Optimizing information flow in small genetic networks. III. A self–interacting gene. Phys Rev E 85, 041903 (2012). https://doi.org/10.1103/PhysRevE.85.041903
  * [203] SC Little, M Tikhonov, and T Gregor, Precise developmental gene expression arises from globally stochastic transcriptional activity. Cell 154, 789 (2013). https://doi.org/10.1016/j.cell.2013.07.025
  * [204] TR Sokolowski and G Tkačik, Optimizing information flow in small genetic networks. IV. Spatial coupling. Phys Rev E 91, 062710 (2015). https://doi.org/10.1103/PhysRevE.91.062710
  * [205] TR Sokolowski, AM Walczak, W Bialek, and G Tkačik, Extending the dynamic range of transcription factor action by translational regulation. Phys Rev E  93, 022404 (2016). https://doi.org/10.1103/PhysRevE.93.022404
  * [206] J Dubnau and G Struhl, RNA recognition and translational regulation by a homeodomain protein. Nature 379, 694 (1996). https://doi.org/10.1038/379694a0
  * [207] R Rivera-Pomar, D Niessing, U Schmidt–Ott, WJ Gehring, and H Jäckle, RNA binding and translational suppression by bicoid. Nature 379, 746 (1996). https://doi.org/10.1038/379746a0
  * [208] D Niessing, W Driever, F Sprengerm. H Taubert, and H Jäckle, Homeodomain position 54 specifies transcriptional versus translational control by Bicoid. Mol Cell 5, 395 (2000). https://doi.org/10.1016/S1097-2765(00)80434-7
  * [209] O Johnstone and P Lasko, Translational regulation and RNA localization in Drosophila oocytes and embryos. Annu Rev Genet 35, 365 (2001). https://doi.org/10.1146/annurev.genet.35.102401.090756
  * [210] TR Sokolowski, T Gregor, W Bialek, and G Tkačik, Deriving a genetic regulatory network from an optimization principle. arXiv:2302.05680 [physics.bio–ph] (2023). https://doi.org/10.48550/arXiv.2302.05680
  * [211] J Monod, J–P Changeux, and F Jacob, Allosteric proteins and cellular control systems. J Mol Biol 6, 306 (1963). https://doi.org/10.1016/S0022-2836(63)80091-1
  * [212] J Monod, J Wyman, and J–P Changeux, On the nature of allosteric transitions: A plausible model. J Mol Biol 12, 88 (1965). https://doi.org/10.1016/S0022-2836(65)80285-6
  * [213] MF Perutz, Mechanisms of Cooperativity and Allosteric Regulation in Proteins (Cambridge University Press, Cambridge, 1990).
  * [214] R Phillips, The Molecular Switch: Signaling and Allostery (Princeton University Press, Princeton NJ, 2020).
  * [215] JJ Hopfield, Relation between structure, co–operativity and spectra in a model of hemoglobin action. J Mol Biol 77, 207 (1973). https://doi.org/10.1016/0022-2836(73)90332-X
  * [216] L Bintu, NE Buchler, HG Garcia, U Geraland, T Hwa, J Kondev, and R Phillips, Transcriptional regulation by the numbers: models. Curr Opin Genet Dev 15, 116 (2005). https://doi.org/10.1016/j.gde.2005.02.007
  * [217] L Bintu, NE Buchler, HG Garcia, U Gerland, T Hwa, J Kondev, T Kuhlman, and Rob Phillips, Transcriptional regulation by the numbers: applications. Curr Opin Genet Dev 15, 125 (2005). https://doi.org/10.1016/j.gde.2005.02.006
  * [218] B Zoller, T Gregor, and G Tkačik, Eukaryotic gene regulation at equilibrium, or non? Curr Opin Syst Biol 31, 100435 (2022). https://doi.org/10.1016/j.coisb.2022.100435
  * [219] PS Swain, MB Elowitz, and ED Siggia, Intrinsic and extrinsic contributions to stochasticity in gene expression. Proc Natl Acad Sci (USA) 99, 12795 (2002). https://doi.org/10.1073/pnas.162041399
  * [220] A Crombach, MA García–Solache, and J Jaeger, Evolution of early development in dipterans: reverse-engineering the gap gene network in the moth midge Clogmia albipunctata (psychodidae). Biosystems 123, 74 (2014). https://doi.org/10.1016/j.biosystems.2014.06.003
  * [221] KR Wotton, E Jiménez–Guri, A Crombach, H Janssens, Anna Alcaine–Colet, S Lemke, U Schmidt–Ott, J Jaeger, Quantitative system drift compensates for altered maternal inputs to the gap gene network of the scuttle fly Megaselia abdita. eLife 4, 04785 (2015). https://doi.org/10.7554/eLife.04785
  * [222] Y Goltsev, W Hsiong, G Lanzaro, and M Levine, Different combinations of gap repressors for common stripes in Anopheles and Drosophila embryos. Dev Biol 275, 435 (2004). https://doi.org/10.1016/j.ydbio.2004.08.021