††thanks: To whom correspondence should be sent. These authors contributed
equally.††thanks: To whom correspondence should be sent. These authors
contributed equally.

# A small-correlation expansion to quantify information in noisy sensory
systems

Gabriel Mahuas  Institut de la Vision, Sorbonne Université, CNRS, INSERM, 17
rue Moreau, 75012, Paris, France  Laboratoire de physique de École normale
supérieure, CNRS, PSL University, Sorbonne University, Université Paris-Cité,
24 rue Lhomond, 75005 Paris, France  Olivier Marre  Institut de la Vision,
Sorbonne Université, CNRS, INSERM, 17 rue Moreau, 75012, Paris, France
Thierry Mora  Laboratoire de physique de École normale supérieure, CNRS, PSL
University, Sorbonne University, Université Paris-Cité, 24 rue Lhomond, 75005
Paris, France  Ulisse Ferrari  Institut de la Vision, Sorbonne Université,
CNRS, INSERM, 17 rue Moreau, 75012, Paris, France

###### Abstract

Neural networks encode information through their collective spiking activity
in response to external stimuli. This population response is noisy and
strongly correlated, with complex interplay between correlations induced by
the stimulus, and correlations caused by shared noise. Understanding how these
correlations affect information transmission has so far been limited to pairs
or small groups of neurons, because the curse of dimensionality impedes the
evaluation of mutual information in larger populations. Here we develop a
small-correlation expansion to compute the stimulus information carried by a
large population of neurons, yielding interpretable analytical expressions in
terms of the neurons’ firing rates and pairwise correlations. We validate the
approximation on synthetic data and demonstrate its applicability to
electrophysiological recordings in the vertebrate retina, allowing us to
quantify the effects of noise correlations between neurons and of memory in
single neurons.

Networks of neurons from sensory systems are characterized by strong
correlations that shape their collective response to stimuli Brivanlou et al.
(1998); Trenholm et al. (2014); Schneidman et al. (2006); Shlens et al.
(2008); Trong and Rieke (2008); Ala-Laurila et al. (2011); Deweese and Zador
(2004); Lin et al. (2015); Kohn and Smith (2005); Montani et al. (2007); Smith
and Kohn (2008); Ponce-Alvarez et al. (2013); Franke et al. (2016); Zylberberg
et al. (2016). These correlations have two sources Brivanlou et al. (1998):
stimulus correlations, which originate from shared or correlated stimuli that
affect the mean activities of different neurons in a concerted way; and noise
correlations, which stem from network interactions that couple noise across
cells. These two sources of correlations impact how well the population
encodes stimulus information, and detailed investigations have explored this
effect both experimentally Montani et al. (2007); Smith and Kohn (2008);
Ponce-Alvarez et al. (2013); Franke et al. (2016); Zylberberg et al. (2016);
Meister et al. (1995); Nirenberg et al. (2001); Pillow et al. (2008); Ruda et
al. (2020); Hazon et al. (2022); Boffi et al. (2022) and theoretically Zohary
et al. (1994); Brunel and Nadal (1998); Abbott and Dayan (1999); Panzeri et
al. (1999); Sompolinsky et al. (2001); Pola et al. (2003); Tkacik et al.
(2010); Ecker et al. (2010, 2011); Hu et al. (2014); Moreno-Bote et al.
(2014), showing a wide variety of scenarios in which noise correlations could
either hurt or improve information transmission (see Azeredo da Silveira and
Rieke (2021) for a recent review).

While geometric arguments about the structure of stimulus and noise
correlations can help interpret and evaluate the impact of their interplay on
information transmission for pairs or small groups of cells Hu et al. (2014);
Azeredo da Silveira and Rieke (2021), specific challenges arise when dealing
with large populations of cells. A common way to quantify these effects is by
computing the mutual information between the stimulus and the activity of the
whole population. However, attempts at quantifying this information are
inherently limited by the curse of dimensionality, whereby the size of the
state space to be sampled grows exponentially with the system’s size. Models
based on the principle of maximum entropy have been proposed to build explicit
probabilistic models of the collective activity of many neurons, based on mean
spike rates and correlation functions Jaynes (1982); Schneidman et al. (2006);
Tang et al. (2008); Roudi et al. (2009); Shimazaki et al. (2012); Watanabe et
al. (2013); Granot-Atedgi et al. (2013); Tkacik et al. (2013, 2014, 2015);
Cofre and Cessac (2014); Mora et al. (2015); Shimazaki et al. (2015); Tavoni
et al. (2017); Meshulam et al. (2017); Ferrari et al. (2017); Nghiem et al.
(2018); Cayco-Gajic et al. (2018); Delamare and Ferrari (2022). These
distributions map onto known models of statistical mechanics, and can be used
to evaluate entropies as well as mutual informations.

In this paper, we leverage these techniques from statistical physics to
compute a small correlation expansion of the mutual information Sessak and
Monasson (2009). This approach outperforms previous approximations of the
mutual information and is computationally efficient. The resulting formulas,
while implicitly relying on the maximum entropy assumption, are expressed in a
model-free manner as a function of the experimental observables, yielding a
very simple picture of how correlations affect information encoding in sensory
systems. We test this approach on synthetic data and showcase it on real
electrophysiological recordings from the retina, to illustrate how it can be
used to quantify the effect of noise correlations between neurons in the
population as well as the effect of temporal correlations in single neurons.

Figure 1: How correlations affect information. A) Visual stimuli drive the
noisy response of sensory neurons (spikes, represented by vertical ticks). We
call $\rho_{ij}^{\rm n}({\bf s})$ the pairwise Pearson correlation of binary
activities $n_{i},n_{j}$ between cells $i$ and $j$ in a short window $\Delta
t$, conditioned on past stimulus $\bf s$. The unconditioned Pearson
correlation, $\rho^{\rm tot}$, can be decomposed into stimulus and noise
contributions, $r^{\text{s}}$ and $r^{\text{n}}$. B) We estimate the mutual
information $I$ between stimulus $\bf s$ and response $\bf n$ through a small
correlation expansion Sessak and Monasson (2009). Noise synergy $\Delta I$
estimates the impact of noise correlations onto mutual information, and is
defined as the gain in information relative to the conditionally independent
information $I_{\text{CI}}$, corresponding to no noise correlations (obtained
by shuffling neural responses across repetitions). In the diagrammatic
expansions, each line corresponds to a correlation term; double lines are sums
of two correlations; multiple lines connecting the same two points are
multiplied.

Small correlation expansion of the mutual information. The collective response
of a neural network of size $N$ can be described by the neuronal activities
${\bf n}=(n_{1},\ldots,n_{N})$, taking value 0 or 1 depending on whether the
neuron spikes or not within a short time window $\Delta t$ (typically 10 – 20
ms). In general, because of processing delays and adaptation, the response is
a stochastic function $P(\bf n|\bf s)$ of the history of the stimulus $\bf s$
up to the response. The mutual information $I({\bf n},{\bf s})$ quantifies the
amount of information conveyed by the neural response about the stimulus
Shannon (1948); Rieke et al. (1999). Since it is expressed as a difference of
entropies $I=H[\mathbf{n}]-\langle H[{\bf n}|{\bf s}]\rangle_{\bf s}$, where
$H[x]=-\sum_{x}P(x)\ln P(x)$, its quantification requires good entropy
estimators. Direct estimation methods from data exist, and can be applied for
relatively small groups of neurons Strong et al. (1998). However, the
estimation problem quickly becomes intractable as the number of neurons
increases and the size of the response space grows exponentially. To deal with
large networks, we thus developed a method based on a small correlation
expansion of entropies Sessak and Monasson (2009), which allows us to express
them as analytical functions of the empirical correlations.

We start by assuming that both $P({\bf n})$ and $P({\bf n}|{\bf s})$ follow
the form of maximum entropy models consistent with empirical pairwise
covariances and spike rates. Later we will discuss the limitations of this
assumption. The total covariance between two cells $i$ and $j$ across stimuli,
$C_{ij}\equiv\mathrm{Cov}(n_{i},n_{j})$, can be decomposed into two
contributions corresponding to the effects of stimulus and noise:
$C_{ij}=C_{ij}^{\rm s}+\langle C_{ij}^{\rm n}({\bf s})\rangle_{\bf s}$, with
$C_{ij}^{\rm s}\equiv\mathrm{Cov}_{\bf s}(\langle n_{i}\rangle_{n_{i}|\bf
s},\langle n_{j}\rangle_{n_{j}|{\bf s}})$, $C_{ij}^{\rm n}({\bf
s})\equiv\mathrm{Cov}(n_{i},n_{j}|{\bf s})$, which can be computed from the
response to repeated presentations of the same stimulus. Likewise, the Pearson
correlation coefficient $\rho_{ij}^{\text{tot}}\equiv
C_{ij}/\sqrt{C_{ii}C_{jj}}$ can also be decomposed into stimulus- and noise-
induced contributions:
$\rho_{ij}^{\text{tot}}=r_{ij}^{\text{s}}+r_{ij}^{\text{n}}$ (Fig. 1A), with
$r_{ij}^{\text{s}}\equiv C^{\rm s}_{ij}/\sqrt{C_{ii}C_{jj}}$ and
$r_{ij}^{\text{n}}\equiv C^{\rm n}_{ij}/\sqrt{C_{ii}C_{jj}}$. Note however
that these two terms are not proper correlation coefficients because of the
normalization. Stimulus correlations may instead be quantified by
$\rho_{ij}^{\text{s}}\equiv C^{\rm s}_{ij}/\sqrt{C^{\rm s}_{ii}C^{\rm
s}_{jj}}$, and noise correlations in a stimulus-dependent manner through:
$\rho_{ij}^{\text{n}}({\bf s})\equiv C^{\rm n}_{ij}({\bf s})/\sqrt{C^{\rm
n}_{ii}({\bf s})C^{\rm n}_{jj}({\bf s})}$.

Following the approach of Sessak and Monasson Sessak and Monasson (2009), we
can expand the entropy of the maximum entropy models—and thus the mutual
information—at small values of the correlation parameter, $I=I_{\rm
0th}+I_{\rm 1st}+I_{\rm 2nd}+\ldots$ (App. A). The leading order of this
expansion is the sum of the information carried by each neuron: $I_{\rm
0th}=\sum_{i}\left[H[{n_{i}}]-\langle H[n_{i}|{\bf s}]\rangle_{\bf s}\right]$.
The first order term vanishes, while the second one reads (Fig. 1B, App. A):

$I_{\rm 2nd}=-\frac{1}{2}\sum_{i<j}\left({\rho_{ij}^{\rm
tot}}^{2}-\langle\rho^{\rm n}_{ij}(\mathbf{s})^{2}\rangle_{\bf s}\right).$ (1)

We can compute higher-order terms using Feynman diagrams rules Kühn and van
Wijland (2022), but they quickly become unwieldy. However, some of these terms
can be re-summed to yield a better approximation of the mutual information
than (1) in terms of first and second order moments Sessak and Monasson (2009)
(App. B):

$\displaystyle I$ $\displaystyle\approx I_{\rm 0th}+I_{\rm
pairs}+I_{\mathcal{G}}-I_{\rm dbl}.$ (2)

$I_{\text{\rm pairs}}$ is the sum of the mutual information gains (with
respect to single cells) of each pair $(i,j)$ calculated one by one, ignoring
the rest of the network. $I_{\mathcal{G}}$ is the mutual information gain
computed through a mean-field (or loop) approximation Cocco and Monasson
(2011); Sessak and Monasson (2009), which is equivalent to assuming that all
fluctuations (stimulus and noise) are Gaussian:

$\displaystyle\begin{split}I_{\mathcal{G}}&=\frac{1}{2}\log\left(|\rho^{\rm
tot}|\right)-\frac{1}{2}\langle\ln\left(|\rho^{\rm
n}(\mathbf{s})|\right)\rangle_{\bf s},\end{split}$ (3)

where $|\rho|$ denotes the determinant of the correlation matrix. Finally
$I_{\text{dbl}}$ corrects for terms that are double-counted in $I_{\rm pairs}$
and $I_{\mathcal{G}}$.

Noise synergy. These expansions can be used to investigate the impact of noise
correlations on information transmission. We define the noise synergy, $\Delta
I\equiv I-I_{\text{CI}}$, as the gain in information relative to the
conditionnally independent case (Fig. 1B, bottom line). $I_{\rm CI}$ can be
computed in practice by shuffling the response of individual neurons across
repetitions of the same stimulus, which preserves stimulus correlations but
destroys noise correlations. At second order we obtain (App. A):

$\displaystyle\Delta
I\approx\sum_{i<j}\left[-r_{i,j}^{\mathrm{n}}r_{i,j}^{\mathrm{s}}+\frac{1}{2}\left(\left<{\rho_{i,j}^{\mathrm{n}}(\mathbf{s})}^{2}\right>_{\mathbf{s}}-{r_{i,j}^{\mathrm{n}}}^{2}\right)\right].$
(4)

This expression shows how noise synergy depends on noise correlations through
$r^{\mathrm{n}}$ and $\mathbf{\rho}^{\mathrm{n}}$. The first term is positive
when noise and stimulus correlations have opposite signs. This effect is known
in the literature as the sign-rule Hu et al. (2014) and can be interpreted in
terms of the whitening of the output power spectrum: it is beneficial for the
network to “cancel out” input correlations by adding noise correlations of
opposite sign, in order to approach a uniformly distributed output, thereby
increasing output entropy and information. The second term of (4), which is of
second order in the noise correlation parameter, can be either positive or
negative in general. However, in the particular case of noise correlations
independent of the stimulus, $\rho^{\text{n}}({\bf s})=\rho^{\rm n}$, the
Cauchy-Schwarz inequality guarantees its non-negativity (see App. A for a
proof). This implies that noise correlations may be beneficial even when the
sign rule is violated. Noise synergy can also be computed using the re-summed
entropies of (2). The formulas are slightly more involved and are reported in
App. B.

Figure 2:  A) A spatially correlated random stimulus activates a network of 5
neurons according to a generalized linear model defined by stimulus, coupling,
and spike-history filters. B) Exact, second order (1) and re-summed (2) values
of the mutual information for various strengths of the noise and stimulus
correlations (averaged over all pairs of neighbors).

Numerical test on synthetic data. To test our approximations (1) and (2), we
built a generalized linear model to mimic the response of a small population
of five retinal neurons with nearest-neighbor interactions (Fig. 2A) for which
mutual informations could be estimated exactly. The stimulus is modeled as a
random Gaussian field sampled at 100Hz, with varying spatial correlations,
allowing us to tune the strength of stimulus correlations (App. C). The
stimulus is convolved with a linear filter consisting of a Gaussian receptive
field with biphasic temporal kernel Mahuas et al. (2020) (App. C). The mean
spike rate is controled by the result of this convolution, to which the effect
of its own spiking history is added, through a non-linear function. In
addition, the past activities of its neighbors control the stochastic part of
firing, through coupling filters (the mean effect of which is subtracted from
the average rate, see App. C). This strategy allows us to tune noise
correlations while keeping stimulus correlation constant. Importantly, this
model is mathematically inconsistent with the maximum entropy assumption. It
thus allows us to test for both the appropriateness of the maximum entropy
approximation in the context of state-of-the-art spiking models, and the
accuracy of the small-correlation expansion.

We then computed the exact mutual information between stimulus and response
using exhaustive numerical simulations, and compared it with the predictions
of our approximations (Fig. 2B). We observed an excellent agreement between
numerical calculations and analytical expressions, in particular for the re-
summed mutual information (2). Although less accurate, the second order
approximation (1) still provided fair estimates for a wide range of
correlation strengths.

Figure 3: Application on retinal population response to visual stimulation. A)
A mosaic of a population of off alpha cells in the rat retina. B) Stimulus
correlation ($\rho^{\rm s}_{ij}$) plotted against the distance between pair of
cells stimulated with a white noise movie. C) Same as (B) but for noise
correlations $\langle\rho^{\rm n}_{ij}({\bf s})\rangle_{\bf s}$. D) Noise
synergy for subset populations of nearby cells. Each boxplot corresponds to
the noise synergy of many subgroup of ganglion cells. Only nearby cells are
considered.

Application on retinal data. We applied our formulas to ex-vivo multi-
electrode array recordings of rat retinal ganglion cells in response to black
and white checkerboard stimulation Deny et al. (2017); Sorochynskyi et al.
(2021). The receptive fields of the cells have a mosaic structure (Fig. 3A),
so that neuronal responses show strong stimulus correlations between
neighbors, which decay with the distance between the receptive field centers
(Fig. 3B). Due to network effects Brivanlou et al. (1998), nearby cells also
show strong noise correlations that decay with distance on a similar length
scale (Fig. 3C).

We computed the noise synergy using our re-summed approximation (2) for many
subgroups of nearby cells of different sizes (Fig. 3D). In this case it is not
possible to estimate mutual informations exactly because of limited data,
making it a good test case for the usefulness of our analytical formulas. To
correct for the bias stemming from noise in estimating correlations, we
subtracted the value obtained after shuffling individual cell activities
across repetitions. We observe that noise correlations impede information
transmission, by the order of 1 bit per neuron per second, for a total
information of around 10 bits per neuron per second. It should be stressed
however that this result is specific to the white-noise stimulus statistics
considered here, and may not be a general feature of retinal processing, as
other stimulus statistics would change both the nature of stimulus
correlations and the input-output relationship as the network adapts.

We also used our method to study the effect of spiking memory in single
neurons, by treating the spike activity of the same neuron in $N$ consecutive
time bins as our activity vector $(n_{1},\ldots,n_{N})$ (treating time bins as
we treated individual neurons previously, see Fig. 4A). Stimulus temporal
auto-correlations are positive for about 50ms (Fig. 4B), then become negative
and go to zero for longer times (not shown). Noise temporal correlations are
driven by refractoriness, which suppresses activity immediately following a
spike, and by burstiness, which induces rippling effects up to 50 ms (Fig.
4C). We find that these correlations improve information transmission by up to
8 bits per second (Fig. 4D), almost doubling it for some cells. This suggests
that information is encoded not just in the average spike rate, but also
through the control of inter-spike timing, consistent with previous findings
Nemenman et al. (2008); Ferrari et al. (2018a); Botella-Soler et al. (2018).

Figure 4: Application on retinal temporal response to visual stimulation. A)
We build a pseudo population of neurons to describe the spiking history of
single neurons. B) Stimulus autocorrelations for different cells responding to
a white noise stimulation. Red and blue lines correspond to two example cells.
C) Same as (B) but for noise autocorrelations. D) Noise synergy for different
cells plotted against increasing temporal integration length.

Discussion. Despite being based on a small-correlation expansion, our
analytical predictions, especially (2), work well even in the presence of
strong correlations, which are ubiquitous in neuroscience Brivanlou et al.
(1998); Schneidman et al. (2006); Moreno-Bote et al. (2014). We showed how our
results can be applied to same-time correlations between neurons, or to neuron
autocorrelations, and they can readily be used on general spatial-temporal
correlations.

Our work shares some connections with previous efforts to estimate or
interpret information in population codes Brunel and Nadal (1998); Panzeri et
al. (1999); Sompolinsky et al. (2001); Pola et al. (2003); Hu et al. (2014);
Wei and Stocker (2016); Azeredo da Silveira and Rieke (2021). Ref. Pola et al.
(2003) proposes decompositions of the mutual information with different
interpretations, but does not provide ways to estimate it. Refs. Brunel and
Nadal (1998) and Hu et al. (2014); Wei and Stocker (2016) are mostly based on
the Fisher information, which in some limit can be related to the mutual
information. While the first term of our simpler expression (4) recovers one
of their main results—the so-called sign-rule—second and higher order terms in
the noise correlation parameter provide important corrections when
correlations are high, as can be seen from deviations from the initial slope
in Fig. 2B. In Panzeri et al. (1999) the authors developed a small time-bin
expansion of the mutual information. Expanding their results for small
correlations (and further assuming Poisson distributed spike counts, see App.
D) gives back our second-order expression (1). Our method however does not
need to assume small time bins, and still works well for large correlations.
Ref. Sompolinsky et al. (2001) provides estimate of the mutual information
when the neuronal responses are correlated but have only small fluctuations
around a large mean activity, which is not appropriate for small time bins or
for low spike rates as in the retina.

Our results are based on the small correlation expansion developed in Sessak
and Monasson (2009). In order to apply this theoretical tool, we assumed that
both the stimulus-conditioned and the marginal responses follow a pairwise
maximum entropy distribution. These models are characterized by many unknown
parameters that in principle need to be inferred from data. However the final
expressions for the mutual information contain only quantities that can be
directly estimated from data, without needing any inference. This makes our
approximations ready and easy to use, without requiring much computational
efforts. We showed that it works well even when the data was generated with a
very different model. Maximum entropy distributions are actually a series of
approximations which, just like Taylor expansions, can be refined by adding
higher-order correlations. A future direction could be to compute corrective
terms to the mutual information corresponding to third- and higher-order
correlation functions, rather than just pairwise correlations as we did in
this work. At the same time, the pairwise approximation has proven very
accurate for both marginal Schneidman et al. (2006); Ganmor et al. (2011);
Tkacik et al. (2014); Ganmor et al. (2015); Ferrari et al. (2017) and
conditional Shimazaki et al. (2012); Granot-Atedgi et al. (2013); Ferrari et
al. (2018b); Delamare and Ferrari (2022) responses of populations of neurons,
and is only expected to break down for very large densely correlated
populations Roudi et al. (2009). We thus expect our results to be applicable
to a wide array of neuronal contexts.

## Acknowledgments

We thank Tobias Kühn for useful discussions. This work was supported by the
Agence Nationale de la Recherche (ANR-21-CE37-0024 NatNetNoise), by LabEx
LIFESENSES (ANR-10-LABX-65), by IHU FOReSIGHT (ANR-18-IAHU-01), by Sorbonne
Université with the Emergence program (CrInforNet), and by Sorbonne Center for
Artificial Intelligence - Sorbonne University - IDEX SUPER 11-IDEX-0004.

## References

  * Brivanlou et al. (1998) I. Brivanlou, D. Warland, and M. Meister, Neuron  20, 527 (1998).
  * Trenholm et al. (2014) S. Trenholm, A. J. McLaughlin, D. Schwab, M. Turner, R. Smith, F. Rieke, and G. Awatramani, Nature neuroscience 17, 1759 (2014).
  * Schneidman et al. (2006) E. Schneidman, M. Berry, R. Segev, and W. Bialek, Nature  440, 1007 (2006).
  * Shlens et al. (2008) J. Shlens, F. Rieke, and E. Chichilnisky, Current opinion in neurobiology 18, 396 (2008).
  * Trong and Rieke (2008) P. K. Trong and F. Rieke, Nature neuroscience 11, 1343 (2008).
  * Ala-Laurila et al. (2011) P. Ala-Laurila, M. Greschner, E. Chichilnisky, and F. Rieke, Nature neuroscience 14, 1309 (2011).
  * Deweese and Zador (2004) M. R. Deweese and A. M. Zador, Journal of Neurophysiology 92, 1840 (2004), pMID: 15115790.
  * Lin et al. (2015) I.-C. Lin, M. Okun, M. Carandini, and K. Harris, Neuron 87, 644 (2015), ISSN 0896-6273.
  * Kohn and Smith (2005) A. Kohn and M. A. Smith, Journal of Neuroscience 25, 3661 (2005).
  * Montani et al. (2007) F. Montani, A. Kohn, M. A. Smith, and S. R. Schultz, Journal of Neuroscience 27, 2338 (2007), ISSN 0270-6474.
  * Smith and Kohn (2008) M. A. Smith and A. Kohn, Journal of Neuroscience 28, 12591 (2008).
  * Ponce-Alvarez et al. (2013) A. Ponce-Alvarez, A. Thiele, T. D. Albright, G. R. Stoner, and G. Deco, Proceedings of the National Academy of Sciences 110, 13162 (2013).
  * Franke et al. (2016) F. Franke, M. Fiscella, M. Sevelev, B. Roska, A. Hierlemann, and R. da Silveira, Neuron 89, 409 (2016).
  * Zylberberg et al. (2016) J. Zylberberg, J. Cafaro, M. Turner, E. Shea-Brown, and F. Rieke, Neuron 89, 369 (2016).
  * Meister et al. (1995) M. Meister, L. Lagnado, and D. A. Baylor, Science 270, 1207 (1995).
  * Nirenberg et al. (2001) S. Nirenberg, S. M. Carcieri, A. L. Jacobs, and P. E. Latham, Nature 411, 698 (2001).
  * Pillow et al. (2008) J. Pillow, J. Shlens, L. Paninski, A. Sher, A. Litke, E. J. Chichilnisky, and E. Simoncelli, Nature  454, 995 (2008).
  * Ruda et al. (2020) K. Ruda, J. Zylberberg, and G. D. Field, Nature communications 11, 1 (2020).
  * Hazon et al. (2022) O. Hazon, V. Minces, D. P. Tomàs, S. Ganguli, M. J. Schnitzer, and P. E. Jercog, Nature Communications 13 (2022).
  * Boffi et al. (2022) J. C. Boffi, B. Bathellier, H. Asari, and R. Prevedel, bioRxiv (2022).
  * Zohary et al. (1994) E. Zohary, M. N. Shadlen, and W. T. Newsome, Nature 370, 140 (1994).
  * Brunel and Nadal (1998) N. Brunel and J.-P. Nadal, Neural computation 10, 1731 (1998).
  * Abbott and Dayan (1999) L. F. Abbott and P. Dayan, Neural computation 11, 91 (1999).
  * Panzeri et al. (1999) S. Panzeri, A. Treves, S. Schultz, and E. T. Rolls, Neural Computation 11, 1553 (1999).
  * Sompolinsky et al. (2001) H. Sompolinsky, H. Yoon, K. Kang, and M. Shamir, Physical Review E 64, 051904 (2001).
  * Pola et al. (2003) G. Pola, A. Thiele, K. Hoffmann, and S. Panzeri, Network: Computation in Neural Systems 14, 35 (2003).
  * Tkacik et al. (2010) G. Tkacik, J. S. Prentice, V. Balasubramanian, and E. Schneidman, Proceedings of the National Academy of Sciences 107, 14419 (2010).
  * Ecker et al. (2010) A. S. Ecker, P. Berens, G. A. Keliris, M. Bethge, N. K. Logothetis, and A. S. Tolias, science 327, 584 (2010).
  * Ecker et al. (2011) A. S. Ecker, P. Berens, A. S. Tolias, and M. Bethge, Journal of Neuroscience 31, 14272 (2011), ISSN 0270-6474.
  * Hu et al. (2014) Y. Hu, J. Zylberberg, and E. Shea-Brown, PLoS computational biology 10, e1003469 (2014).
  * Moreno-Bote et al. (2014) R. Moreno-Bote, J. Beck, I. Kanitscheider, X. Pitkow, P. Latham, and A. Pouget, Nature neuroscience 17, 1410 (2014).
  * Azeredo da Silveira and Rieke (2021) R. Azeredo da Silveira and F. Rieke, Annual Review of Neuroscience 44, 403 (2021).
  * Jaynes (1982) E. T. Jaynes, Proc. IEEE  70, 939 (1982).
  * Tang et al. (2008) A. Tang, D. Jackson, J. Hobbs, W. Chen, J. L. Smith, H. Patel, A. Prieto, D. Petrusca, M. I. Grivich, A. Sher, et al., J. Neurosci. 28, 505 (2008).
  * Roudi et al. (2009) Y. Roudi, S. Nirenberg, and P. E. Latham, PLoS computational biology 5, e1000380 (2009).
  * Shimazaki et al. (2012) H. Shimazaki, S.-i. Amari, E. N. Brown, and S. Grün, PLoS computational biology 8, e1002385 (2012).
  * Watanabe et al. (2013) T. Watanabe, S. Hirose, H. Wada, Y. Imai, T. Machida, I. Shirouzu, S. Konishi, Y. Miyashita, and N. Masuda, Nat. Commun. 4, 1370 (2013).
  * Granot-Atedgi et al. (2013) E. Granot-Atedgi, G. Tkacik, R. Segev, and E. Schneidman, PLOS Computational Biology 9, 1 (2013).
  * Tkacik et al. (2013) G. Tkacik, O. Marre, T. Mora, D. Amodei, M. Berry, and W. Bialek, Journal of Statistical Mechanics: Theory and Experiment 2013, P03011 (2013).
  * Tkacik et al. (2014) G. Tkacik, O. Marre, D. Amodei, E. Schneidman, W. Bialek, and B. M.J, PloS Comput. Biol. 10(1), e1003408 (2014).
  * Tkacik et al. (2015) G. Tkacik, T. Mora, O. Marre, D. Amodei, S. E. Palmer, M. J. Berry, and W. Bialek, Proceedings of the National Academy of Sciences 112, 11508 (2015).
  * Cofre and Cessac (2014) R. Cofre and B. Cessac, Physical Review E 89, 052117 (2014).
  * Mora et al. (2015) T. Mora, S. Deny, and O. Marre, Phys. Rev. Lett.  114, 078105 (2015).
  * Shimazaki et al. (2015) H. Shimazaki, K. Sadeghi, T. Ishikawa, Y. Ikegaya, and T. Toyoizumi, Scientific reports 5, 9821 (2015).
  * Tavoni et al. (2017) G. Tavoni, U. Ferrari, F. Battaglia, S. Cocco, and R. Monasson, Network Neuroscience pp. 1–27 (2017).
  * Meshulam et al. (2017) L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, and W. Bialek, Neuron 96, 1178 (2017).
  * Ferrari et al. (2017) U. Ferrari, T. Obuchi, and T. Mora, Phys. Rev. E 95, 042321 (2017).
  * Nghiem et al. (2018) T.-A. Nghiem, B. Telenczuk, O. Marre, A. Destexhe, and U. Ferrari, Physical Review E 98, 012402 (2018).
  * Cayco-Gajic et al. (2018) N. A. Cayco-Gajic, J. Zylberberg, and E. Shea-Brown, Entropy 20, 489 (2018).
  * Delamare and Ferrari (2022) G. Delamare and U. Ferrari, bioRxiv (2022).
  * Sessak and Monasson (2009) V. Sessak and R. Monasson, J. Phys. A: Math. Theor.  42, 055001 (2009).
  * Shannon (1948) C. E. Shannon, Bell system technical journal 27, 379 (1948).
  * Rieke et al. (1999) F. Rieke, D. Warland, R. d. R. Van Steveninck, and W. Bialek, _Spikes: exploring the neural code_ (MIT press, 1999).
  * Strong et al. (1998) S. P. Strong, R. Koberle, R. R. D. R. Van Steveninck, and W. Bialek, Physical review letters 80, 197 (1998).
  * Kühn and van Wijland (2022) T. Kühn and F. van Wijland, arXiv preprint arXiv:2207.11302 (2022).
  * Cocco and Monasson (2011) S. Cocco and R. Monasson, Phys. Rev. Lett.  106, 090601 (2011).
  * Mahuas et al. (2020) G. Mahuas, G. Isacchini, O. Marre, U. Ferrari, and T. Mora, Advances in neural information processing systems 33, 5070 (2020).
  * Deny et al. (2017) S. Deny, U. Ferrari, E. Mace, P. Yger, R. Caplette, S. Picaud, G. Tkačik, and O. Marre, Nature communications 8, 1964 (2017).
  * Sorochynskyi et al. (2021) O. Sorochynskyi, S. Deny, O. Marre, and U. Ferrari, PLoS computational biology 17, e1008501 (2021).
  * Nemenman et al. (2008) I. Nemenman, G. D. Lewen, W. Bialek, and R. R. de Ruyter van Steveninck, PLoS computational biology 4, e1000025 (2008).
  * Ferrari et al. (2018a) U. Ferrari, S. Deny, O. Marre, and T. Mora, Neural Computation 30 (2018a).
  * Botella-Soler et al. (2018) V. Botella-Soler, S. Deny, G. Martius, O. Marre, and G. Tkačik, PLoS computational biology 14, e1006057 (2018).
  * Wei and Stocker (2016) X.-X. Wei and A. A. Stocker, Neural computation 28, 305 (2016).
  * Ganmor et al. (2011) E. Ganmor, R. Segev, and E. Schneidman, Journal of Neuroscience 31, 3044 (2011).
  * Ganmor et al. (2015) E. Ganmor, R. Segev, and E. Schneidman, Elife 4, e06134 (2015).
  * Ferrari et al. (2018b) U. Ferrari, S. Deny, M. Chalk, G. Tkačik, O. Marre, and T. Mora, Physical Review E 98, 042410 (2018b).
  * Cocco and Monasson (2012) S. Cocco and R. Monasson, Journal of Statistical Physics 147, 252 (2012).
  * Macke et al. (2011) J. H. Macke, I. Murray, and P. E. Latham (2011).
  * Montemurro et al. (2007) M. A. Montemurro, R. Senatore, and S. Panzeri, Neural Computation 19, 2913 (2007).

## References

  * (1)

##  Appendix A Second order approximation

###  A.1 Entropy

Following Sessak and Monasson (2009) we can expand at small correlations the
entropy of a population of $N$ binary neurons
$\mathbf{n}=\left(n_{1}...,n_{N}\right)$ (taking values $0$ or $1$). Let
$\mu_{i}$ be the mean of neuron $i$ and $\rho_{ij}$ the Pearson correlation of
neurons $i$ and $j$. The entropy can be expanded as a function of the small
correlation parameter as:

$\begin{split}H\left[\bf n\right]=&H_{\rm 0th}\left[\bf n\right]+H_{\rm
1st}\left[\bf n\right]+H_{\rm 2nd}\left[\bf n\right]+\ldots\end{split}$ (5)

The first term of this expansion $H_{\rm 0th}$ is the entropy of the neurons
in the total absence of correlations and is thus the sum of the entropies of
the single neurons. Adapting the calculation in Sessak and Monasson (2009) to
binary neurons taking values 0 or 1 yields:

$\begin{split}H_{\rm 0th}\left[\bf
n\right]=&-\sum_{i}(1-\mu_{i})\log{(1-\mu_{i})}+\mu_{i}\log{(\mu_{i})}.\end{split}$
(6)

The first order contribution in the correlation $H_{\rm 1st}$ vanishes and the
second order contribution of the correlations to the entropy is given by
Sessak and Monasson (2009):

$\begin{split}H_{\rm 2nd}\left[\bf
n\right]=&-\frac{1}{2}\sum_{i<j}\rho_{ij}^{2}.\end{split}$ (7)

###  A.2 Mutual Information

We can perform this expansion for the entropy of the marginal distribution
$H\left[\bf n\right]$ as well as for the entropies of the conditional
distributions $H\left[\bf n|\mathbf{s}\right]$, and hence calculate the mutual
information to the second order in the correlations:

$\begin{split}I=I_{\rm 0th}+I_{\rm 2nd},\end{split}$ (8)

where $I_{\rm 0th}$ is the sum of information carried by the neurons
individually $I_{\rm 0th}=\sum_{i}\left[H[{n_{i}}]-\langle H[n_{i}|{\bf
s}]\rangle_{\bf s}\right]$:

$\begin{split}I_{\rm
0th}=&-\sum_{i}(1-\mu_{i})\log{(1-\mu_{i})}+\mu_{i}\log{(\mu_{i})}\\\
&+\left<\sum_{i}(1-\mu_{i}(\mathbf{s}))\log{(1-\mu_{i}(\mathbf{s}))}+\mu_{i}(\mathbf{s})\log{(\mu_{i}(\mathbf{s}))}\right>_{\mathbf{s}},\end{split}$
(9)

and where the first non-zero contribution from the pairwise correlations in
the response is $I_{\rm 2nd}$:

$\begin{split}I_{\rm
2nd}=&-\frac{1}{2}\sum_{i<j}\left({\rho^{\mathrm{tot}}_{ij}}^{2}-\left<{\rho_{ij}^{\mathrm{n}}(\mathbf{s})}^{2}\right>_{\mathbf{s}}\right).\end{split}$
(10)

###  A.3 Noise synergy

Correlations in the marginal response
$\rho_{ij}^{\text{tot}}=r_{ij}^{\text{s}}+r_{ij}^{\text{n}}$ boil down to
their stimulus contribution in the conditionally independent case
$\rho_{ij}^{\text{tot, CI}}=r_{ij}^{\text{s}}$. Thus the noise synergy $\Delta
I\equiv I-I_{\text{CI}}$ is given at second oder by:

$\displaystyle\Delta
I\approx\sum_{i<j}\left[-r_{ij}^{\mathrm{n}}r_{ij}^{\mathrm{s}}+\frac{1}{2}\left(\left<{\rho_{ij}^{\mathrm{n}}(\mathbf{s})}^{2}\right>_{\mathbf{s}}-{r_{ij}^{\mathrm{n}}}^{2}\right)\right].$
(11)

In general the second term under the sum in the noise synergy, $\Delta
I_{ij}^{\rm quad}=\frac{1}{2}\left(\langle{\rho_{ij}^{\mathrm{n}}({\bf
s})}^{2}\rangle_{\bf s}-{r_{ij}^{\mathrm{n}}}^{2}\right)$ can be positive or
negative depending on the level of correlation between
${\rho_{ij}^{\mathrm{n}}({\bf s})}$ and $\sqrt{C_{ii}({\bf s})C_{jj}({\bf
s})}$, as $r_{ij}^{\mathrm{n}}=\langle\rho_{ij}^{\mathrm{n}}({\bf
s})\sqrt{C_{ii}({\bf s})C_{jj}({\bf s})}\rangle_{\bf s}/\sqrt{C_{ii}C_{jj}}$.
However, if we assume noise correlations are independent from the stimulus
$\rho_{ij}^{\mathrm{n}}(\mathbf{s})=\rho_{ij}^{\mathrm{n}}$, we can show that
$\Delta I_{ij}^{\rm
quad}=\frac{1}{2}\left({\rho_{ij}^{\mathrm{n}}}^{2}-{r_{ij}^{\mathrm{n}}}^{2}\right)$
is non-negative. First, the formulation of ${r_{ij}^{\mathrm{n}}}$ in terms of
${\rho_{ij}^{\mathrm{n}}}$ becomes:

${r_{ij}^{\mathrm{n}}}={\rho_{ij}^{\mathrm{n}}}\frac{\left<\sqrt{C_{ii}^{\rm
n}(\mathbf{s})C_{jj}^{\rm n}(\mathbf{s})}\right>_{\rm
s}}{\sqrt{C_{ii}C_{jj}}},$ (12)

which then gives:

$\Delta I_{ij}^{\rm
quad}=\frac{1}{2}{\rho_{ij}^{\mathrm{n}}}^{2}\left(1-\frac{\left<\sqrt{C_{ii}^{\rm
n}(\mathbf{s})C_{jj}^{\rm n}(\mathbf{s})}\right>_{\rm
s}^{2}}{C_{ii}C_{jj}}\right).$ (13)

According to the Cauchy-Schwartz inequality we have that
$\left<\sqrt{C_{ii}^{\rm n}(\mathbf{s})C_{jj}^{\rm n}(\mathbf{s})}\right>_{\rm
s}^{2}\leq\left<C_{ii}^{\rm n}(\mathbf{s})\right>_{\rm s}\left<C_{jj}^{\rm
n}(\mathbf{s})\right>_{\rm s}$. Besides, from the law of total variance we
have that $C_{ii}=C_{ii}^{\rm s}+\left<C_{ii}^{\rm n}(\mathbf{s})\right>_{\rm
s}$, thus $\left<C_{ii}^{\rm n}(\mathbf{s})\right>_{\rm s}\leq C_{ii}$ and
finally ${\left<\sqrt{C_{ii}^{\rm n}(\mathbf{s})C_{jj}^{\rm
n}(\mathbf{s})}\right>_{\rm s}^{2}}\leq{C_{ii}C_{jj}}$. Altogether this gives
that $\Delta I_{ij}^{\rm quad}\geq 0$.

##  Appendix B Resummed expansion

###  B.1 Entropy

In Sessak and Monasson (2009) it is shown that some of the terms in the small-
correlation expansion of the couplings can be resummed to yield a better
approximation. We can proceed in the exact same way for the entropy and resum
some of the diagrams in the small-correlation expansion. All the terms of the
second order approximation above are contained in the resummed expansion we
detail here. Note that this resummed expansion is equivalent to a cluster
expansion truncated to second order, with mean-field reference entropy Cocco
and Monasson (2012). It reads:

$\displaystyle H\left[\bf n\right]$ $\displaystyle\approx H_{\rm 0th}\left[\bf
n\right]+H_{\rm pairs}\left[\bf n\right]+H_{\mathcal{G}}\left[\bf
n\right]-H_{\rm dbl}\left[\bf n\right].$ (14)

The single site contribution $H_{\rm 0th}\left[\bf n\right]$ is the same as
above, and $H_{\rm pairs}\left[\bf n\right]$ corresponds to the entropy gain
of all pairs in the population taken independently compared to the single site
contribution. Interestingly, taking the sum of these two first contributions
amounts to making an independent pair approximation, which would be exact in
the case of a tree-like network topology. $H_{\rm pairs}\left[\bf n\right]$ is
a sum over all pairs of neurons in the population:

$\displaystyle H_{\rm pairs}\left[\bf n\right]=\sum_{i<j}H[n_{i},n_{j}],$ (15)

with $H[n_{i},n_{j}]$ the entropy gain of pair $(ij)$ compared to the single
neurons case:

$\begin{split}H[n_{i},n_{j}]=&-(C_{ij}+\mu_{i}\mu_{j})\log{(1+\frac{C_{ij}}{\mu_{i}\mu_{j}})}\\\
&+(C_{ij}+\mu_{i}(\mu_{j}-1))\log{(1+\frac{C_{ij}}{\mu_{i}(\mu_{j}-1)})}\\\
&+(C_{ij}+\mu_{j}(\mu_{i}-1))\log{(1+\frac{C_{ij}}{\mu_{j}(\mu_{i}-1)})}\\\
&-(C_{ij}+(1-\mu_{i})(1-\mu_{j}))\log{(1+\frac{C_{ij}}{(1-\mu_{i})(1-\mu_{j})})}.\end{split}$
(16)

The second resummed term of this expansion $H_{\mathcal{G}}\left[\bf n\right]$
corresponds to the contribution of interactions to the entropy in the mean-
field approximation. It contains the resummation of all loop diagrams in the
expansion and amounts to assuming the entropic contribution of pairwise
correlations is Gaussian. Noting $\rho$ the correlation matrix of $\bf n$ we
get:

$\displaystyle H_{\mathcal{G}}\left[\bf
n\right]=\frac{1}{2}\log\left(\left|\bf\rho\right|\right).$ (17)

Finally there are some terms in the expansion that are resummed both in
$H_{\rm pairs}\left[\bf n\right]$ and $H_{\mathcal{G}}\left[\bf n\right]$,
therefore we need to substract them once from the expansion through $H_{\rm
dbl}\left[\bf n\right]$. The double counted terms simply correspond to the
Gaussian approximation applied to each pair of neurons:

$\displaystyle H_{\rm dbl}\left[\bf
n\right]=\frac{1}{2}\sum_{i<j}\log\left(1-\rho_{ij}^{2}\right).$ (18)

###  B.2 Mutual Information

Applying the resummed approximation to the marginal and conditional responses
results in a resummed approximation for the mutual information:

$\displaystyle I$ $\displaystyle\approx I_{\rm 0th}+I_{\rm
pairs}+I_{\mathcal{G}}-I_{\rm dbl}.$ (19)

With $I_{\rm pairs}$ the contribution of pairwise correlations to the mutual
information in the independent pairs approximation. If we write
$C_{ij}\equiv\mathrm{Cov}(n_{i},n_{j})$ the total covariance across stimuli
and $C_{ij}^{\rm n}({\bf s})\equiv\mathrm{Cov}(n_{i},n_{j}|{\bf s})$ the
covariance at a given stimulus $\bf s$:

$\begin{split}I_{\rm
pairs}=&\sum_{i<j}\biggl{[}-(C_{ij}+\mu_{i}\mu_{j})\log{(1+\frac{C_{ij}}{\mu_{i}\mu_{j}})}\\\
&+(C_{ij}+\mu_{i}(\mu_{j}-1))\log{(1+\frac{C_{ij}}{\mu_{i}(\mu_{j}-1)})}\\\
&+(C_{ij}+\mu_{j}(\mu_{i}-1))\log{(1+\frac{C_{ij}}{\mu_{j}(\mu_{i}-1)})}\\\
&-(C_{ij}+(1-\mu_{i})(1-\mu_{j}))\log{(1+\frac{C_{ij}}{(1-\mu_{i})(1-\mu_{j})})}\\\
&-\biggl{<}-(C_{ij}^{\mathrm{n}}(\mathbf{s})+\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s}))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})})}\\\
&+(C_{ij}^{\mathrm{n}}(\mathbf{s})+\mu_{i}(\mathbf{s})(\mu_{j}(\mathbf{s})-1))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{\mu_{i}(\mathbf{s})(\mu_{j}(\mathbf{s})-1)})}\\\
&+(C_{ij}^{\mathrm{n}}(\mathbf{s})+\mu_{j}(\mathbf{s})(\mu_{i}(\mathbf{s})-1))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{\mu_{j}(\mathbf{s})(\mu_{i}(\mathbf{s})-1)})}\\\
&-(C_{ij}^{\mathrm{n}}(\mathbf{s})+(1-\mu_{i}(\mathbf{s}))(1-\mu_{j}(\mathbf{s})))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{(1-\mu_{i}(\mathbf{s}))(1-\mu_{j}(\mathbf{s}))})}\biggr{>}_{\mathbf{s}}\biggr{]}.\end{split}$
(20)

If we denote by $\rho^{\rm tot}$ the total correlation matrix across stimuli
and $\rho^{\rm n}\left(\mathbf{s}\right)$ the correlation matrix at given
stimulus $\bf s$, the Gaussian (i.e. mean-field) contribution of correlations
to the mutual information takes the simple form:

$\displaystyle
I_{\mathcal{G}}=\frac{1}{2}\log{\left(|\rho^{\mathrm{tot}}|\right)}-\frac{1}{2}\left<\log{\left(|\rho^{\mathrm{n}}(\mathbf{s})|\right)}\right>_{\mathbf{s}},$
(21)

while the double counting correction becomes:

$\displaystyle I_{\rm
dbl}=\frac{1}{2}\sum_{i<j}\biggl{[}\log\left(1-{\rho_{ij}^{\rm
tot}}^{2}\right)-\left<\log\left(1-{\rho_{ij}^{\rm n}}\left(\bf
s\right)^{2}\right)\right>_{\mathbf{s}}\biggr{]}.$ (22)

###  B.3 Noise synergy

Likewise we can write a resummed approximation of the noise synergy:

$\displaystyle\Delta I$ $\displaystyle\approx\Delta I_{\rm pairs}+\Delta
I_{\mathcal{G}}-\Delta I_{\rm dbl}.$ (23)

Noting that the total covariance across stimuli can be decomposed in terms of
noise covariance and stimulus covariance $C_{ij}=C_{ij}^{\rm s}+C_{ij}^{\rm
n}$, with $C_{ij}^{\rm n}=\langle C_{ij}^{\rm n}(\mathbf{s})\rangle_{\rm s}$,
the independent pairs approximation of the noise synergy $\Delta I_{\rm
pairs}$ would be given by:

$\begin{split}\Delta I_{\rm pairs}=&\sum_{i<j}\biggl{[}-C_{ij}^{\rm
n}\log(1+\frac{C_{ij}}{(\mu_{i}(\mu_{j}-1)+C_{ij})(\mu_{j}(\mu_{i}-1)+C_{ij})})\\\
&-(C_{ij}^{\mathrm{s}}+\mu_{i}\mu_{j})\log(1+\frac{C_{ij}^{\rm
n}}{\mu_{i}\mu_{j}+C_{ij}^{\mathrm{s}}})\\\
&+(C_{ij}^{\mathrm{s}}+\mu_{i}(\mu_{j}-1))\log(1+\frac{C_{ij}^{\rm
n}}{\mu_{i}(\mu_{j}-1)+C_{ij}^{\mathrm{s}}})\\\
&+(C_{ij}^{\mathrm{s}}+\mu_{j}(\mu_{i}-1))\log(1+\frac{C_{ij}^{\rm
n}}{\mu_{j}(\mu_{i}-1)+C_{ij}^{\mathrm{s}}})\\\
&-(C_{ij}^{\mathrm{s}}+(1-\mu_{i})(1-\mu_{j}))\log(1+\frac{C_{ij}^{\rm
n}}{(1-\mu_{i})(1-\mu_{j})+C_{ij}^{\mathrm{s}}})\\\
&-\biggl{<}-(C_{ij}^{\mathrm{n}}(\mathbf{s})+\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s}))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})})}\\\
&+(C_{ij}^{\mathrm{n}}(\mathbf{s})+\mu_{i}(\mathbf{s})(\mu_{j}(\mathbf{s})-1))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{\mu_{i}(\mathbf{s})(\mu_{j}(\mathbf{s})-1)})}\\\
&+(C_{ij}^{\mathrm{n}}(\mathbf{s})+\mu_{j}(\mathbf{s})(\mu_{i}(\mathbf{s})-1))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{\mu_{j}(\mathbf{s})(\mu_{i}(\mathbf{s})-1)})}\\\
&-(C_{ij}^{\mathrm{n}}(\mathbf{s})+(1-\mu_{i}(\mathbf{s}))(1-\mu_{j}(\mathbf{s})))\log{(1+\frac{C_{ij}^{\mathrm{n}}(\mathbf{s})}{(1-\mu_{i}(\mathbf{s}))(1-\mu_{j}(\mathbf{s}))})}\biggr{>}_{\mathrm{s}}\biggr{]}.\end{split}$
(24)

The Gaussian contribution to the noise synergy takes again a simple form:

$\begin{split}\Delta
I_{\mathcal{G}}&=\frac{1}{2}\log\left(\frac{|\mathbf{\rho}^{\mathrm{tot}}|}{|\mathbf{\rho}^{\mathrm{tot,CI}}|}\right)-\frac{1}{2}\left<\log\left(|\mathbf{\rho}^{\mathrm{n}}(\mathbf{s})|\right)\right>_{\mathbf{s}},\end{split}$
(25)

where we recall that correlations in the marginal response can be expressed as
the sum of stimulus and noise contributions
$\mathbf{\rho}^{\mathrm{tot}}=r^{\text{s}}+r^{\text{n}}$. In the conditionally
independent case we have
$\mathbf{\rho}^{\mathrm{tot,CI}}=r^{\text{s}}+\nu^{\text{n}}$ with
$\nu^{\text{n}}$ the diagonal matrix containing the diagonal elements of
$r^{\text{n}}$.

Finally, the double counting correction to the noise synergy is given by:

$\begin{split}\Delta
I^{dbl}=&\frac{1}{2}\sum_{i<j}\biggl{[}\log{\left(\frac{1-{\rho^{\rm
tot}_{ij}}^{2}}{1-{r^{\mathrm{s}}_{ij}}^{2}}\right)}-\left<\log{\left(1-{\rho_{ij}^{\mathrm{n}}(\mathbf{s})}^{2}\right)}\right>_{\mathrm{s}}\biggr{]}.\end{split}$
(26)

##  Appendix C Generalized linear model simulations

The model used to generate the synthetic data is a Generalized Linear Model
Pillow et al. (2008) with sigmoidal nonlinearity. The number of spikes emitted
by cell $i$ in time bin $t$ of size $dt=1$ ms follows a Bernoulli distribution
with mean $\lambda_{i}\left(t\right)$, given by:

$\displaystyle\lambda_{i}\left(t\right)=\left(1+\exp{\left(-h_{i}\left(t\right)\right)}\right)^{-1},$
$\displaystyle h_{i}\left(t\right)=h_{i}^{\rm bias}+h_{i}^{\rm
stim}\left(t\right)+h_{i}^{\rm int}\left(t\right)+h_{i}^{\rm
corr}\left(t\right),$ (27)

where $h_{i}^{\rm bias}$ sets the baseline firing rate of the cell,
$h_{i}^{\rm stim}\left(t\right)$ accounts for how the stimulus drives the
cell’s activity, $h_{i}^{\rm int}\left(t\right)$ accounts for the effect of
couplings and self-coupling, while $h_{i}^{\rm corr}\left(t\right)$ is here to
correct for the contribution of the neuron-neuron couplings to the firing
rate.

The stimulus $S$ is a movie of dimensions $(N_{x},N_{y},N_{t})$ where $N_{x}$
and $N_{y}$ correspond to the two spatial dimensions in pixels, and $N_{t}$ is
the temporal length of the stimulus in number of time bins. We simulated $5$
cells organized on a triangular lattice as represented in Fig. S1B, spaced by
$\xi=4$ pixels. Here the stimulus consisted of $N_{t}$ zero-mean, 2D Gaussian
frames of size $(N_{x},N_{y})$ with covariance function $C({\bf u},{\bf
v})=\delta_{{\bf u},{\bf v}}+(1-\delta_{{\bf u},{\bf
v}})c_{0}\exp{\left(-\|{\bf u}-{\bf v}\|/\lambda\right)}$, where
$\lambda=2\xi$ and $c_{0}$ is varied to change the level of stimulus
correlation in the response. There was no correlations between frames and they
were refreshed at 100Hz.

Figure S1: Parameters of the GLM model. The different parameters of the model
were chosen to be biologically plausible and mimick those of retinal ganglion
cells. A) The spatial component of the spatio-temporal stimulus filter $K^{\rm
space}$ is made of a difference of Gaussians Eq. 30. B) The 5 cells are
arranged according to a triangular lattice. We represented here the receptive
fields of the cells by circles that correspond to the contour where the center
and the surround of the spatial components compensate exactly. Only nearest
neighbours in this population are coupled by non-zero $J_{ij}$ coupling
filters. C) Profile view of the spatial components cut in $y=y^{*}$ on panel
B. D) The temporal component of the stimulus filter has a biphasic profile and
consists of a difference of two raised cosine functions Eq. 29. E) Coupling
filters are non zero for nearest neighbours and are defined by Eq. 34. The
increasing coupling amplitude shown here induces increasing noise correlations
in the response of the cells. F) The self coupling filter, accounting for the
effect of the neuron’s own spiking history, induced refractory effects over
$10$ ms in the past.

The cells’ stimulus filters, of size $(N_{x},N_{y},N_{t}^{\rm stim})$ were
built from the product of a temporal and spatial component which were chosen
to mimick the properties of natural retinal ganglion cells. The temporal
component $K^{\rm time}(\tau)$ is illustrated Fig. S1D and consists of a
difference of two raised cosine functions:

${\rm
rc}(\tau,s,c)=\begin{cases}\cos((\log(\tau+s)-c)\pi)/2+1/2&\text{when}-1\leq\log(\tau+s)-c\leq
1\\\ \\\ 0&\text{otherwise},\end{cases}$ (28)

such that:

$K^{\rm time}(\tau)=a_{1}{\rm rc}(\tau,s,c_{1})-a_{2}{\rm rc}(\tau,s,c_{2}),$
(29)

with $s=50$, $a_{1}=0.35$, $c_{1}=5.3$, $a_{2}=1.15$ and $c_{2}=4.8$. The
spatial component is made of a difference of Gaussian functions and exhibits a
positive center and a negative surround, as shown on Fig. S1A and Fig. S1C.
Noting ${\bf u^{\rm c}_{i}}=(x^{\rm c}_{i},y^{\rm c}_{i})$ the spatial
position of the receptive field center of cell $i$:

$K^{\rm space}_{i}({\bf u})=\frac{1}{r-1}\left[r\exp\left(-\frac{\|{\bf
u}-{\bf u^{\rm c}_{i}}\|^{2}}{2\sigma_{\rm
center}}\right)-\exp\left(-\frac{\|{\bf u}-{\bf u^{\rm
c}_{i}}\|^{2}}{2\sigma_{\rm surround}}\right)\right]$ (30)

where $\sigma_{\rm center}=2$ pixel, $\sigma_{\rm surround}=2.1$ pixel, and
$r=1.12$. The firing rate variance is fixed through a parameter $\alpha_{\rm
s}$ (set to $0.5$ in our synthetic experiments) such that $h_{i}^{\rm
stim}\left(t\right)=\alpha_{\rm s}\cdot z\text{-score}\left(\tilde{h}_{i}^{\rm
stim}\left(t\right)\right)$, where $\tilde{h}_{i}^{\rm stim}\left(t\right)$ is
given by the temporal convolution of stimulus $S$ by the spatio-temporal
stimulus filter:

$h_{i}^{\rm stim}\left(t\right)=\sum_{\tau>0}\sum_{\bf u}K_{i}^{\rm
space}({\bf u})K^{\rm time}(\tau)S({\bf u},t-\tau).$ (31)

Likewise, the spiking history contribution of cell $i$ itself as well as that
of the other cells in the network are accounted for by linear convolutions of
the spiking histories by a set of temporal coupling filters:

$\displaystyle h_{i}^{\rm
int}\left(t\right)=\sum_{j}\sum_{\tau>0}J_{ij}\left(\tau\right)n_{j}(t-\tau),$
(32)

where the self-coupling filters $J_{ii}(\tau)$, shown Fig. S1F, are given by:

$J_{ii}(\tau)=\begin{cases}J_{\rm self}^{0}&\text{if}~{}\tau\leq\tau_{\rm
refr,}\\\ \\\ 0&\text{otherwise,}\end{cases}$ (33)

with $J_{\rm self}^{0}=-10$ and $\tau_{\rm refr}=10\text{ms}$. The neuron-
neuron (i.e. $i\neq j$) couplings $J_{ij}$ follow:

$J_{ij}(\tau)=\begin{cases}J_{\rm
coupl}^{0}~{}\tau\exp\left(-\tau\right)&\text{if i and j are nearest
neighbours,}\\\ \\\ 0&\text{otherwise,}\end{cases}$ (34)

where the coupling strength $J_{\rm coupl}^{0}$ can be varied (as illustrated
on Fig. S1E) to change the amount of noise correlations in the response.

Figure S2: Results of the iterative inference approach. The different
parameters of the model were chosen to be biologically plausible and mimick
those of retinal ganglion cells. A) For the three test stimuli chosen we see
that $\rho^{\rm s}$, the stimulus correlation averaged over all pairs of
neighbouring cells, is constant with respect to the coupling strength $J_{\rm
coupl}^{0}.$ B) Likewise, the stimulus variance averaged across the population
$V^{\rm s}=\langle C_{ii}^{\rm s}\rangle_{i}$ is independent of $J_{\rm
coupl}^{0}$.

In order to vary independently stimulus and noise correlations in the neurons’
response we introduced the field $h_{i}^{\rm corr}$ that corrects for the
contribution of neuron-neuron couplings to the firing rates of the cells. This
correction is needed because changing the strength of the couplings will not
only change the amount of noise correlations in the reponse, but also the
firing rates of the cells. This corrective field is computed via an iterative
inference approach built upon the 2-step inference method Mahuas et al.
(2020). The first step of this iterative procedure is the following: for a
given stimulus movie, we simulate $N_{\rm repe}$ times the response of the
conditionally-independent neurons (i.e. with $J_{\rm coupl}^{0}=0$). We then
fix the strength of neuron-neuron couplings $J_{\rm coupl}^{0}$ to the desired
value so as to induce noise correlations in the response of the cells. From
here we approximate the interaction field $\hat{h}_{i}^{\rm int}$ using the
simulated response and the chosen couplings amplitude $J_{\rm coupl}^{0}$.
Finally we infer the corrective field $h_{i}^{\rm corr}$ on the previously
simulated response similarly to the 2-step inference approach by minimzing the
following single neuron negative log-likelihood for each neuron in the
population:

$nLL_{i}=-\sum_{r=1}^{N_{\rm repe}}\sum_{t=1}^{N_{\rm t}}h_{i}^{\rm
corr}(t)n_{i}(t,r)+\log{\left(1-\hat{\lambda}_{i}(t,r)\right)},$ (35)

where $n_{i}(t,r)$ denotes the simulated response of neuron $i$ in time bin
$t$ and repetition $r$ and where
$\hat{\lambda}_{i}(t,r)=\left(1+\exp{\left(-h_{i}^{\rm bias}+h_{i}^{\rm
stim}(t)+\hat{h}_{i}^{\rm int}(t,r)+h_{i}^{\rm corr}(t)\right)}\right)^{-1}$.
The second step of the procedure consists in simulating the response of the
cells using the previously chosen coupling amplitude and the infered
corrective field $h_{i}^{\rm corr}\left(t\right)$. We can then re-estimate the
interaction field $\hat{h}_{i}^{\rm int}$ on these simulated data and infer
again $h_{i}^{\rm corr}$ to get a better approximation of the couplings’
contribution to the firing rate. This second step is repeated as many times as
needed to match the firing rates of the conditionally-dependent model to those
of the conditionally-independent model within the desired precision. For the
generation of the synthetic data used for testing the approximations we
systematically performed 8 successive inference steps. This approach resulted
for each given stimuli in constant stimulus correlations Fig. S2A as well as
constant stimulus variance Fig. S2B across coupling strengths.

The simulation of the model described above was repeated $N_{\rm repe}=2500$
times for each stimulus movie, then the Mutual Information was computed using
the second order and the resummed approximations as well as by the histogram
(or “exact”) method. In order to be able to compare the Mutual Information
computed via the approximations to that computed via the histogram method, we
first need to correct for the effect of sampling bias on these quantities.
From Macke et al. (2011) the bias in the entropy of a maximum entropy model
with $N_{\rm cells}$ and thus $\frac{N_{\rm cells}(N_{\rm cells}+1)}{2}$
constraints, evaluated on $N_{\rm samples}$ can be approximated by $b^{\rm
maxent}_{\rm H}=-\frac{N_{\rm cells}(N_{\rm cells}+1)}{4N_{\rm samples}}$. In
the case of the marginal entropy we have $N_{\rm samples}=N_{\rm repe}\times
N_{t}$, while in the case of the noise entropy we have $N_{\rm samples}=N_{\rm
repe}$. The sampling bias on the Mutual Information of a maximum entropy model
is thus given by:

$b^{\rm maxent}_{\rm I}=\frac{N_{\rm cells}(N_{\rm cells}+1)}{4N_{\rm
repe}}(1-\frac{1}{N_{t}}),$ (36)

which has then to be subtracted from the raw results of the 2nd order and the
resummed approximations. To evaluate the bias on the entropies evaluated via
the histogram method we used a shuffling approach similar to Montemurro et al.
(2007): we estimate the bias on the entropy $H[{\bf n}]$ as $b^{\rm
exact}_{\rm H}=H^{\rm shuffle}[{\bf n}]-(H^{\rm 0th}[{\bf n}]-b^{0th})$ where
$H^{\rm shuffle}[{\bf n}]$ is the entropy computed on the data shuffled so
that correlations between cells are destroyed, and where $H^{\rm 0th}[{\bf
n}]$ is the single site entropy contribution given by Eq.6. $H^{\rm 0th}[{\bf
n}]$ is also biased, so we need to correct it by $b^{\rm
0th}_{H}=-\frac{N_{\rm cells}}{2N_{\rm samples}}$ (as here we have only one
constraint per neuron). Applying this to the marginal and conditional
entropies gives the following bias for the Mutual Information computed via the
exact method:

$b^{\rm exact}_{\rm I}=\left(H^{\rm shuffle}[{\bf n}]-\langle H^{\rm
shuffle}[{\bf n|s}]\rangle_{\rm s}\right)-\left(H^{\rm 0th}[{\bf n}]-\langle
H^{\rm 0th}[{\bf n|s}]\rangle_{\rm s}\right)+\frac{N_{\rm cells}}{2N_{\rm
repe}}(1-\frac{1}{N_{t}}),$ (37)

which has to be subtracted from the raw result of the histogram method.

##  Appendix D Link to Panzeri et al. 1999

###  D.1 Small time bin expansion

Panzeri et al. Panzeri et al. (1999) introduced a small time bin $t$ expansion
of the mutual information:

$I=tI^{t}+\frac{t^{2}}{2}I^{tt}+...$ (38)

They quantify noise correlations and stimulus correlations in terms of
$n_{i}(\mathbf{s})$ the spike count variable at fixed stimulus $s$,
$\mu_{i}(\mathbf{s})=\overline{n_{i}(\mathbf{s})}$ its average over the noise,
$\mu_{i}=\left<\mu_{i}(\mathbf{s})\right>_{\bf s}$ the average firing rate
across stimuli and use the according rates $n_{i}(\mathbf{s})/t$,
$\mu_{i}(\mathbf{s})/t$ and $\mu_{i}/t$ in the expression of the mutual
information. They introduce the “noise correlation density”:

$\gamma_{ij}(\mathbf{s})=\begin{cases}\frac{\overline{n_{i}(\mathbf{s})n_{j}(\mathbf{s})}}{\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})}-1&\text{if}\
i\neq j\\\ \\\
\frac{\overline{{n_{i}(\mathbf{s})}^{2}}-\mu_{i}(\mathbf{s})}{{\mu_{i}(\mathbf{s})}^{2}}-1&\text{if}\
i=j,\end{cases}$ (39)

and the “stimulus correlation density”:

$\nu_{ij}=\frac{\left<\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})\right>_{s}}{\mu_{i}\mu_{j}}-1.$
(40)

The first order contribution (sum over the single cell contributions) to the
mutual information in the small time bin expansion is:

$tI^{t}=\sum_{i}\left\langle\mu_{i}(\mathbf{s})\ln\frac{\mu_{i}(\mathbf{s})}{\mu_{i}}\right\rangle_{s},$
(41)

The second order contribution contains the correlations contributions:

$\frac{t^{2}}{2}I^{tt}=\frac{t^{2}}{2}\sum_{ij}I^{tt,{\rm(1)}}_{ij}+I^{tt,{\rm(2)}}_{ij}+I^{tt,{\rm(3)}}_{ij},$
(42)

with

$\displaystyle I^{tt,{\rm(1)}}_{ij}=$
$\displaystyle\frac{1}{t^{2}}\mu_{i}\mu_{j}\left[\nu_{ij}+\left(1+\nu_{ij}\right)\ln\left(\frac{1}{1+\nu_{ij}}\right)\right],$
(43) $\displaystyle I^{tt,{\rm(2)}}_{ij}=$
$\displaystyle\frac{1}{t^{2}}\left\langle\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})\gamma_{ij}(\mathbf{s})\right\rangle_{s}\ln\left(\frac{1}{1+\nu_{ij}}\right),$
(44) $\displaystyle I^{tt,{\rm(3)}}_{ij}=$
$\displaystyle\frac{1}{t^{2}}\bigg{\langle}\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})\left(1+\gamma_{ij}(\mathbf{s})\right)\times\ln\left(\frac{\left(1+\gamma_{ij}(\mathbf{s})\right)\left\langle\mu_{i}\left(\mathbf{s}^{\prime}\right)\mu_{j}\left(\mathbf{s}^{\prime}\right)\right\rangle_{s^{\prime}}}{\left\langle\mu_{i}\left(\mathbf{s}^{\prime}\right)\mu_{j}\left(\mathbf{s}^{\prime}\right)\left(1+\gamma_{ij}\left(\mathbf{s}^{\prime}\right)\right)\right\rangle_{s^{\prime}}}\right)\bigg{\rangle}_{s}.$
(45)

The three contributions in $I^{tt}$ render the effects of stimulus and noise
correlations as well as interactions thereof Panzeri et al. (1999):
$I^{tt,{\rm(1)}}_{ij}$ contains the effect of signal correlations, while
$I^{tt,{\rm(2)}}_{ij}$ accounts for how (stimulus independent) noise
correlations interact with stimulus correlations and affect information, and
$I^{tt,{\rm(3)}}_{ij}$ contains information carried by the stimulus-dependency
of noise correlations.

###  D.2 Link to the small correlation expansion

The single site contribution of the small time bin approximation $tI_{\rm t}$
amounts to assuming information is conveyed only by spikes rather than by
spikes and silences together. To illustrate this, we can rewrite the
expression obtained previously for $I_{\rm 0th}$ by regrouping the noise
entropy and marginal entropy contributions separately for spikes on one hand,
and silences on the other hand:

$\begin{split}I_{\rm
0th}=\sum_{i}\left<\mu_{i}(\mathbf{s})\log{\left(\frac{\mu_{i}(\mathbf{s})}{\mu_{i}}\right)}\right>_{\mathbf{s}}+\sum_{i}\left<(1-\mu_{i}(\mathbf{s}))\log{\left(\frac{1-\mu_{i}(\mathbf{s})}{1-\mu_{i}}\right)}\right>_{\mathbf{s}}.\end{split}$
(46)

We see the first term in this rewriting of $I_{\rm 0th}$ corresponds to
$tI^{t}$. Further corrections to the single site information $I_{\rm t}$ are
found in the diagonal terms under the sum in $I^{tt}$. In the small rates
(i.e. Poisson) limit however, this correction is simplified as
$\gamma_{ii}(\mathbf{s})$ vanishes for $C_{ii}({\bf s})=\mu_{i}({\bf s})$. The
single site correction coming from $I^{tt}$ is therefore given by:

$\frac{t^{2}}{2}\sum_{i}I^{tt,{\rm(1)}}_{ii}=\frac{1}{2}\sum_{i}{\mu_{i}}^{2}\left[\nu_{ii}+\left(1+\nu_{ii}\right)\ln\left(\frac{1}{1+\nu_{ii}}\right)\right].$
(47)

The out of diagonal terms ($i\neq j$) in $I^{tt}$ account for the effect of
noise and stimulus cross-correlations. In the small rates and small
correlations limit, we show here that we recover the main result of this
paper. First we notice that in this limit:

$\displaystyle\nu_{ij}$ $\displaystyle=\frac{r_{ij}^{\rm
s}}{\sqrt{C_{ii}C_{jj}}}=\frac{r_{ij}^{\rm s}}{\sqrt{\mu_{i}\mu_{j}}},$ (48)
$\displaystyle\gamma_{ij}({\bf s})$ $\displaystyle=\frac{\rho_{ij}^{\rm
n}({\bf s})}{\sqrt{C_{ii}({\bf s})C_{jj}({\bf s})}}=\frac{\rho_{ij}^{\rm
n}({\bf s})}{\sqrt{\mu_{i}({\bf s})\mu_{j}({\bf s})}}.$ (49)

Replacing these expressions in $I^{tt,{\rm(1)}}_{ij}$, $I^{tt,{\rm(2)}}_{ij}$
and $I^{tt,{\rm(3)}}_{ij}$ then expanding the logarithms at small $r_{ij}^{\rm
s}$ and $\rho_{ij}^{\rm n}({\bf s})$ (and thus $r_{ij}^{\rm n}$) and
truncating at second order gives:

$\displaystyle\frac{t^{2}}{2}I^{tt,{\rm(1)}}_{ij}$
$\displaystyle=\mu_{i}\mu_{j}\left[\frac{r_{ij}^{\rm
s}}{\sqrt{\mu_{i}\mu_{j}}}-\left(1+\frac{r_{ij}^{\rm
s}}{\sqrt{\mu_{i}\mu_{j}}}\right)\ln\left({1+\frac{r_{ij}^{\rm
s}}{\sqrt{\mu_{i}\mu_{j}}}}\right)\right]$
$\displaystyle\approx-\frac{1}{2}{r_{ij}^{\rm s}}^{2},$ (50)
$\displaystyle\frac{t^{2}}{2}I^{tt,{\rm(2)}}_{ij}$
$\displaystyle=-\left\langle\rho_{ij}^{\rm n}({\bf s})\sqrt{C_{ii}({\bf
s})C_{jj}({\bf s})}\right\rangle_{s}\ln\left({1+\frac{r_{ij}^{\rm
s}}{\sqrt{\mu_{i}\mu_{j}}}}\right)$ $\displaystyle=-C_{ij}^{\rm
n}\ln\left({1+\frac{r_{ij}^{\rm s}}{\sqrt{\mu_{i}\mu_{j}}}}\right)$
$\displaystyle\approx-{r_{ij}^{\rm n}}{r_{ij}^{\rm s}},$ (51)
$\displaystyle\frac{t^{2}}{2}I^{tt,{\rm(3)}}_{ij}$
$\displaystyle=\bigg{\langle}\left(\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})+\rho_{ij}(\mathbf{s})\sqrt{C_{ii}({\bf
s})C_{jj}({\bf s})}\right)\times\bigg{[}\ln{\left(1+\frac{\rho_{ij}^{\rm
n}({\bf s})}{\sqrt{\mu_{i}({\bf s})\mu_{j}({\bf s})}}\right)}$
$\displaystyle~{}~{}~{}-\ln\left(1+\frac{\langle\rho_{ij}(\mathbf{s^{\prime}})\sqrt{C_{ii}({\bf
s^{\prime}})C_{jj}({\bf s^{\prime}})}\rangle_{\bf
s^{\prime}}}{\langle\mu_{i}({\bf s^{\prime}})\mu_{j}({\bf
s^{\prime}})\rangle_{\bf s^{\prime}}}\right)\bigg{]}\bigg{\rangle}_{\bf s}$
$\displaystyle=\bigg{\langle}\left(\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})+C_{ij}^{\rm
n}({\bf s})\right)\times\bigg{[}\ln{\left(1+\frac{\rho_{ij}^{\rm n}({\bf
s})}{\sqrt{\mu_{i}({\bf s})\mu_{j}({\bf
s})}}\right)}-\ln\left(1+\frac{C_{ij}^{\rm n}}{C_{ij}^{\rm
s}+\mu_{i}\mu_{j}}\right)\bigg{]}\bigg{\rangle}_{\bf s}$
$\displaystyle=\bigg{\langle}\left(\mu_{i}(\mathbf{s})\mu_{j}(\mathbf{s})+C_{ij}^{\rm
n}({\bf s})\right)\times\ln{\left(1+\frac{\rho_{ij}^{\rm n}({\bf
s})}{\sqrt{\mu_{i}({\bf s})\mu_{j}({\bf s})}}\right)}\bigg{\rangle}_{\bf s}$
$\displaystyle~{}~{}~{}-\mu_{i}\mu_{j}\left(1+\frac{r_{ij}^{\rm
s}}{\sqrt{\mu_{i}\mu_{j}}}+\frac{r_{ij}^{\rm
n}}{\sqrt{\mu_{i}\mu_{j}}}\right)\times\ln\left(1+\frac{r_{ij}^{\rm
n}/\sqrt{\mu_{i}\mu_{j}}}{r_{ij}^{\rm s}/\sqrt{\mu_{i}\mu_{j}}+1}\right)$
$\displaystyle\approx\frac{1}{2}\left<{\rho_{ij}^{\rm n}({\bf
s})}^{2}\right>_{\bf s}-\frac{1}{2}{r_{ij}^{n}}^{2}.$ (52)

Summing up these contributions gives:

$\displaystyle\frac{t^{2}}{2}I^{tt}$
$\displaystyle=\frac{t^{2}}{2}\sum_{ij}I^{tt,{\rm(1)}}_{ij}+I^{tt,{\rm(2)}}_{ij}+I^{tt,{\rm(3)}}_{ij}$
$\displaystyle=\frac{t^{2}}{2}\sum_{i}I^{tt,{\rm(1)}}_{ii}+t^{2}\sum_{i<j}I^{tt,{\rm(1)}}_{ij}+I^{tt,{\rm(2)}}_{ij}+I^{tt,{\rm(3)}}_{ij}$
$\displaystyle\approx\frac{1}{2}\sum_{i}{\mu_{i}}^{2}\left[\nu_{ii}+\left(1+\nu_{ii}\right)\ln\left(\frac{1}{1+\nu_{ii}}\right)\right]-\frac{1}{2}\sum_{i<j}\left((r_{ij}^{\rm
s}+{r_{ij}^{n}})^{2}-\left<{\rho_{ij}^{\rm n}({\bf s})}^{2}\right>_{\bf
s}\right)$
$\displaystyle\approx\frac{1}{2}\sum_{i}{\mu_{i}}^{2}\left[\nu_{ii}+\left(1+\nu_{ii}\right)\ln\left(\frac{1}{1+\nu_{ii}}\right)\right]-\frac{1}{2}\sum_{i<j}\left({\rho_{ij}^{\rm
tot}}^{2}-\left<{\rho_{ij}^{\rm n}({\bf s})}^{2}\right>_{\bf s}\right),$ (53)

and finally:

$\begin{split}I&\approx tI^{t}+\frac{t^{2}}{2}I^{tt}\\\ &\approx I_{\rm
single}+I_{\rm 2nd},\end{split}$ (54)

with the single site term given by:

$I_{\rm
single}=\sum_{i}\left\langle\mu_{i}(\mathbf{s})\ln\frac{\mu_{i}(\mathbf{s})}{\mu_{i}}\right\rangle_{s}+\frac{1}{2}\sum_{i}{\mu_{i}}^{2}\left[\nu_{ii}+\left(1+\nu_{ii}\right)\ln\left(\frac{1}{1+\nu_{ii}}\right)\right],$
(55)

and the second order cross-correlations contribution by:

$I_{\rm 2cd}=-\frac{1}{2}\sum_{i<j}\left({\rho_{ij}^{\rm
tot}}^{2}-\left<{\rho_{ij}^{\rm n}({\bf s})}^{2}\right>_{\bf s}\right).$ (56)

In the small correlations and small rates (or Poisson) limit, the contribution
to the mutual information of cross-correlations as described by the small time
bin expansion coincides with the second order approximation derived in this
paper. Besides, this small correlation expansion allows us to interpret the
different components of $I^{tt}$ directly in terms of pairwise correlations:
First, we see from Eq. 50 that $t^{2}\sum_{i<j}I^{tt,{\rm(1)}}_{ij}$ contains
the systematically detrimental effect of stimulus correlations in the
response. Second, Eq. 51 shows that $t^{2}\sum_{i<j}I^{tt,{\rm(2)}}_{ij}$
quantifies the effect of interactions between stimulus and noise correlations,
in accordance with the “sign-rule” Hu et al. (2014); Azeredo da Silveira and
Rieke (2021). Finally, Eq. 52 shows how $t^{2}\sum_{i<j}I^{tt,{\rm(3)}}_{ij}$
accounts for the beneficial effects of noise correlations beyond the “sign-
rule”. In particular, we can observe that pairwise noise correlations need not
fluctuate for $t^{2}\sum_{i<j}I^{tt,{\rm(3)}}_{ij}$ to be positive (see App. A
for a proof).