# Stimuli Dependent Synergy and Redundancy Dominated Causal Effects in Time
Series

Jan Østergaard (IEEE Senior Member) and Payam S. Boubakani J. Østergaard
<EMAIL_ADDRESS>and P. Boubakani<EMAIL_ADDRESS>are with Aalborg University,
Department of Electronic Systems, Aalborg, Denmark.This work was presented in
part at the IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP) 2022 [1]

###### Abstract

We characterize the degree of synergy- and redundancy-dominated causal
influence a time series has upon the interaction between other time series. We
prove that for a class of time series, the early past of the stimuli yields a
synergistic effect upon the interaction, whereas the late past has a
redundancy-dominated effect. Our information theoretic quantities are easy to
compute in practice, and we provide simulation studies on synthetic time
series and real-world signals. As an example, we use our measures on
intracranial EEG data to demonstrate that the stimuli from specific deep
electrodes cause a synergistic exchange of information to take place between
different brain regions during seizures.

##  I Introduction

Causal inference can be used to determine the effect that one system has upon
another system. For example, the study of effective connectivity in brain
regions can be considered as a problem of causal inference [2].

In general, a system that consists of sub systems can be represented as
interconnected nodes in a network or graph. Each node is then a sub system,
and the connections between the nodes represent the inputs and outputs of each
node. Pearl’s causality framework suggests that causality between nodes in a
network can only be truly assessed when one is allowed to manipulate the
inputs to the nodes in the network and being able to observe the changes in
the outputs of other nodes [3]. It is, however, also known that certain types
of causal connections in specialized system can be observed from empirical
observations without manipulating the system [4]. Nevertheless, regarding
causal inference from complex data, it is crucial to have access to the
external stimuli that induces the activity in the system.

A vast amount of research on general causal inference is devoted to effective
connectivity, where the main objective is to assess directional dependencies
between nodes based on empirical observations. Thus, from an information
theoretic point of view, a directional dependency from node A to node B
indicates that information is conveyed from node A into node B. However, given
only the empirical data, one can generally not conclude whether any effects
observed at B is caused by a change in A [5].

Estimating effective connectivity in real-world time-series data such as
financial data, weather data, and neuro-physiological data, etc., is an active
research area. According to Wiener’s notion of causal prediction [6], if the
prediction of the future value of a time series
$\mathcal{X}=X_{1},X_{2},\dots$ from its own past values can be improved by
incorporating past values of another time series
$\mathcal{Y}=Y_{1},Y_{2},\dots$, then there exist causal dependencies from
$\mathcal{Y}$ to $\mathcal{X}$, which does not necessarily imply that
$\mathcal{Y}$ is causing $\mathcal{X}$ [5].

Conventional methods such as cross-correlation, mutual information, or phase
synchrony [7] are able to find dependencies between time-series data but
cannot determine the direction of these dependencies. Granger’s causality
utilizes linear models to form causal predictions in the spirit of Wiener’s
notion and is thereby able to assess linear directed dependencies between
times-series data. More advanced directional measures such as Massey’s
directed information [8], Schreiber’s transfer entropy [9], and their many
conditional variants are commonly used for quantifying directed nonlinear
dependencies in neuro-physiological data [10, 11].

The emergent field of partial information decomposition provides the means to
decompose the conditional mutual information into relevant quantities such as
unique, common, and synergistic information [12, 13, 14]. Transfer entropy is
based on one particular decomposition, which does not appear to have a unique
operational interpretation. In particular, transfer entropy does not quantify
the amount of information being transmitted via a conventional probabilistic
channel between nodes in a network. Moreover, transfer entropy is by itself
not sufficient for assessing causal inference. This lack of explainability has
led to a lot of confusion in the literature about the interpretability of the
resulting functional and effective connectivity patterns obtained using
estimators of transfer entropy. For example, the amount of transfer entropy
measured between two brain regions is partly due to the information that is
actually being causally transferred between the regions at a given time,
partly due to common information received through other parts of the brain,
and partly due to synergistic information arising since conditioning can
increase the mutual information. In the neuroscience literature there has been
a great interest in developing information theoretic tools for quantifying
degrees of synergistic and redundant information exchange from brain data [15,
16, 14, 17, 18, 19].

In this paper, we are interested in characterizing the degree of synergy- and
redundancy-dominated causal influence a time-series $\mathcal{S}$ has upon the
directional couplings between other time-series $\mathcal{X}$ and
$\mathcal{Y}$. In particular, assume that some system $M_{1}$ contains sub
systems $\mathcal{X},\mathcal{Y}$, and $\mathcal{Z}$, which are all
interconnected. Moreover, assume that another system $M_{2}=\\{\mathcal{S}\\}$
is able to stimulate $M_{1}=\\{\mathcal{X},\mathcal{Y},\mathcal{Z}\\}$. Then,
we are interested in understanding whether the change in information flow
between say $\mathcal{X}$ and $\mathcal{Y}$, which is due to $\mathcal{S}$ but
not due to $\mathcal{Z}$ can be characterized as mainly being synergy- or
redundancy-dominated. A relevant application of such a measure could for
example relate to understanding the synergistic effects within the brain.
Assume that we are observing different brain regions of a subject, and that
some external signal (electrical, acoustic, visual, magnetic, etc.) is
stimulating the subjects brain. To what degree will such an external stimuli
cause a change of synergy- or redundancy-dominated information exchange
between the brain regions? Moreover, how do we exclude the effect of other
brain regions? We provide an example of this in Section IV.

Let $\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{S})$ denote the
conditional transfer entropy from $\mathcal{X}$ to $\mathcal{Y}$ causally
conditioned upon $\mathcal{S}$. Moreover, let $\mathcal{S}^{\mathrm{early}}$
denote the most recent past of $\mathcal{S}$ and let
$\mathcal{S}^{\mathrm{late}}$ denote the late past of $\mathcal{S}$. For
example, if $\mathcal{S}=S_{1},\dotsc,S_{i}$, then we might take
$\mathcal{S}^{\mathrm{early}}=\\{S_{i-1},S_{i}\\}$ and
$\mathcal{S}^{\mathrm{late}}=\\{S_{1},\dotsc,S_{i-2}\\}$. One of our
contributions is to prove that for a class of time-series, the transfer
entropy increases when conditioning upon the early past but decreases when
conditioning upon the late past, that is:

$\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{S}^{\mathrm{early}})>\mathrm{TE}(\mathcal{X}\to\mathcal{Y})>\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{S}^{\mathrm{late}}).$

Thus, a system mainly exploiting the early past of $\mathcal{S}$ would be
synergy-dominated, whereas a system mainly exploiting the late past would be
redundancy-dominated. For general systems, different parts of the past could
lead to different synergistic and redundancy-dominated effects.

Inspired by the above observation, we then form the following information
theoretic quantities, which assess the amount of synergistic and redundant
information between $\mathcal{X}$ and $\mathcal{Y}$, which is due to
$\mathcal{S}$ but not due to $\mathcal{Z}$:

$\displaystyle
I^{\mathrm{syn}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})$
$\displaystyle\quad\triangleq\max_{\hat{\mathcal{S}}\subseteq\mathcal{S}}\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z},\hat{\mathcal{S}})-\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z}).$
$\displaystyle
I^{\mathrm{red}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})$
$\displaystyle\quad\triangleq\mathrm{TE}(\mathcal{X};\mathcal{Y}\|\mathcal{Z})-\min_{\hat{\mathcal{S}}\subseteq\mathcal{S}}\mathrm{TE}(\mathcal{X};\mathcal{Y}\|\mathcal{Z},\hat{S}).$

The difference
$I_{\mathcal{S}}^{\mathrm{diff}}=I^{\mathrm{red}}_{\mathcal{S}}-I^{\mathrm{syn}}_{\mathcal{S}}$
quantifies the overall effect of $\mathcal{S}$:

$\displaystyle
I_{\mathcal{S}}^{\mathrm{diff}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\triangleq
I^{\mathrm{red}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})-I^{\mathrm{syn}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z}).$

If $I_{\mathcal{S}}^{\mathrm{diff}}<0$, the overall effect due to
$\mathcal{S}$ is mainly synergy-dominated, whereas if
$I_{\mathcal{S}}^{\mathrm{diff}}>0$ it is redundancy-dominated.

Our information quantities
$I^{\mathrm{syn}}_{\mathcal{S}},I^{\mathrm{red}}_{\mathcal{S}}$, and
$I_{\mathcal{S}}^{\mathrm{diff}}$ are easy to compute on real-world time-
series data using e.g., publicly available transfer entropy estimators as we
demonstrate with examples in Section IV.

##  II Notation

Let $\mathbf{X}_{1},\mathbf{X}_{2},\dotsc$, where
$\mathbf{X}_{j}\in\mathbb{R}^{N},\forall j,$ be an $N$-dimensional random
vector process, which can be written as a general non-linear dynamical
systems:

$\mathbf{X}_{i}=f(\bar{X}^{i-1})+\mathbf{W}_{i},$ (1)

where $f$ is a non-linear stable map, and
$\mathbf{W}_{1},\mathbf{W}_{2},\dots$, is a stationary (spatio-temporal)
random vector process. To exemplify we will restrict to the case where $N=4$,
and use $X_{i},Y_{i},Z_{i},$ and $S_{i}$ to denote the individual dimensions
of $\mathbf{X}_{i}$, i.e., $\mathbf{X}_{i}=[X_{i},Y_{i},Z_{i},S_{i}]^{T}$.
Moreover, we will consider the following non-linear dynamical system as a show
case:

$\displaystyle X_{i}$
$\displaystyle=\alpha^{xx}_{1}f_{xx}(X_{i-1})+\alpha^{zx}_{2}f_{zx}(Z_{i-2})+\alpha^{sx}_{1}f_{sx}(S_{i-1})+W^{x}_{i}$
(2) $\displaystyle Y_{i}$
$\displaystyle=\alpha^{yy}_{1}f_{yy}(Y_{i-1})+\alpha^{xy}_{2}f_{xy}(X_{i-2})+\alpha^{zy}_{2}f_{zy}(Z_{i-2})$
$\displaystyle\quad+\alpha^{sy}_{1}f_{sy}(S_{i-1})+W^{y}_{i}$ (3)
$\displaystyle Z_{i}$ $\displaystyle=\alpha^{zz}_{1}f_{zz}(Z_{i-1})+W^{z}_{i}$
(4) $\displaystyle S_{i}$
$\displaystyle=\alpha^{ss}_{1}f_{ss}(S_{i-1})+\alpha^{ss}_{2}f_{ss}(S_{i-2})+W^{s}_{i},$
(5)

where $f_{\cdot,\cdot}$ are arbitrary (non-linear) differentiable functions
with convergent Taylor series, and where
$W^{\phi}_{i},W^{\phi^{\prime}}_{\ell},\phi\neq\phi^{\prime}\in\\{x,y,z,s\\}$
are mutually independent for all $i,\ell$, and otherwise arbitrarily
distributed.

Let $i$ denote the current time instance, and let
$X^{i-1}=X_{1},\dotsc,X_{i-1}$ denote the sequence of $(i-1)$ past samples of
the process. Similarly notation applies to $Y,Z,$ and $S$.

Let $A,B$ be discrete random variables having elements in the alphabets
$\mathcal{A}$ and $\mathcal{B}$, respectively. We use the notation
$\mathrm{P}_{A|B}(a|b)$ to denote the probability that $A$ takes on the
outcome $a\in\mathcal{A}$ conditioned upon that $B=b\in\mathcal{B}$.

##  III New Synergy- and Redundancy-Dominated Information Measures

###  III-A Conditional Mutual Information

The mutual information $I(A;B)$ measures the degree of dependency between two
random variables $A$ and $B$. If $A$ and $B$ are independent, then $I(A;B)=0$.
On the other hand, if $A$ and $B$ are dependent then $I(A;B)>0$. If we
condition the mutual information upon a third random variable, say $C$, then
the resulting conditional mutual information reflects both a potential
decrease of common (or redundant) information as well as a potential increase
of synergistic information. Specifically, by conditioning upon $C$, the mutual
information between $A$ and $B$ is reduced by the amount of common information
that $C$ contains about both $A$ and $B$. On the other hand, there could
potentially be a synergistic effect, where knowledge of $C$ would reveal new
information in $B$ about $A$. This is demonstrated by Theorem 1 below and
illustrated in Fig. 1.

######  Theorem 1.

Let $A,C,C^{\prime}$ be mutually independent binary random variables, where
$C,C^{\prime}$ are uniformly distributed on $\\{0,1\\}$, and $A$ is
arbitrarily distributed with $p=\mathrm{P}_{A}(0)=1-\mathrm{P}_{A}(1)$. Let
$B=A\wedge C$ and $D=A\wedge C^{\prime}$, where $\wedge$ denotes the Boolean
logical AND operator. Then for any $0\leq p\leq 1$:

$H(A)\geq I(A;B|C)\overset{(a)}{\geq}I(A;B)\overset{(b)}{\geq}I(A;B|D),$ (6)

with equality in $(a)$ and $(b)$ if and only if $p=0$ or $p=1$.

Figure 1: The conditional mutual information exhibits mainly synergistic
$I(A;B|C)>I(A;B)$ and redundancy-dominated $I(A;B|D)<I(A;B)$ effects, when
conditioning upon $C$ and $D$, respectively. See Theorem 1 for details.

###  III-B Transfer Entropy

The causally conditioned transfer entropy is defined for stationary processes
via the conditional mutual information in the following way [20]:

$\displaystyle\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\triangleq
I(X^{i-1};Y_{i}|Y^{i-1},Z^{i-1}).$ (7)

Thus, at any time instance $i$, the conditional transfer entropy quantifies
the conditional mutual information between the whole past of $X^{i-1}$ and the
current $Y_{i}$ given knowledge of the past of $Y^{i-1}$ and $Z^{i-1}$. It is
clear that transfer entropy is a special case of conditional mutual
information. Moreover, transfer entropy is closely related to Massey’s notions
of directed information $I(\mathcal{X}\to\mathcal{Y})$ and causally
conditioned directed information $I(\mathcal{X}\to\mathcal{Y}\|\mathcal{Y})$,
which for $N$-length sequences are given by [8, 21]:

$\displaystyle I(\mathcal{X}\to\mathcal{Y})$
$\displaystyle\triangleq\sum_{i=1}^{N}I(X^{i};Y_{i}|Y^{i-1}).$ (8)
$\displaystyle I(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})$
$\displaystyle\triangleq\sum_{i=1}^{N}I(X^{i};Y_{i}|Y^{i-1},Z^{i-1}).$ (9)

###  III-C Synergistic Versus Common Information in Dynamical Systems

We turn our attention to the non-linear continuous-alphabet dynamical system
given in (2) – (5). Theorem 2 below demonstrates that conditioning upon
different parts of the past of a process can have different effects upon the
conditional mutual information between other processes. Specifically, for the
part of the dynamical system that is considered in the theorem, the recent
past has mainly a synergistic effect, whereas the late past mainly accounts
for a reduction of common information.

######  Theorem 2.

Consider the non-linear dynamical system in (2) – (5). Let
$\alpha=\alpha_{1}^{s}=\alpha_{2}^{s}=\alpha_{1}^{sy}=\alpha_{1}^{sx}$ and
$\sigma^{2}_{W^{s}_{i-j}}=c\alpha^{-2},\forall j$, so that
$\alpha^{2}\sigma^{2}_{W^{s}_{i-j}}=c$. Then, for any $c>0$:

$\displaystyle\lim_{\alpha\to
0}\mathrm{TE}(\mathcal{Z}\to\mathcal{X}\|\mathcal{Y},S_{i-1})>\lim_{\alpha\to
0}\mathrm{TE}(\mathcal{Z}\to\mathcal{X}\|\mathcal{Y})$
$\displaystyle\quad>\lim_{\alpha\to
0}\mathrm{TE}(\mathcal{Z}\to\mathcal{X}\|\mathcal{Y},S^{i-2}).$ (10)

###### Proof.

The second inequality is proved by Lemma 1 in the appendix. Thus, we only need
to prove that the first inequality holds. To do this we expand the first
conditional transfer entropy in (10) as follows:

$\displaystyle I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1},S_{i-1})$
$\displaystyle=I(Z^{i-1};\alpha^{xx}_{1}f_{xx}(X_{i-1})+\alpha^{zx}_{2}f_{zx}(Z_{i-2})+\alpha^{sx}_{1}f_{sx}(S_{i-1})$
$\displaystyle\quad+W^{x}_{i}|X^{i-1},Y^{i-1},S_{i-1})$ (11)
$\displaystyle=I(Z^{i-1};\alpha^{zx}_{2}f_{zz}(Z_{i-2})+W^{x}_{i}|X^{i-1},Y^{i-1},S_{i-1})$
$\displaystyle=I(Z_{i-2};\alpha^{zx}_{2}f_{zz}(Z_{i-2})+W^{x}_{i}|X^{i-1},Y^{i-1},S_{i-1})$
$\displaystyle\quad+I(Z_{i-1},Z^{i-3};W^{x}_{i}|X^{i-1},Y^{i-1},S_{i-1},Z_{i-2})$
(12)
$\displaystyle=I(Z_{i-2};\alpha^{zx}_{2}f_{zz}(Z_{i-2})+W^{x}_{i}|X^{i-1},Y^{i-1},S_{i-1}).$
(13)

When we are not conditioning upon $S_{i-1}$, we obtain from (36) in the
appendix:

$\displaystyle
I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1})=I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha\beta
W_{i-1}^{s}$ $\displaystyle\quad+W^{x}_{i}+o(\alpha)|X^{i-1},Y^{i-1}),$ (14)

where $o(\alpha)/\alpha\to 0$ as $\alpha\to 0$. Since
$X^{i-1},Y^{i-1},S_{i-1}$ are independent of $W^{x}_{i}$ and
$\alpha^{2}\sigma_{W^{s}}^{2}=c>0$ it is clear that
$\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha\beta W_{i-1}^{s}+W^{x}_{i}$ is a more
noisy version of $Z_{i-2}$ than $\alpha^{zx}_{2}f_{z}(Z_{i-2})+W^{x}_{i}$.
Thus, (13) is strictly smaller than (14). This proves the theorem. ∎

######  Remark 1.

Theorem 2 proves that for this class of time series, conditioning upon the
early past of $\mathcal{S}$ results in a synergy-dominated effect on the
conditional transfer entropy between $\mathcal{Z}$ and $\mathcal{X}$ given
$\mathcal{Y}$. On the other hand, conditioning upon the late past results in a
redundancy-dominated effect on the conditional transfer entropy between
$\mathcal{Z}$ and $\mathcal{X}$ given $\mathcal{Y}$.

###  III-D Synergy- and Redundancy-Dominated Information Measures

We are now in a position to define a notion of coupling strength for the
synergistic information occuring between time series due the external stimuli.
Since conditioning upon different parts of the past lead to different effects
upon the conditional transfer entropy, we introduce a maximization:

$\displaystyle I^{\mathrm{syn}}_{S}$
$\displaystyle(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\triangleq\max_{f(\hat{S}|S)}I(X^{i-1};Y_{i}|Y^{i-1},Z^{i-1},\hat{S})$
$\displaystyle\quad-I(X^{i-1};Y_{i}|Y^{i-1},Z^{i-1}).$ (15)

The maximization is over all distributions on $\mathcal{S}$ as suggested in
[13]. The reasons for forming a difference of information quantities are
twofold. First, the excess synergistic information due to conditioning only
upon $\mathcal{Z}$ is in (15) removed by forming a difference of causally
conditioned mutual informations. Thus, we are able to remove the effect that
$\mathcal{Z}$ has by itself upon the mutual information between $\mathcal{X}$
and $\mathcal{Y}$. Yet, any potential synergistic information due to the
combined knowledge of $\mathcal{S}$ and $\mathcal{Z}$ are not excluded.
Second, it allows us to separate the part of the external stimuli that mainly
has a synergistic effect. Whilst $\mathcal{S}$ does not explicitly appear in
the second term in (15), it should be noted that $\mathcal{X},\mathcal{Y},$
and $\mathcal{Z}$ could potentially be causal functions of $\mathcal{S}$. It
follows that (15) quantifies the part of the information that is causally and
synergistically occuring between $\mathcal{X}$ and $\mathcal{Y}$ due to
$\mathcal{S}$ but not due to $\mathcal{Z}$ by itself.

The maximization in (15) is generally computationally intractable. Instead we
introduce a lower bound to (15), which is simple to compute on time-series
data as we demonstrate in Section IV. In particular, instead of extremizing
over all measurable functions, we simply extremize over subsets of the
elements of $\mathcal{S}$ as we earlier suggested in [1]. Specifically, we
introduce the following information theoretic quantity:

$\displaystyle\hat{I}^{\mathrm{syn}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\triangleq\max_{\hat{\mathcal{S}}\subseteq\mathcal{S}}\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z},\hat{\mathcal{S}})-\mathrm{TE}(\mathcal{X};\mathcal{Y}\|\mathcal{Z})$
$\displaystyle=\max_{\hat{\mathcal{S}}\subseteq
S^{i-1}}I(X^{i-1};Y_{i}|Y^{i-1},Z^{i-1},\hat{S})$
$\displaystyle\qquad-I(X^{i-1};Y_{i}|Y^{i-1},Z^{i-1},\hat{S}).$ (16)

It is easy to see that
$\hat{I}^{\mathrm{syn}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\leq
I^{\mathrm{syn}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})$, since
we restrict the maximization to a subset of the feasible set. Thus, we
potentially underestimate the true amount of synergistic information.

To characterize the redundancy-dominated effect, we introduce a minimization
over $\mathcal{S}$ and form the following quantity and a corresponding lower
bound:

$\displaystyle
I^{\mathrm{red}}_{S}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\triangleq\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\\!-\\!\\!\min_{f(\hat{S}|S)}\\!\\!\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z},\hat{\mathcal{S}})$
$\displaystyle\geq\hat{I}^{\mathrm{red}}_{S}(X\to Y\|Z)$
$\displaystyle\triangleq\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})-\min_{\hat{\mathcal{S}}\subseteq\mathcal{S}}\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z},\hat{\mathcal{S}})$
$\displaystyle=I(X^{i-1};Y_{i}|Y^{i-1},Z^{i-1})$
$\displaystyle\qquad-\min_{\hat{\mathcal{S}}\subseteq
S^{i-1}}I(X^{i-1};Y_{i}|Y^{i-1},Z^{i-1},\hat{S}).$ (17)

Note that $\hat{I}^{\mathrm{red}}_{S}$ is a lower bound to
$I^{\mathrm{red}}_{S}$ and therefore potentially underestimates the true
amount of redundancy-dominated information.

We also introduce the difference
$\hat{I}^{\mathrm{red}}_{\mathcal{S}}-\hat{I}^{\mathrm{syn}}_{\mathcal{S}}$ to
assess whether the overall effect of $\mathcal{S}$ is mainly synergy or
redundancy-dominated:

$\displaystyle\hat{I}_{\mathcal{S}}^{\mathrm{diff}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})\triangleq\hat{I}^{\mathrm{red}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})-\hat{I}^{\mathrm{syn}}_{\mathcal{S}}(\mathcal{X}\to\mathcal{Y}|\mathcal{Z})$
$\displaystyle=2\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z})$
$\displaystyle-(\max_{\hat{\mathcal{S}}\subseteq\mathcal{S}}\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z},\hat{\mathcal{S}})+\min_{\hat{\mathcal{S}}\subseteq\mathcal{S}}\mathrm{TE}(\mathcal{X}\to\mathcal{Y}\|\mathcal{Z},\hat{\mathcal{S}})).$
(18)

If $I_{\mathcal{S}}^{\mathrm{diff}}>0$, then the effect of $\mathcal{S}$ is
mainly redundancy-dominated, whereas if $I_{\mathcal{S}}^{\mathrm{diff}}<0$
then it is mainly synergy-dominated.

##  IV Examples

In this section, we demonstrate our results on synthetic and real-world
signals.

###  IV-A Synthetic data

Consider the following non-linear instance of the dynamical system in (2) –
(5):

$\displaystyle X_{i}$ $\displaystyle=0.35X_{i-1}+0.5Z_{i-2}+\alpha
S_{i-1}+W^{x}_{i}$ (19) $\displaystyle Y_{i}$
$\displaystyle=0.35Y_{i-1}+0.5X_{i-2}^{2}+0.5Z_{i-2}+\alpha S_{i-1}+W^{y}_{i}$
(20) $\displaystyle Z_{i}$ $\displaystyle=0.35Z_{i-1}+W^{z}_{i}$ (21)
$\displaystyle S_{i}$ $\displaystyle=0.35S_{i-1}-0.2S_{i-2}+W^{s}_{i},$ (22)

where $\\{W^{\phi}_{i}\\}$ is a zero-mean and unit-variance i.i.d. Gaussian
processes, and where $\\{W^{\phi}_{i}\\},\\{W^{\phi^{\prime}}_{i}\\}$ are
mutually independent for $\phi\neq\phi^{\prime}\in\\{x,y,z,s\\}$.

Figure 2: The Conditional $\mathrm{TE}$ from $\mathcal{Z}\to\mathcal{X}$ due
to the early past $S_{i-1}$ (solid) and the late past $(S_{i-4},S_{i-5})$
(dashed-dotted). Also shown is the case without conditioning upon
$\mathcal{S}$ (dashed). Figure 3: The synergy-dominated effect
$\hat{I}_{S}^{\mathrm{syn}}$ from $\mathcal{Z}\to\mathcal{X}$ (solid) and the
redundancy-dominated effect $\hat{I}_{S}^{\mathrm{syn}}$ (dashed). The overall
dominance $\hat{I}_{S}^{\mathrm{diff}}$ is also shown (dashed-dotted).

For $\alpha\in\\{0.0,0.1,\dotsc,1\\}$ we generate 50 realizations each having
10000 samples of the processes
$\mathcal{X},\mathcal{Y},\mathcal{Z},\mathcal{S}$. We use the freely available
$k$-nearest neighbor based conditional mutual information estimator from the
TRENTTOOL Matlab toolbox [22].

From the simulated data we estimate the conditional mutual information and
average the results over the 50 realizations. For the case of
$\mathrm{TE}(\mathcal{Z}\to\mathcal{X}\|\mathcal{Y},\mathcal{S})$ as in
Theorem 2, the resulting averages are shown in Fig. 2. It can be seen that for
quite a range of $\alpha$ values,
$\mathrm{TE}(\mathcal{Z}\to\mathcal{X}\|\mathcal{Z},S_{i-1})>\mathrm{TE}(\mathcal{Z}\to\mathcal{X}\|\mathcal{Z})>\mathrm{TE}(\mathcal{Z}\to\mathcal{X}\|\mathcal{Z},S_{i-3},S_{i-4},S_{i-5})$.
Whilst Theorem 2 considers the asymptotical case $\alpha\to 0$, this example
demonstrates that in the non-asymptotical case where $\alpha>0$, we also
observe the behavior that the early past of $\mathcal{S}$ leads to synergy-
dominated couplings, whereas the late past is mainly redundancy-dominated.

The synergy-dominated effect $\hat{I}_{S}^{\mathrm{syn}}$ from
$\mathcal{Z}\to\mathcal{X}$ is shown in Fig. 3 (solid) and the redundancy-
dominated effect $\hat{I}_{S}^{\mathrm{red}}$ is also shown (dashed). For
$\alpha>0.2$, the system changes from being mainly redundancy-dominated to
becoming synergy-dominated as is illustrated by $\hat{I}_{S}^{\mathrm{diff}}$
(dashed-dotted).

###  IV-B Real-word data

Locating areas in the brain that are causing seizures is an important issue in
the study of epilepsy in humans. In this section, we consider intracranial EEG
recordings from a patient with drug-resistant epilepsy as a real example of
our proposed measures. The data were recorded by an implanted array of
$8\times 8$ cortical electrodes referred to as electrodes
$E_{1},\dotsc,E_{64}$, and two left hippocampal depth electrodes (each having
six electrodes), referred to as depths electrodes $E_{65},\dotsc,E_{76}$. The
data are available from [23] and described in details in [24].

In [25], it was suggested that synergistic interaction between depths and
cortical electrodes were greatest from depth electrodes $E_{75}$ and $E_{76}$;
the last two electrodes in the second depth electrode. A subsequent study
demonstrated that the _unique_ information exchange from $E_{76}$ and to all
the cortical electrodes were greater than the _unique_ information exchange
from $E_{75}$ during periods of seizure [26].

Inspired by [25, 26], we hypothesize that stimulation by the depth electrodes
causes a population of neurons in one region of the brain to synchronize and
propagate their activity to other parts of the brain, which then results in a
seizure. Based on this we formulate the problem of identifying which of the
depth electrodes $E_{75}$ and $E_{76}$ causes the greatest synergistic
information exchange _between_ the cortical electrodes $E_{1},\dotsc,E_{64},$
as quantified by $\hat{I}^{\mathrm{syn}}$ in (16).

A total of eight epileptic seizures (Ictal) and eight pre-ictal (Pre) periods
were recorded. Intracranial EEG data were recorded at 400 Hz sampling
frequency. A 50 Hz anti-aliasing low-pass filter was applied before
downsampling the signals to 100 Hz. We used the TRENTTOOL conditional mutual
information estimator with an embedding delay of 1, lag dimension of 5, and
$k=10$ nearest neighbors [22].

We used (16) to estimate the synergistic information
$\hat{I}_{E_{76}}^{\mathrm{syn}}(i)\triangleq\hat{I}_{E_{76}}^{\mathrm{syn}}(\mathcal{X}\to\mathcal{Y}\|E_{75})$,
which is due to $E_{76}$ and not due to $E_{75}$. Here
$\mathcal{X}=E_{i},i=1,\dotsc,64$, and
$\mathcal{Y}=\\{E_{1},E_{2},\cdots,E_{64}\\}\backslash\\{\mathcal{X}\\}$.
Thus, we measured the synergistic information from the $i$th electrode to the
remaining 63 cortical electrodes. We similarly estimated
$\hat{I}_{E_{75}}^{\mathrm{syn}}(i)\triangleq\hat{I}_{E_{75}}^{\mathrm{syn}}(\mathcal{X}\to\mathcal{Y}\|E_{76})$.
Their difference
$\hat{I}_{E_{76}}^{\mathrm{syn}}(i)-\hat{I}_{E_{75}}^{\mathrm{syn}}(i)$ for
$i=1,\dotsc,64,$ is plotted in Fig. 4 for the Ictal period (circles) and the
pre-Ictal period (stars). During seizures (i.e., the Ictal period), it can be
observed that the effect of $E_{76}$ dominates that of $E_{75}$ in the sense
that the difference
$\hat{I}_{E_{76}}^{\mathrm{syn}}(i)-\hat{I}_{E_{75}}^{\mathrm{syn}}(i)$ is
positive for most electrodes, and the average difference is 0.011, which is
positive. This suggests that there emerges a greater synergistic information
exchange between cortical brain regions due to stimulation by $E_{76}$ than
due to stimulation by $E_{75}$ during seizures.

Figure 4: The difference
$\hat{I}_{E_{76}}^{\mathrm{syn}}(i)-\hat{I}_{E_{75}}^{\mathrm{syn}}(i)$ for
$i=1,\dotsc,64$. Positive numbers indicate that the effect of electrode
$E_{76}$ dominates over that of $E_{75}$, whereas the opposite is true for
negative numbers.

##  V Conclusions

We were interested in assessing the effect that a system has upon the
interaction between other systems. In particular, we demonstrated that for a
class of simulated dynamical systems, the early past of a system cause a
synergy-dominated interaction between other systems, whereas the late past
cause a redundancy-dominated interaction. We also applied our measures on a
real-world intracranial EEG data. Previous works have demonstrated that there
occur an exchange of _unique_ information from deep electrodes to certain
brain regions during seizures [26]. The results obtain with our new measures
are able to complement the existing findings by demonstrating that the stimuli
from particular deep electrodes cause a synergistic exchange of information to
take place between different brain regions. This could suggest that
integration of information is taken place in particular brain regions during
seizures.

###### Proof of Theorem 1.

We first consider the left hand side of (6) and rewrite
$I(A;B|C)=H(A|C)-H(A|B,C)=H(A)-H(A|B,C)$, where the latter conditional entropy
can be further re-written as:

$\displaystyle H(A|B,C)=\sum_{b,c}\mathrm{P}_{B,C}(b,c)H(A|B=b,C=c)$
$\displaystyle=-\mathrm{P}_{B,C}(0,0)\big{(}\mathrm{P}_{A|B,C}(0|0,0)\log_{2}(\mathrm{P}_{A|B,C}(0|0,0))$
$\displaystyle\quad+\mathrm{P}_{A|B,C}(1|0,0)\log_{2}(\mathrm{P}_{A|B,C}(1|0,0))\big{)}$
$\displaystyle\quad-\mathrm{P}_{B,C}(0,1)\big{(}\mathrm{P}_{A|B,C}(0|0,1)\log_{2}(\mathrm{P}_{A|B,C}(0|0,1))$
$\displaystyle\quad+\mathrm{P}_{A|B,C}(1|0,1)\log_{2}(\mathrm{P}_{A|B,C}(1|0,1))\big{)}$
$\displaystyle\quad-\mathrm{P}_{B,C}(1,0)\big{(}\mathrm{P}_{A|B,C}(0|1,0)\log_{2}(\mathrm{P}_{A|B,C}(0|1,0))$
$\displaystyle\quad+\mathrm{P}_{A|B,C}(1|1,0)\log_{2}(\mathrm{P}_{A|B,C}(1|1,0))\big{)}$
$\displaystyle\quad-\mathrm{P}_{B,C}(1,1)\big{(}\mathrm{P}_{A|B,C}(0|1,1)\log_{2}(\mathrm{P}_{A|B,C}(0|1,1))$
$\displaystyle\quad+\mathrm{P}_{A|B,C}(1|1,1)\log_{2}(\mathrm{P}_{A|B,C}(1|1,1))\big{)}$
$\displaystyle=-\mathrm{P}_{B,C}(0,0)\big{(}\mathrm{P}_{A|B,C}(0|0,0)\log_{2}(\mathrm{P}_{A|B,C}(0|0,0))$
$\displaystyle\quad+\mathrm{P}_{A|B,C}(1|0,0)\log_{2}(\mathrm{P}_{A|B,C}(1|0,0))\big{)}$
$\displaystyle\quad-\mathrm{P}_{B,C}(0,1)\big{(}\mathrm{P}_{A|B,C}(0|0,1)\log_{2}(\mathrm{P}_{A|B,C}(0|0,1))$
$\displaystyle\quad+\mathrm{P}_{A|B,C}(1|0,1)\log_{2}(\mathrm{P}_{A|B,C}(1|0,1))\big{)}$
$\displaystyle=-\frac{1}{2}\big{(}p\log_{2}(p)+(1-p)\log_{2}(1-p)\big{)}$
$\displaystyle=\mathrm{P}_{B,C}(0,0)H(A)$ $\displaystyle=0.5H(A),$

which follows since
$\mathrm{P}_{B,C}(0,0)=p,\mathrm{P}_{B,C}(0,1)=\frac{1}{2}(1-p),\mathrm{P}_{B,C}(1,0)=0,\mathrm{P}_{A|B,C}(0|1,0)=0,\mathrm{P}_{A|B,C}(1|1,1)=1,\mathrm{P}_{A|B,C}(0|0,1)=p$,
and $\mathrm{P}_{A|B,C}(1|0,1)=1-p.$ It follows that:

$\displaystyle I(A;B|C)=0.5H(A).$ (23)

We now consider the middle term in (6), i.e., $I(A;B)=H(A)-H(A|B)$, where the
latter term can be written as:

$\displaystyle
H(A|B)\overset{(a)}{=}\sum_{b\in\\{0,1\\}}\mathrm{P}_{B}(b)H(A|B=b)$
$\displaystyle=-\sum_{b\in\\{0,1\\}}\sum_{a\in\\{0,1\\}}\mathrm{P}_{B}(b)\mathrm{P}_{A|B}(a|b)\log_{2}(\mathrm{P}_{A|B}(a|b))$
$\displaystyle=-\mathrm{P}_{B}(0)\mathrm{P}_{A|B}(0|0)\log_{2}(\mathrm{P}_{A|B}(0|0))$
$\displaystyle\quad-\mathrm{P}_{B}(0)\mathrm{P}_{A|B}(1|0)\log_{2}(\mathrm{P}_{A|B}(1|0))$
$\displaystyle\quad-\mathrm{P}_{B}(1)\mathrm{P}_{A|B}(0|1)\log_{2}(\mathrm{P}_{A|B}(0|1))$
$\displaystyle=-\mathrm{P}_{A}(0)\log_{2}\frac{\mathrm{P}_{A}(0)}{\mathrm{P}_{B}(0)}+\frac{1}{2}\mathrm{P}_{A}(1)\log_{2}\frac{1}{2}\frac{\mathrm{P}_{A}(1)}{\mathrm{P}_{B}(0)}$
$\displaystyle\quad-\frac{1}{2}\mathrm{P}_{A}(1)\log_{2}\frac{1}{2}\frac{\mathrm{P}_{A}(1)}{\mathrm{P}_{B}(1)}$
$\displaystyle=H(A)+\mathrm{P}_{A}(0)\log_{2}(\mathrm{P}_{B}(0))$
$\displaystyle\quad+\frac{1}{2}\mathrm{P}_{A}(1)(\log_{2}(\mathrm{P}_{B}(0))+\log_{2}(\mathrm{P}_{B}(1))+\mathrm{P}_{A}(1)$
$\displaystyle=\frac{1}{2}H(A)+\varphi$

where

$\displaystyle\varphi$
$\displaystyle=\frac{1}{2}H(A)+\mathrm{P}_{A}(0)\log_{2}(\mathrm{P}_{B}(0))$
$\displaystyle\quad+\frac{1}{2}\mathrm{P}_{A}(1)(\log_{2}(\mathrm{P}_{B}(0))+\log_{2}(\mathrm{P}_{B}(1))+\mathrm{P}_{A}(1).$

The second derivative of $\varphi$ with respect to $p=\mathrm{P}_{B}(0)$ is
given by:

$\displaystyle\frac{\partial^{2}}{\partial
p^{2}}\varphi=-\frac{1}{2(1-p)p\ln(2)}.$ (24)

It follows that $\varphi$ is a strictly concave function in $p$ and to find
its minima it is sufficient to check the two boundary points $p=0$ and $p=1$,
which when inserted into (24) yields $\varphi_{|p=0}=\varphi_{|p=1}=0$. Thus,
$\varphi$ is a non-negative function. We can now show that $I(A;B)<I(A;B|C)$:

$\displaystyle I(A;B)$ $\displaystyle=H(A)-H(A|B)$ (25)
$\displaystyle=H(A)-\frac{1}{2}H(A)-\phi$ (26)
$\displaystyle\overset{(a)}{\leq}\frac{1}{2}H(A)$ (27)
$\displaystyle=I(A;B|C),$ (28)

with equality in $(a)$ only for $p=0$ or $p=1$, which follows since the
gradient of $\varphi$ is non-zero at these extremes.

Let us now consider the last term in (6):

$\displaystyle I(A;B|D)=H(A|D)-H(A|B,D)$
$\displaystyle=\sum_{d}\mathrm{P}_{D}(d)H(A|D=d)$
$\displaystyle\quad-\sum_{b,d}\mathrm{P}_{B,D}(b,d)H(A|B=b,D=d).$

Since we are considering binary alphabets, we can express the probabilities
explicitly and determine, which ones that are trivially 0 or 1 – in which case
they can be excluded.

$\displaystyle\mathrm{P}_{A|D}(0|0)=\mathrm{P}_{D|A}(0|0)p/\mathrm{P}_{D}(0)=p/\mathrm{P}_{D}(0).$
$\displaystyle\mathrm{P}_{A|D}(1|0)=\frac{1}{2}(1-p)/\mathrm{P}_{D}(0).$
$\displaystyle P_{B,D}(0,0)=\frac{1}{4}(1-p)+p.$ $\displaystyle
P_{B,D}(0,1)=\frac{1}{4}(1-p).$
$\displaystyle\mathrm{P}_{A|B,D}(0|0,0)=\frac{p}{\mathrm{P}_{B,D}(0,0)}.$
$\displaystyle\mathrm{P}_{A|B,D}(0|1,0)=0.$
$\displaystyle\mathrm{P}_{A|B,D}(1|0,0)=\frac{1}{4}\frac{(1-p)}{\mathrm{P}_{B,D}(0,0)}.$
$\displaystyle\mathrm{P}_{A|B,D}(1|1,0)=1.$

We can now expand $H(A|B,D)$ as a function of $p$:

$\displaystyle H(A|B,D)=-\mathrm{P}_{B,D}(0,0)$
$\displaystyle\quad\times\bigg{(}\frac{p}{\mathrm{P}_{B,D}(0,0)}\log_{2}\frac{p}{\mathrm{P}_{B,D}(0,0)}$
$\displaystyle\quad+\frac{1}{4}\frac{(1-p)}{\mathrm{P}_{B,D}(0,0)}\log_{2}\frac{1}{4}\frac{1-p}{\mathrm{P}_{B,D}(0,0)}\bigg{)}$
$\displaystyle=-\bigg{(}p\log_{2}\frac{p}{\mathrm{P}_{B,D}(0,0)}+\frac{1}{4}(1-p)\log_{2}\frac{1}{4}\frac{(1-p)}{\mathrm{P}_{B,D}(0,0)}\bigg{)}$
$\displaystyle=-p\log_{2}\frac{p}{p+\frac{1}{4}(1-p)}-\frac{1}{4}(1-p)\log_{2}\frac{1}{4}\frac{(1-p)}{p+\frac{1}{4}(1-p)}$
$\displaystyle=\frac{1}{2}(1-p)+(p+\frac{1}{4}(1-p))\log_{2}(p+\frac{1}{4}(1-p))$
$\displaystyle\quad-\frac{3}{4}p\log_{2}(p)+\frac{1}{4}H(A).$

Similarly, we can express $H(A|B)$ as a function of $p$:

$\displaystyle
H(A|D)=-\mathrm{P}_{D}(0)\big{(}\mathrm{P}_{A|D}(0|0)\log_{2}(\mathrm{P}_{A|D}(0|0))$
(29) $\displaystyle\quad+\mathrm{P}_{A|D}(1|0)\log_{2}(P_{A|D}(1|0)\big{)}$
(30)
$\displaystyle=-p\log_{2}\frac{p}{\mathrm{P}_{D}(0)}-\frac{1}{2}(1-p)\log_{2}\frac{\frac{1}{2}(1-p)}{\mathrm{P}_{D}(0)},$
(31)

where $\mathrm{P}_{D}(0)=1/2+p/2$. We have now expressed $I(A;B)$ and
$I(A;B|D)$ as simple functions of $p$. This allows us to easily determine
convexity by use of the second derivative of $I(A;B)-I(A;B|D)$ with respect to
$p$, that is:

$\displaystyle\frac{\partial^{2}}{\partial
p^{2}}(I(A;B)-I(A;B|D))=\frac{1}{(3p^{3}-2p^{2}-p)\ln(2)}.$ (32)

It is clear that $\frac{\partial^{2}}{\partial p^{2}}(3p^{3}-2p^{2}-p)=18p$,
which implies that $3p^{3}-2p^{2}-p$ is strictly convex. Thus, the maxima are
at the boundaries, and these are furthermore the minima of $I(A;B)-I(A;B|D)$.
Inserting $p=0$ and $p=1$ into $I(A;B)-I(A;B|D)$ yields $0$, which shows that
$I(A;B)-I(A;B|D)$ is a non-negative function, and it is zero only at the
boundaries. This proves the theorem. ∎

######  Lemma 1.

Consider the non-linear dynamical system in (2) – (5). Let
$\alpha=\alpha_{1}^{s}=\alpha_{2}^{s}=\alpha_{1}^{sy}=\alpha_{1}^{sx}$ and
$\sigma^{2}_{W^{s}_{i-j}}=c\alpha^{-2},\forall j$, so that
$\alpha^{2}\sigma^{2}_{W^{s}_{i-j}}=c$. Then, for any $c>0$:

$\displaystyle\lim_{\alpha\to 0}$ $\displaystyle
I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1})$ $\displaystyle>\lim_{\alpha\to
0}I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1},S^{i-2}).$ (33)

###### Proof.

Let us first consider the left term in (33):

$\displaystyle
I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1})=I(Z^{i-1};\alpha^{xx}_{1}f_{x}(X_{i-1})$
$\displaystyle+\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}f_{s}(S_{i-1})+W^{x}_{i}|X^{i-1},Y^{i-1})$
$\displaystyle=I(Z^{i-1};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}f_{s}(S_{i-1})+W^{x}_{i}|X^{i-1},Y^{i-1})$
$\displaystyle\overset{(a)}{=}I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}f_{s}(S_{i-1})+W^{x}_{i}|X^{i-1},Y^{i-1})$
$\displaystyle\quad+I(Z_{i-1},Z^{i-3};\alpha^{sx}_{1}f_{s}(S_{i-1})+W^{x}_{i}|X^{i-1},Y^{i-1},Z_{i-2})$
$\displaystyle\overset{(b)}{=}I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}f_{s}(S_{i-1})+W^{x}_{i}|X^{i-1},Y^{i-1}),$
(34)

where $(a)$ follows from the chain rule of mutual information and $(b)$
follows since $W_{i}^{x},S_{i-1},Z^{i-1}$ are mutually independent and also
conditionally independent given $X^{i-1},Y^{i-1},Z_{i-2}$.

At this point, let
$\alpha=\alpha_{1}^{s}=\alpha_{2}^{s}=\alpha_{1}^{sy}=\alpha_{1}^{sx}$ and
$\sigma^{2}_{W^{s}_{i-j}}=c\alpha^{-2},\forall j$, so that
$\alpha^{2}\sigma^{2}_{W^{s}_{i-j}}=c$. The first-order (linear) series
expansion of $f_{s}$ is: $f_{s}(S_{i-1})=\beta W_{i-1}^{s}+o(\alpha)$, for
some constant $\beta$. We use this in order to rewrite (34) as:

$\displaystyle I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha
f_{s}(S_{i-1})+W^{x}_{i}|X^{i-1},Y^{i-1})$ (35)
$\displaystyle=I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha\beta
W_{i-1}^{s}+W^{x}_{i}$ $\displaystyle\quad+o(\alpha)|X^{i-1},Y^{i-1})$ (36)
$\displaystyle=I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+V+o(\alpha)|F),$ (37)

where $F=\\{X^{i-1},Y^{i-1}\\}$ and $V=\alpha\beta W_{i-1}^{s}+W^{x}_{i}$.
Note that $F$ is independent of $V$ and $F$ is asymptotically independent of
the terms within $o(\alpha)$ in the limit as $\alpha\to 0$. It follows that
for small $\alpha$, conditioning upon $F$ results in very little synergistic
information, since the noise $V$ added to $f_{z}(Z_{i-2})$ cannot be predicted
(and hence reduced) from knowledge of $F$.

We now consider the right side of (33), where we can use similar arguments as
above to obtain:

$\displaystyle I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1},S^{i-2})$
$\displaystyle=I(Z^{i-1};\alpha^{xx}_{1}f_{x}(X_{i-1})+\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}f_{s}(S_{i-1})$
$\displaystyle\quad+W^{x}_{i}|X^{i-1},Y^{i-1},S^{i-2})$
$\displaystyle=I(Z^{i-1};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}\beta
W_{i-1}^{s}+W^{x}_{i}$ $\displaystyle\quad+o(\alpha)|X^{i-1},Y^{i-1},S^{i-2})$
$\displaystyle=I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}\beta
W_{i-1}^{s}+W^{x}_{i}$ $\displaystyle\quad+o(\alpha)|X^{i-1},Y^{i-1},S^{i-2})$
$\displaystyle\quad+I(Z_{i-1},Z^{i-3};\alpha^{sx}_{1}\beta
W_{i-1}^{s}+W^{x}_{i}$
$\displaystyle\quad+o(\alpha)|X^{i-1},Y^{i-1},S^{i-2},Z_{i-2})$
$\displaystyle=I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+\alpha^{sx}_{1}\beta
W_{i-1}^{s}$ (38)
$\displaystyle\quad+W^{x}_{i}+o(\alpha)|X^{i-1},Y^{i-1},S^{i-2})$
$\displaystyle=I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+V+o(\alpha)|F,S^{i-2}).$

Since $S^{i-2}$ and $F$ are independent of $W_{i-1}^{s}$, it follows that
knowing $S^{i-2}$ and $F$ do not provide further synergistic information than
what is possible knowing only $F$, at least asymptotically in $\alpha$.

Let us now consider the potential additional reduction in common information
between $Z_{i-2}$ and $\alpha^{zx}_{2}f_{z}(Z_{i-2})+V$ due to knowledge of
$S^{i-2}$ in addition to $F$. Towards that end, let us define $\phi$ as a
conditional sufficient statistics of $X_{i-1}$ and $Y_{i-1}$ with respect to
$Z_{i-2}$ conditioned upon
$F^{\prime}=F\backslash\\{X_{i-1},Y_{i-1}\\}=\\{X^{i-2},Y^{i-2}\\}$. From (2)
and (3) it follows that $X_{i-1}$ and $Y_{i-1}$ can be written as:

$\displaystyle X_{i-1}$
$\displaystyle=\alpha^{xx}_{1}f_{x}(X_{i-2})+\alpha^{zx}_{2}f_{z}(Z_{i-3})+\alpha^{sx}_{1}f_{s}(S_{i-2})+W^{x}_{i-1}$
$\displaystyle=\alpha^{xx}_{1}f_{x}(X_{i-2})+\alpha^{zx}_{2}f_{z}(Z_{i-3})+\alpha^{sx}_{1}\beta
W_{i-2}^{s}+W^{x}_{i-1}$ $\displaystyle\quad+o(\alpha).$ $\displaystyle
Y_{i-1}$
$\displaystyle=\alpha^{yy}_{1}f_{y}(Y_{i-2})+\alpha^{xy}_{2}f_{x}(X_{i-3})+\alpha^{sx}_{1}\beta^{\prime}W_{i-2}^{s}$
$\displaystyle\quad+\alpha^{zy}_{2}f_{z}(Z_{i-3})+W^{y}_{i-1}+o(\alpha).$

It may be deduced that
$\phi=(\alpha^{zx}_{2}f_{z}(Z_{i-3})+\alpha^{sx}_{1}\beta
W_{i-1}^{s}+W^{x}_{i-1},\alpha^{zy}_{2}f_{z}(Z_{i-3})+\alpha^{sx}_{1}\beta^{\prime}W_{i-2}^{s}+W^{y}_{i-1})$
is (asymptotically in $\alpha\to 0$) a sufficient statistics for $Z_{i-2}$
given $F^{\prime}$. Thus, the following conditional Markov chain holds
asymptotically in $\alpha$:

$\displaystyle
Z_{i-2}|_{F^{\prime}}-\phi|_{F^{\prime}}-(X_{i-1},Y_{i-1})|_{F^{\prime}}.$
(39)

Consider now $(\alpha^{zx}_{2}f_{z}(Z_{i-4})+\alpha^{sx}_{1}\beta
W_{i-2}^{s}+W^{x}_{i-2},\alpha^{zy}_{2}f_{z}(Z_{i-4})+\alpha^{sx}_{1}\beta^{\prime}W_{i-3}^{s}+W^{y}_{i-2})$,
which is asymptotically a conditional sufficient statistics for $Z_{i-2}$
given $X_{i-2}$ and $X_{i-2}$. Iteratively applying the above arguments proves
that

$\displaystyle\bar{\phi}$
$\displaystyle\triangleq\\{\alpha^{zx}_{2}f_{z}(Z_{i-j-2})+\alpha^{sx}_{1}\beta
W_{i-j}^{s}+W^{x}_{i-j},\alpha^{zy}_{2}f_{z}(Z_{i-j-2})$
$\displaystyle\quad+\alpha^{sx}_{1}\beta^{\prime}W_{i-j}^{s}+W^{y}_{i-j}\\}_{j=1}^{i-2}$

is a sufficient statistics for $\\{X^{i-1},Y^{i-1}\\}$ with respect to
$Z_{i-2}$. Thus, for any $c>0$:

$\displaystyle\lim_{\alpha\to 0}I(Z_{i-2}$
$\displaystyle;\alpha^{zx}_{2}f_{z}(Z_{i-2})+V|F)$
$\displaystyle=\lim_{\alpha\to
0}I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+V|\bar{\phi}).$

Let us now consider the case, where we condition upon $F$ and $S^{i-2}$. Then
$\alpha^{zx}_{2}f_{z}(Z_{i-3})+W^{x}_{i-1}$ is a conditional sufficient
statistics for $X_{i-1}$ with respect to $Z_{i-2}$. It follows that
$\bar{\phi}^{s}$ given by

$\displaystyle\bar{\phi}^{s}\triangleq\\{\alpha^{zx}_{2}f_{z}(Z_{i-j-2})+W^{x}_{i-j},\alpha^{zy}_{2}f_{z}(Z_{i-j-2})+W^{y}_{i-j}\\}_{j=1}^{i-2}$

is a sufficient statistics for $(F,S^{i-2})$ with respect to $Z_{i-2}$. Thus,

$\displaystyle\lim_{\alpha\to 0}$ $\displaystyle
I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+V|F,S^{i-2})$
$\displaystyle=\lim_{\alpha\to
0}I(Z_{i-2};\alpha^{zx}_{2}f_{z}(Z_{i-2})+V|\bar{\phi}^{s}).$

Clearly, for any
$c>0,\mathbb{E}[Z_{i-2}|\bar{\phi}^{s}]<\mathbb{E}[Z_{i-2}|\bar{\phi}]$, i.e.,
the MMSE due to predicting $Z_{i-2}$ from $\bar{\phi}^{s}$ is strictly smaller
than when predicting from $\bar{\phi}$, since $\bar{\phi}^{s}$ is a less noisy
version of $Z_{i-3}$ than $\bar{\phi}^{s}$. Thus, we deduce that:

$\displaystyle\lim_{\alpha\to 0}$ $\displaystyle
I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1})$ $\displaystyle>\lim_{\alpha\to
0}I(Z^{i-1};X_{i}|X^{i-1},Y^{i-1},S^{i-2}).$

This proves the lemma. ∎

## References

  * [1] P. S. Baboukani, S. Theodoridis, and J. Østergaard, “A stimuli-relevant directed dependency index for time series,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5812–5816, 2022.
  * [2] A. Reid, D. Headley, R. Mill, R. Sanchez-Romero, L. Uddin, D. Marinazzo, D. Lurie, P. Valdés-Sosa, S. Hanson, B. Biswal, V. Calhoun, R. Poldrack, and M. Cole, “Advancing functional connectivity research from association to causation,” Nature Neuroscience, vol. 22, 2019.
  * [3] J. Pearl, “Causality: Models, reasoning, inference,” New York, NY: Cambridge University Press, 2009.
  * [4] S. Guo, K. Anil, A. Seth, K. Kendrik, C. Zhou, and J. Feng, “Partial granger causality—eliminating exogenous inputs and latent variables,” Journal of Neuroscience Methods, vol. 172, no. 1, 2008.
  * [5] M. Eichler, “Causal inference with multiple time series: Principles and problems,” Philos. Trans. R. Soc. A., 2013.
  * [6] C. Granger, “Testing for causality - a personal viewpoint,” J. Econ. Dyn. Control, vol. 2, 1980.
  * [7] P. Baboukani, G. Azemi, B. Boashash, P. Colditz, and A. Omidvarnia, “A novel multivariate phase synchrony measure: Application to multichannel newborn eeg analysis,” Digital Signal Processing, vol. 84, pp. 59–68, 2019.
  * [8] J. Massey, “Causality, feedback and directed information,” The International Symposium on Information Theory and Its Applications, 1990.
  * [9] T. Schreiber, “Measuring information transfer,” Phys. Rev. Lett., vol. 85, 2000.
  * [10] K. Friston, “Functional and effective connectivity: a review,” Brain Connect, vol. 1, 2011.
  * [11] P. Baboukani, C. Graversen, E. Alickovic, and J. Østergaard, “Estimating conditional transfer entropy in time series using mutual information and nonlinear prediction,” Entropy, vol. 22, no. 10, 2020.
  * [12] M. Wibral, V. P. J. Kay, J. Lizier, and W. Philips, “Partial information decomposition as a unified approach to the specification of neural goal functions,” Brain and Cognition, vol. 112, 2017.
  * [13] R. James, J. Emenheiser, and J. Crutchfield, “Unique information and secret key agreement,” MDPI Entropy, vol. 21, no. 1, 2021.
  * [14] P. Mediano, F. Rosas, A. Luppi, R. Carhart-Harris, D. Bor, A. Seth, and A. Barrett, “Towards an extended taxonomy of information dynamics via integrated information decomposition,” arxiv.org:, 2021.
  * [15] F. Rosas, P. Mediano, M. Gastpar, and H. Jensen, “Quantifying high-order interdependencies via multivariate extensions of the mutual information,” arxiv.org, 2019.
  * [16] F. Rosas, P. Mediano, H. Jensen, A. Seth, A. Barrett, R. Carhart-Harris, and D. Bor, “Reconciling emergences: An information- theoretic approach to identify causal emergence in multivariate data,” PLoS Computational Biology, vol. 16, no. 12, 2021.
  * [17] T. Varley, M. Pope, J. Faskowitz, and O. Sporns, “Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex,” arxiv.org, 2022.
  * [18] A. Luppi, P. Mediano, F. Rosas, N. Holland, T. Fryer, J. O’Brien, J. Rowe, D. Menon, D.Bor, and E. Stamatakis, “A synergistic core for human brain evolution and cognition,” Nature Neuroscience, vol. 25, 2022.
  * [19] T. Scagliarini, D. Nuzzi, Y. Antonacci, L. Faes, F. Rosas, D. Marinazzo, and S. Stramaglia, “Gradients of o-information: low-order descriptors of high-order dependencies,” arxiv.org, 2022.
  * [20] L. Faes, D. Marinazzo, G. Nollo, and A. Porta, “An information-theoretic framework to map the spatiotemporal dynamics of the scalp electroencephalogram,” IEEE Transactions on Biomedical Engineering, vol. 63, 2016.
  * [21] G. Kramer, “Capacity results for the discrete memoryless network,” IEEE Transactions on Information Theory, vol. 49, no. 1, pp. 4–21, 2003.
  * [22] M. Lindner, R. Vicente, V. Priesemann, and et al., “Trentool: A matlab open source toolbox to analyse information flow in time series data with transfer entropy,” BMC Neuroscience, vol. 12, no. 119, 2011.
  * [23] “Earth system research laboratory.,” Available online: http://math.bu.edu/people/kolaczyk/datasets.html.
  * [24] M. Kramer, E. Kolaczyk, and H. Kirsch, “Emergent network topology at seizure onset in humans.,” Epilepsy Res., vol. 79, pp. 173 – 186, 2008.
  * [25] S. Stramaglia, J. Cortes, and D. Marinazzo, “Synergy and redundancy in the granger causal analysis of dynamical networks.,” New J. Phys., vol. 16, 2014.
  * [26] L. Faes, D. Marinazzo, and S. Stramaglia, “Multiscale information decomposition: Exact computation for multivariate gaussian processes.,” Entropy, 2017.