#  Fundamental Inequalities and Identities Involving Mutual and Directed
Informations in Closed-Loop Systems

Milan S. Derpich, Eduardo I. Silva and Jan Østergaard  M.S. Derpich and E.I
Silva are with the Department of Electronic Engineering, Universidad Técnica
Federico Santa María, Casilla 110-V, Valparaíso, Chile (email:
<EMAIL_ADDRESS>eduardo.silva@usm.cl). Their work was supported in part
by CONICYT through grants FONDECYT Nr. 1120468, Nr. 1130459, and Anillo
ACT-53. J. Østergaard is with the Department of Electronic Systems, Aalborg
University, Niels Jernes Vej 12, DK-9220, Aalborg, Denmark (email:
janoe@ieee.org).

###### Abstract

We present several novel identities and inequalities relating the mutual
information and the directed information in systems with feedback. The
internal blocks within such systems are restricted only to be causal mappings,
but are allowed to be non-linear, stochastic and time varying. Moreover, the
involved signals can be arbitrarily distributed. We bound the directed
information between signals inside the feedback loop by the mutual information
between signals inside and outside the feedback loop. This fundamental result
has an interesting interpretation as a law of conservation of information
flow. Building upon it, we derive several novel identities and inequalities,
which allow us to prove some existing information inequalities under less
restrictive assumptions. Finally, we establish new relationships between
nested directed informations inside a feedback loop. This yields a new and
general data-processing inequality for systems with feedback.

##  I Introduction

The notion of directed information introduced by Massey in [1] assesses the
amount of information that causally “flows” from a given random and ordered
sequence to another. For this reason, it has increasingly found use in diverse
applications, from characterizing the capacity of channels with feedback [1,
2, 3, 4], the rate distortion function under causality constraints [5],
establishing some of the fundamental limitations in networked control [6, 7,
8, 9, 10, 11], determining causal relationships in neural networks [12], to
portfolio theory and hypothesis testing [13], to name a few.

The directed information from a random111 Hereafter we use non-italic letters
(such as $\operatorname{x}$) for random variables, denoting a particular
realization by the corresponding italic character, $x$.  sequence
$\operatorname{x}^{k}$ to a random sequence $\operatorname{y}^{k}$ is defined
as

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k})\triangleq\sum\limits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{i}|\operatorname{y}^{i-1}),$
(1)

where the notation $\operatorname{x}^{i}$ represents the sequence
$\operatorname{x}(1),\operatorname{x}(2),\ldots,\operatorname{x}(i)$. The
causality inherent in this definition becomes evident when comparing it with
the mutual information between $\operatorname{x}^{k}$ and
$\operatorname{y}^{k}$, given by
$I(\operatorname{x}^{k},\operatorname{y}^{k})=\sum\nolimits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{k}|\operatorname{y}^{i-1})$.
In the latter sum, what matters is the amount of information about the entire
sequence $\operatorname{x}^{k}$ present in $\operatorname{y}(i)$, given the
past values $\operatorname{y}^{i-1}$. By contrast, in the conditional mutual
informations in the sum of (1), only the past and current values of
$\operatorname{x}^{k}$ are considered, that is, $\operatorname{x}^{i}$. Thus,
$I(\operatorname{x}^{k}\to\operatorname{y}^{k})$ represents the amount of
information causally conveyed from $\operatorname{x}^{k}$ to
$\operatorname{y}^{k}$.

There exist several results characterizing the relationship between
$I(\operatorname{x}^{k}\to\operatorname{y}^{k})$ and
$I(\operatorname{x}^{k};\operatorname{y}^{k})$. First, it is well known that
$I(\operatorname{x}^{k}\to\operatorname{y}^{k})\leq
I(\operatorname{x}^{k};\operatorname{y}^{k})$, with equality if and only if
$\operatorname{y}^{k}$ is causally related to $\operatorname{x}^{k}$ [1]. A
conservation law of mutual and directed information has been found in [14],
which asserts that
$I(\operatorname{x}^{k}\to\operatorname{y}^{k})+I(0*\operatorname{y}^{k-1}\to\operatorname{x}^{k})=I(\operatorname{x}^{k};\operatorname{y}^{k})$,
where $0*\operatorname{y}^{k-1}$ denotes the concatenation
$0,\operatorname{y}(1),\ldots,\operatorname{y}^{k-1}$.

Given its prominence in settings involving feedback, it is perhaps in these
scenarios where the directed information becomes most important. For instance,
the directed information has been instrumental in characterizing the capacity
of channels with feedback (see, e.g., [3, 15, 4] and the references therein),
as well as the rate-distortion function in setups involving feedback [16, 9,
11, 10, 5].

In this paper, our focus is on the relationships (inequalities and identities)
involving directed and mutual informations within feedback systems, as well as
between directed informations involving different signals within the
corresponding feedback loop. In order to discuss some of the existing results
related to this problem, it is convenient to consider the general feedback
system shown in Fig. 1-(a). In this diagram, the blocks
$\mathcal{S}_{1},\ldots,\mathcal{S}_{4}$ represent possibly non-linear and
time-varying causal systems such that the total delay of the loop is at least
one sample. In the same figure,
$\operatorname{r},\operatorname{p},\operatorname{s},\operatorname{q}$ are
exogenous random signals (scalars, vectors or sequences), which could
represent, for example, any combination of disturbances, noises, random
initial states or side informations. We note that any of these exogenous
signals, in combination with its corresponding deterministic mapping
$\mathcal{S}_{i}$, can also yield any desired stochastic causal mapping.

For the simple case in which all the systems $\\{\mathcal{S}_{i}\\}_{i=1}^{4}$
are linear time invariant (LTI) and stable, and assuming
$\operatorname{p},\operatorname{x},\operatorname{q}=0$ (deterministically), it
was shown in [17] that $I(\operatorname{r}^{k}\to\operatorname{e}^{k})$ does
not depend on whether there is feedback from $\operatorname{e}$ to
$\operatorname{u}$ or not. Inequalities between mutual and directed
informations in a less restricted setup, shown in Fig. 1-(b), have been found
in [7, 8]. In that setting (a networked-control system), $G$ is a strictly
causal LTI dynamic system having (vector) state sequence
$\\{\mathbf{x}(i)\\}_{i=0}^{\infty}$, with
$\mathbf{x}_{0}\triangleq\mathbf{x}(0)$ being the random initial state in its
state-space representation. The external signal $\operatorname{r}$ (which
could correspond to a disturbance) is statistically independent of
$\operatorname{s}$, the latter corresponding to, for example, side information
or channel noise. Both are also statistically independent of $\mathbf{x}_{0}$.

$\operatorname{x}$$\operatorname{y}$$\operatorname{s}$$\operatorname{p}$$\operatorname{r}$$\operatorname{q}$$\mathcal{S}_{1}$$\mathcal{S}_{3}$$\mathcal{S}_{4}$$\mathcal{S}_{2}$$\operatorname{u}$$G$$\mathbf{x}_{0}$$D$$E$$\operatorname{x}$$\operatorname{e}$$\operatorname{u}$$f$$\operatorname{w}$$\operatorname{r}$$\operatorname{e}$$\operatorname{y}$$\operatorname{s}$(a)(b)

Figure 1: (a): The general system considered in this work. (b): A special
case, corresponding to the closed-loop system studied in [7].

The blocks labeled $E$, $D$ and $f$ correspond to an encoder, a decoder and a
channel, respectively, all of which are causal. The channel $f$ maps
$\operatorname{s}^{k}$ and $\operatorname{x}^{k}$ to $\operatorname{y}(k)$ in
a possibly time-varying manner, i.e.,
$\operatorname{y}(k)=f(k,\operatorname{x}^{k},\operatorname{s}^{k}).$
Similarly, the concatenation of the encoder, the channel and the decoder, maps
$\operatorname{s}^{k}$ and $\operatorname{w}^{k}$ to $\operatorname{u}(k)$ as
a possibly time-dependent function
$\operatorname{u}(k)=\psi(k,\operatorname{w}^{k},\operatorname{s}^{k}).$ Under
these assumptions, the following fundamental result was shown in [8, Lemma
5.1]:

$\displaystyle
I(\mathbf{x}_{0},\operatorname{r}^{k}\,;\,\operatorname{u}^{k})-I(\operatorname{r}^{k};\operatorname{u}^{k})\geq
I(\mathbf{x}_{0};\operatorname{e}^{k}).$ (2)

By further assuming in [8] that the decoder $D$ in Fig. 1-(b) is
deterministic, the following Markov chain naturally holds,

$\displaystyle(\mathbf{x}_{0},\operatorname{r}^{k})\longleftrightarrow\operatorname{y}^{k}\longleftrightarrow\operatorname{u}^{k},$
(3)

leading directly to

$\displaystyle
I(\mathbf{x}_{0},\operatorname{r}^{k}\,;\,\operatorname{y}^{k})-I(\operatorname{r}^{k};\operatorname{u}^{k})\geq
I(\mathbf{x}_{0};\operatorname{e}^{k}),$ (4)

which is found in the proof of [8, Corollary 5.3]. The deterministic nature of
the decoder $D$ played a crucial role in the proof of this result, since
otherwise the Markov chain (3) does not hold, in general, due to the feedback
from $\operatorname{u}$ to $\operatorname{y}$.

Notice that both (2) and (4) provide lower bounds to the difference between
two mutual informations, each of them relating a signal external to the loop
(such as $\mathbf{x}_{0},\operatorname{r}^{k}$) to a signal internal to the
loop (such as $\operatorname{u}^{k}$ or $\operatorname{y}^{k}$). Instead, the
inequality

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})\geq
I(\operatorname{r}^{k};\operatorname{y}^{k}),$ (5)

which holds for the system in Fig. 1-(a) and appears in [1, Theorem 3] (and
rediscovered later in [6, Lemma 4.8.1]), involves the directed information
between two internal signals and the mutual information between the second of
these and an external sequence. A related bound, similar to (4) but involving
information rates and with the leftmost mutual information replaced by the
directed information from $\operatorname{x}^{k}$ to $\operatorname{y}^{k}$
(which are two signals internal to the loop), has been obtained in [7, Lemma
4.1]:

$\displaystyle\bar{I}(\operatorname{x}\to\operatorname{y})-\bar{I}(\operatorname{r};\operatorname{u})\geq\lim_{k\to\infty}\frac{I(\mathbf{x}(0);\operatorname{e}^{k})}{k},$
(6)

with
$\bar{I}(\operatorname{x}\to\operatorname{y})\triangleq\lim_{k\to\infty}\frac{1}{k}I(\operatorname{x}^{k}\to\operatorname{y}^{k})$
and
$\bar{I}(\operatorname{r};\operatorname{u})\triangleq\lim_{k\to\infty}\frac{1}{k}I(\operatorname{r}^{k};\operatorname{u}^{k})$,
provided $\sup_{i\geq
0}\operatorname{E}\left[\mathbf{x}(i)^{T}\mathbf{x}(i)\right]<\infty$. This
result relies on three assumptions: a) that the channel $f$ is memory-less and
satisfies a “conditional invertibility” property, b) a finite-memory
condition, and c) a fading-memory condition, these two related to the decoder
$D$ (see Fig. 1). It is worth noting that, as defined in [7], these
assumptions upon $D$ exclude the use of side information by the decoder and/or
the possibility of $D$ being affected by random noise or having a random
internal state which is non-observable (please see [7] for a detailed
description of these assumptions).

The inequality (5) has recently been extended in [4, Theorem 1], for the case
of discrete-valued random variables and assuming
$\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p},\operatorname{q})$,
as the following identity (written in terms of the signals and setup shown in
Fig. 1-(a)):

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k})=I(\operatorname{p}^{k},\operatorname{y}^{k})+I(\operatorname{x}^{k}\to\operatorname{y}^{k}|\operatorname{p}^{k}).$
(7)

Letting $\operatorname{q}=\operatorname{s}$ in Fig. 1-(a) and with the
additional assumption that
$(\operatorname{p},\operatorname{s})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{q}$, it was also shown in [4,
Theorem 1] that

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k})=I(\operatorname{p}^{k};\operatorname{y}^{k})+I(\operatorname{q}^{k-1};\operatorname{y}^{k})+I(\operatorname{p}^{k};\operatorname{q}^{k-1}|\operatorname{y}^{k}),$
(8)

for the cases in which
$\operatorname{u}(i)=\operatorname{y}(i)+\operatorname{q}(i)$ (i.e., when the
concatenation of $\mathcal{S}_{4}$ and $\mathcal{S}_{1}$ corresponds to a
summing node). In [4], (7) and (8) play important roles in characterizing the
capacity of channels with noisy feedback.

To the best of our knowledge, (2), (4), (5) (6), (7) and (8) are the only
results available in the literature which lower bound the difference between
an internal-to-internal directed information and an external-to-internal
mutual information. There exist even fewer published results in relation to
inequalities between two directed informations involving only signals internal
to the loop. To the best of our knowledge, the only inequality of this type in
the literature is the one found in the proof of Theorem 4.1 of [9]. The latter
takes the form of a (quasi) data-processing inequality for directed
informations in closed-loop systems, and states that

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k}\parallel\operatorname{q}^{k})\geq
I(\operatorname{x}^{k}\to\operatorname{u}^{k}),$ (9)

provided222Here, and in the sequel, we use the notation
$\operatorname{x}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{y}$ to mean “$\operatorname{x}$
is independent of $\operatorname{y}$”.
$\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$ and if
$\mathcal{S}_{4}$ is such that $\operatorname{y}^{i}$ is a function of
$(\operatorname{u}^{i},\operatorname{q}^{i})$ (i.e., if $\mathcal{S}_{4}$ is
conditionally invertible) $\forall i$. In (9),

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k}\parallel\operatorname{q}^{k})\triangleq\sum\limits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{i}|\operatorname{y}^{i-1},\operatorname{q}^{i})$
(10)

corresponds to the causally conditioned directed information defined in [2].
Inequality (9) plays a crucial role [9], since it allowed lower bounding the
average data rate across a digital error-free channel by a directed
information. (In [9], $\operatorname{q}$ corresponded to a random dither
signal in an entropy-coded dithered quantizer.)

In this paper, we derive a set of information identities and inequalities
involving pairs of sequences (internal or external to the loop) in feedback
systems. The first of these is an identity which, under an independence
condition, can be interpreted as a law of conservation of information flows.
The latter identity is the starting point for most of the results which follow
it. Among other things, we extend (4) and (6) to the general setup depicted in
Fig. 1-(a), where none of the assumptions made in [7, 8, 9] (except causality)
needs to hold. Moreover, we will prove the validity of (9) without assuming
the conditional invertibility of $\mathcal{S}_{4}$ nor that
$\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$. The
latter result is one of four novel data-processing inequalities derived in
Section III-B, each involving two nested directed informations valid for the
system depicted in Fig. 1-(a). The last of these is a complete closed-loop
counterpart of the traditional open-loop data-processing inequality.

The remainder of this paper begins with a description of the systems under
study and the extension of Massey’s directed information to the case in which
each of the blocks in the loop may introduce an arbitrary, non-negative delay
(i.e., we do not allow for anticipation). The information identities and
inequalities are presented in Section III. For clarity of the exposition, all
the proofs are deferred to Section IV. A brief discussion of potential
applications of our results is presented in Section V, which is followed by
the conclusions in Section VI.

##  II Preliminaries

###  II-A System Description

We begin by providing a formal description of the systems labeled
$\mathcal{S}_{1}\ldots\mathcal{S}_{4}$ in Fig. 1-(a). Their input-output
relationships are given by the possibly-varying deterministic mappings333For
notational simplicity, we omit writing their time dependency explicitly.

$\displaystyle\operatorname{e}(i)$
$\displaystyle=\mathcal{S}_{1}(\operatorname{u}^{i-d_{1}(i)},\operatorname{r}^{i}),$
(11a) $\displaystyle\operatorname{x}(i)$
$\displaystyle=\mathcal{S}_{2}(\operatorname{e}^{i-d_{2}(i)},\operatorname{p}^{i}),$
(11b) $\displaystyle\operatorname{y}(i)$
$\displaystyle=\mathcal{S}_{3}(\operatorname{x}^{i-d_{3}(i)},\operatorname{s}^{i}),$
(11c) $\displaystyle\operatorname{u}(i)$
$\displaystyle=\mathcal{S}_{4}(\operatorname{y}^{i-d_{4}(i)},\operatorname{q}^{i}),$
(11d)

where $\operatorname{r},\operatorname{p},\operatorname{s},\operatorname{q}$
are exogenous random signals and the (possibly time-varying) delays
$d_{1},d_{2},d_{3},d_{4}\in\\{0,1,\ldots\\}$ are such that

$d_{1}(k)+d_{2}(k)+d_{3}(k)+d_{4}(k)\geq 1,\;\;\;\;\forall k\in\mathbb{N}.$

That is, the concatenation of $\mathcal{S}_{1},\ldots,\mathcal{S}_{4}$ has a
delay of at least one sample. For every $i\in\\{1,\ldots,k\\}$,
$\operatorname{r}(i)\in\mathbb{R}^{n_{\operatorname{r}}(i)}$, i.e.,
$\operatorname{r}(i)$ is a real random vector whose dimension is given by some
function $n_{\operatorname{r}}:\\{1,\ldots,k\\}\to\mathbb{N}$. The other
sequences
($\operatorname{q},\operatorname{p},\operatorname{s},\operatorname{x},\operatorname{y},\operatorname{u}$)
are defined likewise.

###  II-B A Necessary Modification of the Definition of Directed Information

As stated in [1], the directed information (as defined in (1)) is a more
meaningful measure of the flow of information between $\operatorname{x}^{k}$
and $\operatorname{y}^{k}$ than the conventional mutual information
$I(\operatorname{x}^{k};\operatorname{y}^{k})=\sum\nolimits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{k}|\operatorname{y}^{i-1})$
when there exists causal feedback from $\operatorname{y}$ to
$\operatorname{x}$. In particular, if $\operatorname{x}^{k}$ and
$\operatorname{y}^{k}$ are discrete-valued sequences, input and output,
respectively, of a forward channel, and if there exists strictly causal,
perfect feedback, so that $\operatorname{x}(i)=\operatorname{y}(i-1)$ (a
scenario utilized in [1] as part of an argument in favor of the directed
information), then the mutual information becomes

$\displaystyle I(\operatorname{x}^{k};\operatorname{y}^{k})$
$\displaystyle=H(\operatorname{y}^{k})-H(\operatorname{y}^{k}|\operatorname{x}^{k})=H(\operatorname{y}^{k})-H(\operatorname{y}^{k}|\operatorname{y}^{k-1})=H(\operatorname{y}^{k})-H(\operatorname{y}(k)|\operatorname{y}^{k-1})=H(\operatorname{y}^{k-1}).$

Thus, when strictly causal feedback is present,
$I(\operatorname{x}^{k};\operatorname{y}^{k})$ fails to account for how much
information about $\operatorname{x}^{k}$ has been conveyed to
$\operatorname{y}^{k}$ through the forward channel that lies between them.

It is important to note that, in [1] (as well as in many works concerned with
communications), the forward channel is instantaneous, i.e., it has no delay.
Therefore, if a feedback channel is utilized, then this feedback channel must
have a delay of at least one sample, as in the example above. However, when
studying the system in Fig. 1-(a), we may need to evaluate the directed
information between signals $\operatorname{x}^{k}$ and $\operatorname{y}^{k}$
which are, respectively, input and output of a strictly casual forward channel
(i.e., with a delay of at least one sample), whose output is instantaneously
fed back to its input. In such case, if one further assumes perfect feedback
and sets $\operatorname{x}(i)=\operatorname{y}(i)$, then, in the same spirit
as before,

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})$
$\displaystyle=\sum\limits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{i}|\operatorname{y}^{i-1})=\sum\limits_{i=1}^{k}\left[H(\operatorname{y}(i)|\operatorname{y}^{i-1})-H(\operatorname{y}(i)|\operatorname{x}^{i},\operatorname{y}^{i-1})\right]=H(\operatorname{y}^{k}).$

As one can see, Massey’s definition of directed information ceases to be
meaningful if instantaneous feedback is utilized.

It is natural to solve this problem by recalling that, in the latter example,
the forward channel had a delay, say $d$, greater than one sample. Therefore,
if we are interested in measuring how much of the information in
$\operatorname{y}(k)$, not present in $\operatorname{y}^{i-1}$, was conveyed
from $\operatorname{x}^{i}$ through the forward channel, we should look at the
mutual information
$I(\operatorname{y}(i);\operatorname{x}^{i-d}|\operatorname{y}^{i-1})$,
because only the input samples $\operatorname{x}^{i-d}$ can have an influence
on $\operatorname{y}(i)$. For this reason, we introduce the following,
modified notion of directed information

######  Definition 1 (Directed Information with Forward Delay)

In this paper, the directed information from $\operatorname{x}^{k}$ to
$\operatorname{y}^{k}$ through a forward channel with a non-negative time
varying delay of $d(i)$ samples is defined as

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k})\triangleq\sum\limits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{i-d(i)}|\operatorname{y}^{i-1}).$
(12)

For a zero-delay forward channel, the latter definition coincides with
Massey’s.

Likewise, we adapt the definition of causally-conditioned directed information
to the definition

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k}\parallel\operatorname{e}^{k})\triangleq\sum\limits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{i-d_{3}(i)}|\operatorname{y}^{i-1},\operatorname{e}^{i-d_{2}(i)}).$

when the signals $\operatorname{e}$, $\operatorname{x}$ and $\operatorname{y}$
are related according to (11).

Before finishing this section, it is convenient to recall the following
identity (a particular case of the chain rule of conditional mutual
information [18]), which will be extensively utilized in the proofs of our
results:

$\displaystyle
I(\operatorname{a},\operatorname{b};\operatorname{c}|\operatorname{d})=I(\operatorname{b};\operatorname{c}|\operatorname{d})+I(\operatorname{a};\operatorname{c}|\operatorname{b},\operatorname{d}).$
(13)

##  III Information Identities and Inequalities

###  III-A Relationships Between Mutual and Directed Informations

We begin by stating a fundamental result, which relates the directed
information between two signals within a feedback loop, say $\operatorname{x}$
and $\operatorname{y}$, to the mutual information between an external set of
signals and $\operatorname{y}$:

###### Theorem 1

In the system shown in Fig. 1-(a), it holds that

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k})=I(\operatorname{q}^{k},\operatorname{r}^{k},\operatorname{p}^{k}\to\operatorname{y}^{k})-I(\operatorname{q}^{k},\operatorname{r}^{k},\operatorname{p}^{k}\to\operatorname{y}^{k}\parallel\operatorname{x}^{k})\leq
I(\operatorname{p}^{k},\operatorname{q}^{k},\operatorname{r}^{k}\,;\operatorname{y}^{k}),\;\;\;\;\forall
k\in\mathbb{N},$ (14)

with equality achieved if $\operatorname{s}$ is independent of
$(\operatorname{p},\operatorname{q},\operatorname{r})$. $\blacktriangle$

This fundamental result, which for the cases in which
$\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{p},\operatorname{q},\operatorname{r})$
can be understood as a law of conservation of information flow, is illustrated
in Fig. 2. For such cases, the information causally conveyed from
$\operatorname{x}$ to $\operatorname{y}$ equals the information flow from
$(\operatorname{q},\operatorname{r},\operatorname{p})$ to $\operatorname{y}$.
When $(\operatorname{p},\operatorname{q},\operatorname{r})$ are not
independent of $\operatorname{s}$, part of the mutual information between
$(\operatorname{p},\operatorname{q},\operatorname{r})$ and $\operatorname{y}$
(corresponding to the term
$I(\operatorname{q}^{k},\operatorname{r}^{k},\operatorname{p}^{k}\to\operatorname{y}^{k}\parallel\operatorname{x}^{k})$)
can be thought of as being “leaked” through $\operatorname{s}$, thus bypassing
the forward link from $\operatorname{x}$ to $\operatorname{y}$. This provides
an intuitive interpretation for (14).

$\operatorname{r}$$\mathcal{S}_{1}$$\operatorname{u}$$\operatorname{p}$$\mathcal{S}_{2}$$\operatorname{e}$$\operatorname{q}$$\mathcal{S}_{4}$$\operatorname{x}$$\operatorname{s}$$\mathcal{S}_{3}$$\operatorname{y}$

Figure 2: The flow of information between exogenous signals
$(\operatorname{p},\operatorname{q},\operatorname{r})$ and the internal signal
$\operatorname{y}$ equals the directed information from $\operatorname{x}^{k}$
to $\operatorname{y}^{k}$ when $\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{p},\operatorname{q},\operatorname{r})$.

###### Remark 1

Theorem 1 implies that $I(\operatorname{x}^{k}\to\operatorname{y}^{k})$ is
only a part of (or at most equal to) the information “flow” between all the
exogenous signals entering the loop outside the link
$\operatorname{x}\to\operatorname{y}$ (namely
$(\operatorname{q},\operatorname{r},\operatorname{p})$), and
$\operatorname{y}$. In particular, if
$(\operatorname{p},\operatorname{q},\operatorname{r})$ were deterministic,
then $I(\operatorname{x}^{k}\to\operatorname{y}^{k})=0$, regardless of the
blocks $\mathcal{S}_{1},\ldots,\mathcal{S}_{4}$ and irrespective of the nature
of $\operatorname{s}$. $\blacktriangle$

###### Remark 2

By using (13),
$I(\operatorname{p}^{k},\operatorname{q}^{k},\operatorname{r}^{k};\operatorname{y}^{k})=I(\operatorname{r}^{k};\operatorname{y}^{k})+I(\operatorname{p}^{k},\operatorname{q}^{k};\operatorname{y}^{k}|\operatorname{r}^{k})$.
Then, applying Theorem 1, we recover (5), whenever
$\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{q},\operatorname{r},\operatorname{p})$.
Thus, [1, Theorem 3] and [6, Lemma 4.8.1]) can be obtained as a corollary of
Theorem 1. $\blacktriangle$

The following result provides an inequality relating
$I(\operatorname{x}^{k}\to\operatorname{y}^{k})$ with the separate flows of
information $I(\operatorname{r}^{k};\operatorname{y}^{k})$ and
$I(\operatorname{p}^{k},\operatorname{q}^{k}\,;\,\operatorname{y}^{k})$.

###### Theorem 2

For the system shown in Fig. 1-(a), if
$\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{p},\operatorname{q},\operatorname{r})$
and $\operatorname{r}^{k}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{p}^{k},\operatorname{q}^{k})$,
then

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})$
$\displaystyle\geq
I(\operatorname{r}^{k};\operatorname{y}^{k})+I(\operatorname{p}^{k},\operatorname{q}^{k}\,;\,\operatorname{y}^{k}).$
(15)

with equality if and only if the Markov chain
$(\operatorname{p}^{k},\operatorname{q}^{k})\leftrightarrow\operatorname{y}^{k}\leftrightarrow\operatorname{r}^{k}$
holds.

Theorem 2 shows that, provided
$(\operatorname{p},\operatorname{q},\operatorname{r})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{s}$,
$I(\operatorname{x}^{k}\to\operatorname{y}^{k})$ is lower bounded by the sum
of the individual flows from all the subsets in any given partition of
$(\operatorname{p}^{k},\operatorname{q}^{k},\operatorname{r}^{k})$, to
$\operatorname{y}^{k}$, provided these subsets are mutually independent.
Indeed, both theorems 1 and 2 can be generalized for any appropriate choice of
external and internal signals. More precisely, let $\Theta$ be the set of all
external signals in a feedback system. Let $\alpha$ and $\beta$ be two
internal signals in the loop. Define $\Theta_{\alpha,\beta}\subset\Theta$ as
the set of exogenous signals which are introduced to the loop at every
subsystem $\mathcal{S}_{i}$ that lies in the path going from $\alpha$ to
$\beta$. Thus, for any $\rho\in\Theta\setminus\Theta_{\alpha,\beta}$, if
$\Theta_{\alpha,\beta}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\Theta\setminus\Theta_{\alpha,\beta}$, we have
that (14) and (15) become

$\displaystyle I(\alpha\to\beta)$
$\displaystyle=I(\Theta\setminus\\{\Theta_{\alpha,\beta}\\};\beta),$ (16)
$\displaystyle I(\alpha\to\beta)-I(\rho;\beta)$ $\displaystyle\geq
I(\Theta\setminus\\{\rho\cup\Theta_{\alpha,\beta}\\};\beta),$ (17)

respectively.

To finish this section, we present a stronger, non-asymptotic version of
inequality (6):

###### Theorem 3

In the system shown in Fig. 1-(a), if
$(\operatorname{r},\operatorname{p},\operatorname{q},\operatorname{s})$ are
mutually independent, then

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})$
$\displaystyle=I(\operatorname{r}^{k};\operatorname{u}^{k})+I(\operatorname{p}^{k};\operatorname{e}^{k})+I(\operatorname{q}^{k};\operatorname{y}^{k})+I(\operatorname{p}^{k};\operatorname{u}^{k}|\operatorname{e}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k}).$
(18)

$\blacktriangle$

As anticipated, Theorem 3 can be seen as an extension of (6) to the more
general setup shown in Fig. 1-(a), where the assumptions made in [7, Lemma
4.1] do not need to hold. In particular, letting the decoder $D$ and
$\operatorname{x}_{0}$ in Fig. 1-(b) correspond to $\mathcal{S}_{4}$ and
$\operatorname{p}^{k}$ in Fig. 1-(a), respectively, we see that inequality
(15) holds even if $D$ and $E$ have dependent initial states, or if the
internal state of $D$ is not observable [19].

Theorem 3 also admits an interpretation in terms of information flows. This
can be appreciated in the diagram shown in Fig. 3, which depicts the
individual full-turn flows (around the entire feedback loop) stemming from
$\operatorname{q}$, $\operatorname{r}$ and $\operatorname{p}$. Theorem 3
states that the sum of these individual flows is a lower bound for the
directed information from $\operatorname{x}$ to $\operatorname{y}$, provided
$\operatorname{q},\operatorname{r},\operatorname{p},\operatorname{s}$ are
independent.

$\operatorname{r}$$\mathcal{S}_{1}$$\operatorname{u}$$\operatorname{p}$$\mathcal{S}_{2}$$\operatorname{e}$$\operatorname{q}$$\mathcal{S}_{4}$$\operatorname{x}$$\operatorname{s}$$\mathcal{S}_{3}$$\operatorname{y}$

Figure 3: A representation of the three first information flows on the right-
hand-side of (18).

###  III-B Relationships Between Nested Directed Informations

This section presents three closed-loop versions of the data processing
inequality relating two directed informations, both between pairs of signals
internal to the loop. As already mentioned in Section I, to the best of our
knowledge, the first inequality of this type to appear in the literature is
the one in Theorem 4.1 in [9] (see (9)). Recall that the latter result stated
that
$I(\operatorname{x}^{k}\to\operatorname{y}^{k}\parallel\operatorname{q}^{k})\geq
I(\operatorname{x}^{k}\to\operatorname{u}^{k})$, requiring $\mathcal{S}_{4}$
to be such that $\operatorname{y}^{i}$ is a deterministic function of
$(\operatorname{u}^{i},\operatorname{q}^{i})$ and that
$\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$. The
following result presents another inequality which also relates two nested
directed informations, namely,
$I(\operatorname{x}^{k}\to\operatorname{y}^{k})$ and
$I(\operatorname{e}^{k}\to\operatorname{y}^{k})$, but requiring only that
$\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{q},\operatorname{r},\operatorname{p})$.

###### Theorem 4

For the closed-loop system in Fig. 1-(b), if
$(\operatorname{q},\operatorname{r},\operatorname{p})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{s}$, then

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})$
$\displaystyle\geq I(\operatorname{e}^{k}\to\operatorname{y}^{k}).$ (19)

$\blacktriangle$

Notice that Theorem 4 does not require $\operatorname{p}$ to be independent of
$\operatorname{r}$ or $\operatorname{q}$. This may seem counter-intuitive upon
noting that $\operatorname{p}$ enters the loop between the link from
$\operatorname{e}$ to $\operatorname{x}$.

The following theorem is an identity between two directed informations
involving only internal signals. It can also be seen as a complement to
Theorem 4, since it can be directly applied to establish the relationship
between $I(\operatorname{e}^{k}\to\operatorname{y}^{k})$ and
$I(\operatorname{e}^{k}\to\operatorname{u}^{k})$.

###### Theorem 5

For the system shown in Fig. 1-(a), if
$(\operatorname{q},\operatorname{s})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$, then

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})\leq
I(\operatorname{x}^{k}\to\operatorname{u}^{k})+I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k})+I(\operatorname{q}^{k};\operatorname{r}^{k}|\operatorname{u}^{k},\operatorname{y}^{k}).$
(20)

with equality if, in addition, $\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{s}$. In the latter case, it
holds that

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k})=I(\operatorname{x}^{k}\to\operatorname{u}^{k})+I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k}).$
(21)

$\blacktriangle$

Notice that, by requiring additional independence conditions upon the
exogenous signals (specifically, $\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{s}$), Theorem 5 (and, in
particular, (21)) yields

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})\geq
I(\operatorname{x}^{k}\to\operatorname{u}^{k}),$ (22)

which strengthens the inequality in [9, Theorem 4.1] (stated above in (9)).
More precisely, (22) does not require conditioning one of the directed
informations and holds irrespective of the invertibility of the mappings in
the loop.

A closer counterpart of (9) (i.e., of [9, Theorem 4.1]), involving
$I(\operatorname{x}^{k}\to\operatorname{y}^{k}\parallel\operatorname{q}^{k})$,
is presented next.

###### Theorem 6

For the system shown in Fig. 1-(a), if
$(\operatorname{q},\operatorname{s})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$, then

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k}|\operatorname{q}^{k})=I(\operatorname{x}^{k}\to\operatorname{u}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k})+I(\operatorname{q}^{k};\operatorname{r}^{k}|\operatorname{u}^{k},\operatorname{y}^{k})\overset{(\dagger)}{=}I(\operatorname{x}^{k}\to\operatorname{y}^{k}\parallel\operatorname{q}^{k}).$
(23)

where the equality labeled $(\dagger)$ hods if, in addition, the Markov chain

$\displaystyle\operatorname{q}_{i+1}^{k}\longleftrightarrow\operatorname{q}^{i}\longleftrightarrow\operatorname{s}^{i}$
(24)

is satisfied for all $i\in\\{1,\ldots,k\\}$. $\blacktriangle$

Thus, provided $(\operatorname{q},\operatorname{s})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$, (23)
yields that (9) holds regardless of the invertibility of $\mathcal{S}_{4}$,
requiring instead that, for all $i\in\\{1,\ldots,k\\}$, any statistical
dependence between $\operatorname{q}^{k}$ and $\operatorname{s}^{i}$ resides
only in $\operatorname{q}^{i}$ (i.e., that Markov chain (24) holds).

The results derived so far relate directed informations having either the same
“starting” sequence or the same “destination” sequence. We finish this section
with the following corollary, which follows directly by combining theorems 4
and 5 and relates directed informations involving four different sequences
internal to the loop.

######  Corollary 1 (Full Closed-Loop Directed Data Processing Inequality)

For the system shown in Fig. 1-(a), if
$(\operatorname{q},\operatorname{s})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$ and
$\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{s}$, then

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k})\overset{(a)}{\geq}I(\operatorname{e}^{k}\to\operatorname{u}^{k})+I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k};\operatorname{y}^{k}|\operatorname{u}^{k})\geq
I(\operatorname{e}^{k}\to\operatorname{u}^{k}).$ (25)

Equality holds in $(a)$ if, in addition,
$\operatorname{r}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{p}$ (i.e., if
$(\operatorname{q},\operatorname{r},\operatorname{p},\operatorname{s})$ are
mutually independent). $\blacktriangle$

To the best of our knowledge, Corollary 1 is the first result available in the
literature providing a lower bound to the gap between two nested directed
informations, involving four different signals inside the feedback loop. This
result can be seen as the first full extension of the open-loop (traditional)
data-processing inequality, to arbitrary closed-loop scenarios. (Notice that
there is no need to consider systems with more than four mappings, since all
external signals entering the loop between a given pair of internal signals
can be regarded as exogenous inputs to a single equivalent deterministic
mapping.)

##  IV Proofs

We start with the proof of Theorem 1.

###### Proof:

It is clear from Fig. 1-(a) and from (11) that the relationship between
$\operatorname{r}$, $\operatorname{p}$, $\operatorname{q}$,
$\operatorname{s}$, $\operatorname{x}$ and $\operatorname{y}$ can be
represented by the diagram shown in Fig. 4.

$q^{i}$$p^{i}$$r^{i}$$\theta^{i}$$s^{i}$$y(i)$$x_{i+1-d_{3}}^{i-1+d_{1}+d_{2}+d_{4}}$$x^{i-d_{3}}$$y^{i-1}$

Figure 4: Representation of the system of Fig. 1-(b) highlighting the
dependency between $p$, $q$, $r$, $s$, $x$ and $y$. The dependency on $i$ of
the delays $d_{1}(i),\ldots,d_{4}(i)$ is omitted for clarity.

From this diagram and Lemma 1 (in the appendix) it follows that if
$\operatorname{s}$ is independent of
$(\operatorname{r},\operatorname{p},\operatorname{q})$, then the following
Markov chain holds:

$\displaystyle\operatorname{y}(i)$
$\displaystyle\longleftrightarrow(\operatorname{x}^{i-d_{3}(i)},\operatorname{y}^{i-1})\longleftrightarrow(\operatorname{p}^{i},\operatorname{q}^{i},\operatorname{r}^{i}).$
(26)

Denoting the triad of exogenous signals
$\operatorname{p}^{k},\operatorname{q}^{k},\operatorname{r}^{k}$ by

$\displaystyle\theta^{k}\triangleq(\operatorname{p}^{k},\operatorname{q}^{k},\operatorname{r}^{k}),$
(27)

we have the following

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{i})$
$\displaystyle=\sum\limits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{i-d_{3}(i)}|\operatorname{y}^{i-1})$
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}\sum\limits_{i=1}^{k}\left[I(\theta^{i},\operatorname{x}^{i-d_{3}(i)};\operatorname{y}(i)|\operatorname{y}^{i-1})-I(\theta^{i};\operatorname{y}(i)|\operatorname{x}^{i-d_{3}(i)},\operatorname{y}^{i-1})\right]$
$\displaystyle\overset{(a)}{=}\sum\limits_{i=1}^{k}\left[I(\theta^{i};\operatorname{y}(i)|\operatorname{y}^{i-1})-I(\theta^{i};\operatorname{y}(i)|\operatorname{x}^{i-d_{3}(i)},\operatorname{y}^{i-1})\right]$
(28a)
$\displaystyle\overset{(b)}{\leq}\sum\limits_{i=1}^{k}I(\theta^{i};\operatorname{y}(i)|\operatorname{y}^{i-1})\overset{(c)}{\leq}\sum\limits_{i=1}^{k}I(\theta^{k};\operatorname{y}(i)|\operatorname{y}^{i-1})$
(28b) $\displaystyle=I(\theta^{k};\operatorname{y}^{k}).$ (28c)

In the above, $(a)$ follows from the fact that, if $\operatorname{y}^{i-1}$ is
known, then $\operatorname{x}^{i-d_{3}(i)}$ is a deterministic function of
$\theta^{i}$. The resulting sums on the right-hand side of (28a) correspond to
$I(\operatorname{q}^{k},\operatorname{r}^{k},\operatorname{p}^{k}\to\operatorname{y}^{k})-I(\operatorname{q}^{k},\operatorname{r}^{k},\operatorname{p}^{k}\to\operatorname{y}^{k}\parallel\operatorname{x}^{k})$,
and thereby proving the first part of the theorem, i.e., the equality in (14).
In turn, $(b)$ stems from the non-negativity of mutual informations, turning
into equality if $\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p},\operatorname{q})$,
as a direct consequence of the Markov chain in (26). Finally, equality holds
in $(c)$ if $\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{q},\operatorname{r},\operatorname{p})$,
since $\operatorname{y}$ depends causally upon $\theta$. This shows that
equality in (14) is achieved if $\operatorname{s}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{q},\operatorname{r},\operatorname{p})$,
completing the proof. ∎

###### Proof:

Apply the chain-rule identity (13) to the RHS of (14) to obtain

$\displaystyle
I(\theta^{k};\operatorname{y}^{k})=I(\operatorname{p}^{k},\operatorname{q}^{k},\operatorname{r}^{k};\operatorname{y}^{k})=I(\operatorname{p}^{k},\operatorname{q}^{k};\operatorname{y}^{k}|\operatorname{r}^{k})+I(\operatorname{r}^{k};\operatorname{y}^{k}).$
(29)

Now, applying (13) twice, one can express the term
$I(\operatorname{p}^{k},\operatorname{q}^{k};\operatorname{y}^{k}|\operatorname{r}^{k})$
as follows:

$\begin{split}I(\operatorname{p}^{k},\operatorname{q}^{k};\operatorname{y}^{k}|\operatorname{r}^{k})&=I(\operatorname{p}^{k},\operatorname{q}^{k}\,;\,\operatorname{y}^{k},\operatorname{r}^{k})-I(\operatorname{p}^{k},\operatorname{q}^{k};\operatorname{r}^{k})=I(\operatorname{p}^{k},\operatorname{q}^{k}\,;\,\operatorname{y}^{k},\operatorname{r}^{k})\\\
&=I(\operatorname{p}^{k},\operatorname{q}^{k};\operatorname{y}^{k})+I(\operatorname{p}^{k},\operatorname{q}^{k};\operatorname{r}^{k}|\operatorname{y}^{k}),\end{split}$
(30)

where the second equality follows since
$(\operatorname{p}^{k},\operatorname{q}^{k})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{r}^{k}$. The result then follows
directly by combining (30) with (29) and (14). ∎

###### Proof:

Since $\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p},\operatorname{s})$,

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})$
$\displaystyle\overset{(a)}{=}I(\operatorname{x}^{k}\to\operatorname{u}^{k})+I(\operatorname{q}^{k};\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k})$
(31)
$\displaystyle\overset{(b)}{=}I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{u}^{k})+I(\operatorname{q}^{k};\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k})$
(32)
$\displaystyle\overset{(c)}{=}I(\operatorname{r}^{k};\operatorname{u}^{k})+I(\operatorname{p}^{k};\operatorname{u}^{k}|\operatorname{r}^{k})+I(\operatorname{q}^{k};\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k}),$
(33)

where $(a)$ is due to Theorem 5, $(b)$ follows from Theorem 1 and the fact
that $(\operatorname{s},\operatorname{q})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$ and $(c)$
from the chain rule of mutual information. For the second term on the RHS of
the last equation, we have

$\displaystyle
I(\operatorname{p}^{k};\operatorname{u}^{k}|\operatorname{r}^{k})$
$\displaystyle\overset{(a)}{=}I(\operatorname{p}^{k};\operatorname{u}^{k}|\operatorname{r}^{k})+I(\operatorname{p}^{k};\operatorname{r}^{k})=I(\operatorname{p}^{k};\operatorname{r}^{k},\operatorname{u}^{k})$
(34)
$\displaystyle\overset{(b)}{=}I(\operatorname{p}^{k};\operatorname{r}^{k},\operatorname{u}^{k},\operatorname{e}^{k})-I(\operatorname{p}^{k};\operatorname{e}^{k}|\operatorname{r}^{k},\operatorname{u}^{k})$
(35)
$\displaystyle\overset{(c)}{=}I(\operatorname{p}^{k};\operatorname{r}^{k},\operatorname{u}^{k},\operatorname{e}^{k})$
(36)
$\displaystyle\overset{(d)}{=}I(\operatorname{p}^{k};\operatorname{e}^{k})+I(\operatorname{p}^{k};\operatorname{r}^{k},\operatorname{u}^{k}|\operatorname{e}^{k})$
(37)
$\displaystyle\overset{(e)}{=}I(\operatorname{p}^{k};\operatorname{e}^{k})+I(\operatorname{p}^{k};\operatorname{u}^{k}|\operatorname{e}^{k})+I(\operatorname{p}^{k};\operatorname{r}^{k}|\operatorname{u}^{k},\operatorname{e}^{k})$
(38)
$\displaystyle\overset{(f)}{=}I(\operatorname{p}^{k};\operatorname{e}^{k})+I(\operatorname{p}^{k};\operatorname{u}^{k}|\operatorname{e}^{k}),$
(39)

where $(a)$ holds since $\operatorname{r}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{p}$, $(b)$, $(d)$ and $(e)$ stem
from the chain rule of mutual information (13), and $(c)$ is a consequence of
the Markov chain
$\operatorname{e}^{k}\leftrightarrow(\operatorname{u}^{k},\operatorname{r}^{k})\leftrightarrow\operatorname{p}^{k}$
which is due to the fact that
$\operatorname{e}^{k}=\mathcal{S}_{1}(\operatorname{u}^{k-d_{1}(k)},\operatorname{r}^{k})$.
Finally, $(f)$ is due to the Markov chain
$\operatorname{r}^{k}\leftrightarrow(\operatorname{u}^{k},\operatorname{e}^{k})\leftrightarrow\operatorname{p}^{k}$,
which holds because $\operatorname{r}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{p},\operatorname{s},\operatorname{q})$
as a consequence of Lemma 1 in the appendix (see also Fig. 1-(a)).
Substitution of (39) into (33) yields (18), thereby completing the proof. ∎

###### Proof:

Since
$(\operatorname{p},\operatorname{q},\operatorname{r})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{s}$, we can apply (5) (where now
$(\operatorname{q},\operatorname{r})$ plays the role of $\operatorname{r}$),
and obtain

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})\geq
I(\operatorname{q}^{k},\operatorname{r}^{k};\operatorname{y}^{k}).$ (40)

Now, we apply Theorem 1, which gives

$\displaystyle
I(\operatorname{q}^{k},\operatorname{r}^{k};\operatorname{y}^{k})\geq
I(\operatorname{e}^{k}\to\operatorname{y}^{k}),$ (41)

completing the proof. ∎

###### Proof:

Applying Theorem 1, since
$(\operatorname{r},\operatorname{p})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{s},\operatorname{q})$,

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{u}^{k})=I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k}).$
(42)

For the other directed information, we have that

$\displaystyle I(\operatorname{x}^{k}\to\operatorname{y}^{k})$
$\displaystyle\overset{(a)}{\leq}I(\operatorname{r}^{k},\operatorname{p}^{k},\operatorname{q}^{k}\,;\,\operatorname{y}^{k})$
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{y}^{k}|\operatorname{q}^{k})$
(43)
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k},\operatorname{y}^{k}|\operatorname{q}^{k})-I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{u}^{k}|\operatorname{q}^{k},\operatorname{y}^{k})$
$\displaystyle\overset{(b)}{=}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k},\operatorname{y}^{k}|\operatorname{q}^{k})$
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k},\operatorname{y}^{k},\operatorname{q}^{k})-I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{q}^{k})$
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k},\operatorname{q}^{k}|\operatorname{u}^{k})-I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{q}^{k})$
$\displaystyle\overset{(c)}{\leq}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k},\operatorname{q}^{k}|\operatorname{u}^{k})$
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k})+I(\operatorname{q}^{k};\operatorname{r}^{k}|\operatorname{u}^{k},\operatorname{y}^{k})$
(44)
$\displaystyle\overset{(d)}{\leq}I(\operatorname{q}^{k}\,;\,\operatorname{y}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k}),$
(45)

where $(a)$ follows from Theorem 1, which also states that equality is reached
if and only if
$(\operatorname{r},\operatorname{p},\operatorname{q})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{s}$. In turn, $(b)$ is due to
the fact that $\operatorname{u}^{k}$ is a deterministic function of
$\operatorname{q}^{k},\operatorname{y}^{k}$. Equality $(c)$ holds if and only
if $(r,\operatorname{p})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{q}$. Finally, from Lemma 1 (in
the appendix), $(d)$ turns into equality if
$\operatorname{q}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p},\operatorname{s})$.
Substitution of (42) into (45) yields (21), completing the proof. ∎

###### Proof:

We begin with the second part of the theorem, proving the validity of the
equality $(\dagger)$ in (23). We have the following:

$\displaystyle
I(\operatorname{x}^{k}\to\operatorname{y}^{k}\parallel\operatorname{q}^{k})$
$\displaystyle=\sum\limits_{i=1}^{k}I(\operatorname{y}(i);\operatorname{x}^{i-d_{3}(i)}|\operatorname{y}^{i-1},\operatorname{q}^{i})$
(46)
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}\sum\limits_{i=1}^{k}\left[I(\operatorname{r}^{i},\operatorname{p}^{i},\operatorname{x}^{i-d_{3}(i)};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i})-I(\operatorname{r}^{i},\operatorname{p}^{i};\operatorname{y}(i)|\operatorname{x}^{i-d_{3}(i)},\operatorname{y}^{i-1},\operatorname{q}^{i})\right]$
(47)
$\displaystyle\overset{(a)}{\leq}\sum\limits_{i=1}^{k}I(\operatorname{r}^{i},\operatorname{p}^{i},\operatorname{x}^{i-d_{3}(i)};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i})$
(48)
$\displaystyle\overset{(b)}{=}\sum\limits_{i=1}^{k}I(\operatorname{r}^{i},\operatorname{p}^{i};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i})$
(49)
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}\sum\limits_{i=1}^{k}\left[I(\operatorname{r}^{i},\operatorname{p}^{i},\operatorname{q}_{i+1}^{k};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i})-I(\operatorname{q}_{i+1}^{k};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i},\operatorname{r}^{i},\operatorname{p}^{i})\right]$
(50)
$\displaystyle\overset{(c)}{=}\sum\limits_{i=1}^{k}\left[I(\operatorname{r}^{i},\operatorname{p}^{i},\operatorname{q}_{i+1}^{k};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i})\right]$
(51)
$\displaystyle\overset{\eqref{eq:chainrule_I}}{=}\sum\limits_{i=1}^{k}\left[I(\operatorname{r}^{i},\operatorname{p}^{i};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{k})+I(\operatorname{q}_{i+1}^{k};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i})\right]$
(52)
$\displaystyle\overset{(d)}{=}\sum\limits_{i=1}^{k}I(\operatorname{r}^{i},\operatorname{p}^{i};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{k})$
(53)
$\displaystyle\overset{(e)}{\leq}\sum\limits_{i=1}^{k}I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{k})=I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{i}|\operatorname{q}^{k})$
(54)

where equality holds in $(a)$ if and only if the Markov chain
$\operatorname{s}^{i}\leftrightarrow\operatorname{q}^{i}\leftrightarrow(\operatorname{r}^{i},\operatorname{p}^{i})$
holds for all $i\in\\{1,\ldots,k\\}$ (as a straightforward extension of Lemma
1). In our case, the latter Markov chain holds since we are assuming
$(\operatorname{q}^{k},\operatorname{s}^{k})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r}^{k},\operatorname{p}^{k})$.
In turn, $(b)$ stems from the fact that, for all $i\in\\{1,\ldots,k\\}$,
$\operatorname{x}^{i-d_{3}(i)}$ is a function of
$\operatorname{y}^{i-1},\operatorname{q}^{i},\operatorname{r}^{i},\operatorname{p}^{i}$.
To prove $(c)$, we resort to (13) and write

$\displaystyle
I(\operatorname{q}_{i+1}^{k};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i},\operatorname{r}^{i},\operatorname{p}^{i})$
$\displaystyle=I(\operatorname{q}_{i+1}^{k};\operatorname{y}^{i},\operatorname{r}^{i},\operatorname{p}^{i}|\operatorname{q}^{i})-I(\operatorname{q}_{i+1}^{k};\operatorname{y}^{i-1},\operatorname{r}^{i},\operatorname{p}^{i}|\operatorname{q}^{i})$
(55)

From the definitions of the blocks (in (11)), it can be seen that, given
$\operatorname{q}^{i}$, the triad of random sequences
$(\operatorname{y}^{i},\operatorname{r}^{i},\operatorname{p}^{i})$ is a
deterministic function of (at most)
$(\operatorname{s}^{i},\operatorname{r}^{i},\operatorname{p}^{i})$. Recalling
that $(\operatorname{q}^{k},\operatorname{s}^{k})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p}^{k})$ and
that
$\operatorname{q}_{i+1}^{k}\leftrightarrow\operatorname{q}^{i}\leftrightarrow\operatorname{s}^{i}$
(see (24)), it readily follows that
$\operatorname{q}_{i+1}^{k}\leftrightarrow\operatorname{q}^{i}\leftrightarrow(\operatorname{r}^{i},\operatorname{p}^{i},\operatorname{s}^{i})$,
and thus each of the mutual informations on the right-hand-side of (55) is
zero. To verify the validity of $(d)$, we use (13) and obtain

$\displaystyle
I(\operatorname{q}_{i+1}^{k};\operatorname{y}(i)|\operatorname{y}^{i-1},\operatorname{q}^{i})=I(\operatorname{q}_{i+1}^{k};\operatorname{y}^{i}|\operatorname{q}^{i})-I(\operatorname{q}_{i+1}^{k};\operatorname{y}^{i-1}|\operatorname{q}^{i}),$
(56)

where $(d)$ now follows since $0\leq
I(\operatorname{q}_{i+1}^{k};\operatorname{y}^{i-1}|\operatorname{q}^{i})\leq
I(\operatorname{q}_{i+1}^{k};\operatorname{y}^{i}|\operatorname{q}^{i})\leq
I(\operatorname{q}_{i+1}^{k};\operatorname{y}^{i},\operatorname{r}^{i},\operatorname{p}^{i}|\operatorname{q}^{i})$,
where the last term in this chain of inequalities was shown to be zero in the
proof of $(d)$. Equality holds in $(e)$ if and only if
$(\operatorname{r}^{k},\operatorname{p}^{k})\leftrightarrow(\operatorname{r}^{i},\operatorname{p}^{i},\operatorname{q}^{i},\operatorname{y}^{i-1})\leftrightarrow\operatorname{y}(i)$,
a Markov chain which is satisfied in our case from the fact that
$(\operatorname{q},\operatorname{s})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{r},\operatorname{p})$ and from
Lemma 1.

Finally, since
$(\operatorname{r}^{k},\operatorname{p}^{k})\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}(\operatorname{q}^{k},\operatorname{s}^{k})$,
we have that the chain of equalities from (43) to (44) holds, from which we
conclude that

$\displaystyle
I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{i}|\operatorname{q}^{k})=I(\operatorname{r}^{k},\operatorname{p}^{k}\,;\,\operatorname{u}^{k})+I(\operatorname{r}^{k},\operatorname{p}^{k};\operatorname{y}^{k}|\operatorname{u}^{k})+I(\operatorname{q}^{k};\operatorname{r}^{k}|\operatorname{u}^{k},\operatorname{y}^{k}).$
(57)

Inserting this result into (54) and invoking Theorem 1 we arrive at equality
$(\dagger)$ in (23).

To prove the first equality the (23), it suffices to notice that
$I(\operatorname{x}^{k}\to\operatorname{y}^{k}|\operatorname{q}^{k})$
corresponds to the sum on the right-hand-side of (53), from where we proceed
as with the first part. This completes the proof of the theorem. ∎

##  V Potential Applications

Information inequalities and, in particular, the data-processing inequality,
have played a fundamental role in Information Theory and its applications [20,
21, 22, 23, 24, 25, 26, 27]. It is perhaps the lack of a similar body of
results associated with the directed information (and with non-asymptotic,
causal information transmission) which has limited the extension of many
important information-theoretic ideas and insights to situations involving
feedback or causality constraints [28, 5]. Two such areas, already mentioned
in this paper, are the understanding of the fundamental limitations arising in
networked control systems over noiseless digital channels, and causal rate
distortion problems. In those contexts, causality is of paramount relevance an
thus the directed information appears, naturally, as the appropriate measure
of information flow (see, for example, [9, 5, 11, 29, 30] and [7]). We believe
that our results might help gaining insights into the fundamental trade-offs
underpinning those problems, and might also allow for the solution of open
problems such as, for instance, characterizing the minimal average data-rate
that guarantees a given performance level [10] (an improved version of the
latter paper, which extensively uses the results derived here, is currently
under preparation by the authors). On a different vein, directed mutual
information plays a role akin to that of (standard) mutual information when
characterizing channel feedback capacity (see, e.g., [3, 4] and the references
therein). Our results may also play a role in expanding the understanding of
communication problems over channels used with feedback, particularly when
including in the analysis additional exogenous signals such as a random
channel state, interference and, in general, any form of side information.
Thus, we hope that the inequalities and identities presented in Section III
may help in extending results such as dirty-paper coding [31], watermarking
[32], distributed source coding [33, 34, 25, 26], multi-terminal coding [35,
36], and data encryption [37], to scenarios involving causal feedback.

##  VI Conclusions

In this paper, we have derived fundamental relations between mutual and
directed informations in general discrete-time systems with feedback. The
first of these is an inequality between the directed information between to
signals inside the feedback loop and the mutual information involving a subset
of all the exogenous incoming signals. The latter result can be interpreted as
a law of conservation of information flows for closed-loop systems. Crucial to
establishing these bounds was the repeated use of chain rules for conditional
mutual information as well as the development of new Markov chains. The proof
techniques do not rely upon properties of entropies or distributions, and the
results hold in very general cases including non-linear, time-varying and
stochastic systems with arbitrarily distributed signals. Indeed, the only
restriction is that all blocks within the system must be causal mappings, and
that their combined delay must be at least one sample. A new generalized data
processing inequality was also proved, which is valid for nested directed
informations within the loop. A key insight to be gained from this inequality
was that the further apart the signals are in the loop, the lower is the
directed information between them. This closely resembles the behavior of
mutual information in open loop systems, where it is well known that any
independent processing of the signals can only reduce their mutual
information.

##  VII Appendix

###### Lemma 1

In the system shown in Fig. 5, the exogenous signals
$\operatorname{r},\operatorname{q}$ are mutually independent and
$\mathcal{S}_{1},\mathcal{S}_{2}$ are deterministic (possibly time-varying)
causal maps characterized by
$\operatorname{y}^{i}=\mathcal{S}_{1}(\operatorname{r}^{i},\operatorname{u}^{i})$,
$\operatorname{u}^{i}=\mathcal{S}_{2}(\operatorname{q}^{i},\operatorname{y}^{i})$,
$\forall i\in\\{1,\ldots,k\\}$, for some $k\subset\mathbb{N}$.

$\mathcal{S}_{1}$$r$$\mathcal{S}_{2}$$q$$u$$y$

Figure 5: Two arbitrary causal systems $\mathcal{S}_{1},\mathcal{S}_{2}$
interconnected in a feedback loop. The exogenous signals
$\operatorname{r},\operatorname{q}$ are mutually independent.

For this system, the following Markov chain holds

$\displaystyle\operatorname{r}^{k}\longleftrightarrow(\operatorname{u}^{k},\operatorname{y}^{k})\longleftrightarrow\operatorname{q}^{k},\;\;\;\;\forall
k\in\mathbb{K}.$ (58)

###### Proof:

Since
$\operatorname{y}^{k}=\mathcal{S}_{1}(\operatorname{r}^{k},\operatorname{u}^{k})$
and
$\operatorname{u}^{k}=\mathcal{S}_{2}(\operatorname{q}^{k},\operatorname{y}^{k})$
are deterministic functions, it follows that for every possible pair of
sequences $y^{k},u^{k}$, the sets
$\rho_{y^{k},u^{k}}\triangleq\\{r^{k}:y^{k}=\mathcal{S}_{1}(r^{k},u^{k})\\}$
and
$\phi_{y^{k},u^{k}}\triangleq\\{q^{k}:u^{k}=\mathcal{S}_{2}(q^{k},y^{k})\\}$
are also deterministic. Thus,
$(\operatorname{u}^{k},\operatorname{y}^{k})=(u^{k},y^{k})\iff\operatorname{r}^{k}\in\rho_{y^{k},u^{k}}$
and
$(\operatorname{u}^{k},\operatorname{y}^{k})=(u^{k},y^{k})\iff\operatorname{q}^{k}\in\phi_{y^{k},u^{k}}$.
This means that for every pair of Borel sets $(R,Q)$ of appropriate
dimensions,

$\displaystyle\Pr\\{\operatorname{r}^{k}\in R$
$\displaystyle,\operatorname{q}^{k}\in
Q|\operatorname{y}^{k}=y^{k},\operatorname{u}^{k}=u^{k}\\}$
$\displaystyle\overset{\hphantom{(a)}}{=}\Pr\\{\operatorname{r}^{k}\in
R,\operatorname{q}^{k}\in
Q|\operatorname{r}^{k}\in\rho_{y^{k},u^{k}}\,,\,\operatorname{q}^{k}\in\phi_{y^{k},u^{k}}\\}$
$\displaystyle\overset{\hphantom{(a)}}{=}\Pr\\{\operatorname{r}^{k}\in
R|\operatorname{r}^{k}\in\rho_{y^{k},u^{k}}\,,\,\operatorname{q}^{k}\in\phi_{y^{k},u^{k}}\\}\Pr\\{\operatorname{q}^{k}\in
Q|\operatorname{r}^{k}\in(\rho_{y^{k},u^{k}}\cap
R)\,,\,\operatorname{q}^{k}\in\phi_{y^{k},u^{k}}\\}$
$\displaystyle\overset{(a)}{=}\Pr\\{\operatorname{r}^{k}\in
R|\operatorname{r}^{k}\in\rho_{y^{k},u^{k}}\\}\Pr\\{\operatorname{q}^{k}\in
Q|\operatorname{q}^{k}\in\phi_{y^{k},u^{k}}\\}$
$\displaystyle\overset{\hphantom{(a)}}{=}\Pr\\{\operatorname{r}^{k}\in
R|\operatorname{y}^{k}=y^{k},\operatorname{u}^{k}=u^{k}\\}\Pr\\{\operatorname{q}^{k}\in
Q|\operatorname{y}^{k}=y^{k},\operatorname{u}^{k}=u^{k}\\},$

where $(a)$ follows from the fact that
$\operatorname{r}^{k}\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\operatorname{q}^{k}$. This completes the
proof. ∎

## References

  * [1] J. Massey, “Causality, feedback and directed information,” in _Proc. Intl. Symp. Inf. Theory and its Appl._ , Hawaii, USA, Nov. 1990, pp. 303–305.
  * [2] G. Kramer, “Directed information for channels with feedback.” Ph.D. dissertation, Swiss federal institute of technology, 1998.
  * [3] S. Tatikonda and S. Mitter, “The capacity of channels with feedback,” _IEEE Trans. Inf. Theory_ , vol. 55, no. 1, pp. 323–349, Jan. 2009.
  * [4] C. Li and N. Elia, “The information flow and capacity of channels with noisy feedback,” _Submitted to IEEE Trans. Inf. Theory_ , Aug. 2011. [Online]. Available: http://arxiv.org/abs/1108.2815v2
  * [5] M. S. Derpich and J. Østergaard, “Improved upper bounds to the causal quadratic rate-distortion function for Gaussian stationary sources,” _IEEE Trans. Inf. Theory_ , vol. 58, no. 5, pp. 3131–3152, May 2012.
  * [6] S. C. Tatikonda, “Control under communication constraints,” Ph.D. dissertation, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, 2000.
  * [7] N. C. Martins and M. A. Dahleh, M., “Fundamental limitations of performance in the presence of finite capacity feedback,” in _Proc. American Control Conf._ , June 2005.
  * [8] N. Martins and M. Dahleh, “Feedback control in the presence of noisy channels: “Bode-Like’ ’ fundamental limitations of performance,” _IEEE Trans. Autom. Control_ , vol. 53, no. 7, pp. 1604–1615, Aug. 2008.
  * [9] E. I. Silva, M. S. Derpich, and J. Østergaard, “A framework for control system design subject to average data-rate constraints,” _IEEE Trans. Autom. Control_ , vol. 56, no. 8, pp. 1886–1899, June 2011.
  * [10] ——, “On the minimal average data-rate that guarantees a given closed loop performance level,” in _Proc. 2nd IFAC Workshop on Distributed Estimation and Control in Networked Systems, NECSYS_ , Annecy, France, 2010, pp. 67–72.
  * [11] ——, “An achievable data-rate region subject to a stationary performance constraint for LTI plants,” _IEEE Trans. Autom. Control_ , vol. 56, no. 8, pp. 1968–1973, Aug. 2011.
  * [12] C. Quinn, T. Coleman, N. Kiyavash, and N. Hatsopoulos, “Estimating the directed information to infer causal relationships in ensemble neural spike train recordings,” _Journal of Computational Neuroscience_ , vol. 30, pp. 17–44, 2011.
  * [13] H. H. Permuter, Y.-H. Kim, and T. Weissman, “Interpretations of directed information in portfolio theory, data compression, and hypothesis testing,” _IEEE Trans. Inf. Theory_ , vol. 57, no. 6, pp. 3248–3259, June 2011.
  * [14] J. Massey and P. Massey, “Conservation of mutual and directed information,” in _Proc. IEEE Int. Symp. Information Theory_ , Sept. 2005, pp. 157–158.
  * [15] Y.-H. Kim, “A coding theorem for a class of stationary channels with feedback,” _IEEE Trans. Inf. Theory_ , vol. 54, no. 4, pp. 1488–1499, Apr. 2008.
  * [16] R. Zamir, Y. Kochman, and U. Erez, “Achieving the Gaussian rate-distortion function by prediction.” _IEEE Trans. Inf. Theory_ , vol. 54, no. 7, pp. 3354–3364, 2008.
  * [17] H. Zhang and Y.-X. Sun, “Directed information and mutual information in linear feedback tracking systems,” in _Proc. 6-th World Congress on Intelligent Control and Automation_ , June 2006, pp. 723–727.
  * [18] R. W. Yeung, _A first course in information theory_. Springer, 2002.
  * [19] G. C. Goodwin, S. Graebe, and M. E. Salgado, _Control System Design_. New Jersey: Prentice Hall, 2001\.
  * [20] J. Ziv and M. Zakai, “On functionals satisfying a data-processing theorem,” _IEEE Trans. Inf. Theory_ , vol. 19, no. 3, pp. 275–283, May 1973.
  * [21] A. Dembo, T. M. Cover, and J. A. Thomas, “Information theoretic inequalities,” _IEEE Trans. Inf. Theory_ , vol. 37, no. 6, pp. 1501–1518, Nov. 1991.
  * [22] R. Zamir, “A proof of the Fisher information inequality via a data processing argument,” _IEEE Trans. Inf. Theory_ , vol. 44, no. 3, pp. 1246–1250, May 1998.
  * [23] T. M. Cover and J. A. Thomas, _Elements of Information Theory_ , 2nd ed. Hoboken, N.J: Wiley-Interscience, 2006.
  * [24] J. Zola, M. Aluru, A. Sarje, and S. Aluru, “Parallel information-theory-based construction of genome-wide gene regulatory networks,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 21, no. 12, pp. 1721–1733, Dec. 2010.
  * [25] W. Kang and S. Ulukus, “A new data processing inequality and its applications in distributed source and channel coding,” _IEEE Trans. Inf. Theory_ , vol. 57, no. 1, pp. 56–69, Jan. 2011.
  * [26] S. Tridenski and R. Zamir, “Bounds for joint source-channel coding at high SNR,” in _Proc. IEEE Int. Symp. Information Theory_ , Aug. 2011, pp. 771–775.
  * [27] N. Merhav, “Data-processing inequalities based on a certain structured class of information measures with application to estimation theory,” _IEEE Trans. Inf. Theory_ , vol. 58, no. 8, pp. 5287–5301, Aug. 2012.
  * [28] G. N. Nair, F. Fagnani, S. Zampieri, and R. J. Evans, “Feedback control under data rate constraints: an overview,” _Proc. IEEE_ , vol. 95, no. 1, pp. 108–137, January 2007.
  * [29] S. Tatikonda, “Cooperative control under communication constraints,” in _Proc. Information Theory Workshop_ , May 2008, pp. 243–246.
  * [30] N. Elia, “When Bode meets Shannon: Control oriented feedback communication schemes,” _IEEE Transactions on Automatic Control_ , vol. 49, no. 9, pp. 1477–1488, 2004.
  * [31] M. Costa, “Writing on dirty paper (corresp.),” _Information Theory, IEEE Transactions on_ , vol. 29, no. 3, pp. 439–441, May 1983.
  * [32] A. S. Cohen and A. Lapidoth, “The Gaussian watermarking game,” _IEEE Trans. Inf. Theory_ , vol. 48, no. 6, pp. 1639–1667, June 2002.
  * [33] D. Slepian and J. Wolf, “Noiseless coding of correlated information sources,” _IEEE Trans. Inf. Theory_ , vol. 19, no. 4, pp. 471– 480, July 1973.
  * [34] A. D. Wyner and J. Ziv, “The rate-distortion function for source coding with side information at the decoder,” _IEEE Trans. Inf. Theory_ , vol. IT-22, pp. 1–10, Jan. 1976.
  * [35] Y. Oohama, “Gaussian multiterminal source coding,” _IEEE Trans. Inf. Theory_ , vol. 43, pp. 1912–1923, Nov. 1997.
  * [36] R. Zamir, S. Shamai, and U. Erez, “Nested linear/lattice codes for structured multiterminal binning,” _IEEE Trans. Inf. Theory_ , no. special A.D. Wyner issue, pp. 1250-1276, June 2002., pp. 1250–1276, June 2002.
  * [37] M. Johnson, P. Ishwar, V. Prabhakaran, D. Schonberg, and K. Ramchandran, “On compressing encrypted data,” _IEEE Trans. Signal Process._ , vol. 52, pp. 2992–3006, Oct. 2004.