# System Information Decomposition

Aobo Lyu  Department of Electrical and Systems Engineering, Washington
University in St. Louis, St. Louis, Missouri, United States of America, 63130
Swarma Research, Beijing, China, 102308  Bing Yuan  Swarma Research, Beijing,
China, 102308  Ou Deng  Graduate School of Human Sciences, Waseda University,
Tokorozawa city, Saitama, Japan, 359-1192  Mingzhe Yang  School of Systems
Science, Beijing Normal University, Beijing, China, 100875  Swarma Research,
Beijing, China, 102308  Andrew Clark  Department of Electrical and Systems
Engineering, Washington University in St. Louis, St. Louis, Missouri, United
States of America, 63130  Jiang Zhang  School of Systems Science, Beijing
Normal University, Beijing, China, 100875  Swarma Research, Beijing, China,
102308

###### Abstract

In order to characterize complex higher-order interactions among variables in
a system, we introduce a new framework for decomposing the information entropy
of variables in a system, termed System Information Decomposition (SID).
Diverging from Partial Information Decomposition (PID) correlation methods,
which quantify the interaction between a single target variable and a
collection of source variables, SID extends those approaches by equally
examining the interactions among all system variables. Specifically, we
establish the robustness of the SID framework by proving all the information
atoms are symmetric, which detaches the unique, redundant, and synergistic
information from the specific target variable, empowering them to describe the
relationship among variables. Additionally, we analyze the relationship
between SID and existing information measures and propose several properties
that SID quantitative methods should follow. Furthermore, by employing an
illustrative example, we demonstrate that SID uncovers a higher-order
interaction relationships among variables that cannot be captured by current
measures of probability and information and provide two approximate
calculation methods verified by this case. This advance in higher-order
measures enables SID to explain why Holism posits that some systems cannot be
decomposed without loss of characteristics under existing measures, and offers
a potential quantitative framework for higher-order relationships across a
broad spectrum of disciplines.

_K_ eywords Information decomposition $\cdot$ Information entropy $\cdot$
Complex systems $\cdot$ Multivariate system $\cdot$ System decomposition

##  1 Introduction

Systems Science is a multidisciplinary field investigating the relationships
and interactions among internal variables within a system, with applications
spanning neuroscience, biology, social sciences, engineering, and finance [1,
2]. Complex systems are defined by many interconnected variables that engage
in intricate interactions, the understanding of which is critical for
predicting emergent properties, devising novel treatments, and optimizing
system performance.

In the field of information theory, mutual information is a widely employed
method for quantifying interactions between two variables by encapsulating
shared information or the reduction in uncertainty facilitated by each
variable [3]. However, mutual information is restricted to describing pairwise
interactions, which often proves inadequate for analyzing complex systems that
necessitate multivariate interaction assessments.

As a solution, Beer et al. introduced the Partial Information Decomposition
(PID) method, which characterizes information interactions between a target
variable and multiple source variables by decomposing the mutual information
shared among them [4]. In the past ten years, PID and related theories, such
as Information Flow Modes [5] and integrated information theory [6], have been
applied in many fields, such as quantitative identification of Causal
Emergence [7], dynamical process analysis [8] and information disclosure [9,
10]. However, PID-related techniques only decompose the partial information of
a single target variable at a time. This leads to the fact that selecting or
constructing a suitable and plausible target variable can be challenging or
even unfeasible when addressing complex systems problems, and also raising
questions as to why certain variables are prioritized as targets over others.
Moreover, this variable-specific perspective results in a unidirectional
relationship between the specified target variable and source variable, which
makes information atoms bound to a specific target variable and insufficient
for a comprehensive description of the relationships among variables. This
further limits our exploration of system functions and properties, as many of
them originate from the relationship between system variables rather than
specific variables or its asymmetric properties.

To overcome these limitations, we need a system analysis method based on a
system perspective, analogous to the synchronization model [11] or the Ising
model [12], rather than a variable perspective like PID. Furthermore, this
method should capture the nature and characteristics of the system without
specifying or introducing any special variable, and also take into account all
the interactive relationships among all variables in the system, including
pairwise and higher-order relationships. Therefore, we propose System
Information Decomposition (SID), an innovative method that treats all system
variables equally and effectively captures their intricate interactions. This
novel approach enhances our capacity to scrutinize and understand the
complexities of multivariate systems.

Specifically, we firstly expand the PID’s conceptual framework to a system
horizon by taking all variables in the system as target variable separately.
Then, without relying on any PID quantitative method, we proving the symmetry
properties of information decomposition based on a set theory perspective of
information theory. That means the value of information atoms, the non-
overlapping units obtained by decomposing variables’ information entropy
according to their relationship, will not be affected by the the choice of
target variable. Therefore, we put forward a general SID framework, wherein
redundant, synergistic, and unique information atoms become a multivariate
system’s property, reflecting the complex (pairwise and higher-order)
relationships among variables. Furthermore, we explore the connections between
existing information entropy indicators and the information atoms within the
SID framework while proposing the necessary properties for information atom
quantification and several variable calculation approaches. Through a detailed
case analysis, we provide an intuitive demonstration that SID can unveil
higher-order relationships within the system that cannot be captured by
existing probability or information measures. Finally, we discuss the
potential application scenarios and implications of SID from the philosophical
perspective of system decomposition as well as from areas such as Higher-order
Networks and theory of Causality.

Our contributions to Information and System Science are twofold. Firstly, the
SID framework broadens the application of information decomposition methods in
complex systems by introducing a methodology to decompose all variables’
entropy within a system. This achievement also unifies information entropy and
information decomposition onto one Venn diagram, where three variables can be
well represented on a two-dimensional plane. Secondly, this framework reveals
previously unexplored higher-order relationship that cannot be represented by
existing probability or information measures, providing a potential data-
driven quantitative framework for Higher-order Networks related research.

The remainder of this paper is organized as follows. Section 2 reviews the
development of information theory, PID and related research. Section 3 extends
the PID method to multivariate system scenarios, defines SID, shows the
connections between existing information entropy indicators and the
information atom. Section 4 presents the characteristics of the SID framework
through a case analysis. Then, Section 5 gives the properties for information
atom calculation and there possible calculation approaches. The significance
and potential applications of SID are discussed in Section 6.

##  2 Information Decomposition

###  2.1 Information Theory Framework

Shannon’s classical information theory has provided a robust foundation for
understanding information entropy [3]. Mutual information and conditional
entropy further decompose information and joint entropy according to the
pairwise relationship between variables, which can be intuitively shown in
Venn diagrams 1, a precise tool for depicting the information composition
within systems. In this paper, we explore the potential of Venn diagrams to
provide valuable insights into the complex decomposition of multivariate
systems and extend the entropy decomposition methods of classical information
theory.

Figure 1: Information Theory Venn Diagram.

###  2.2 Partial Information Decomposition Framework

In classical information theory, the joint mutual information may occasionally
be larger or smaller than the sum of the mutual information between individual
variables. Consequently, traditional redundant information calculations may
yield negative values, contradicting our intuitive understanding. To address
this phenomenon, Beer et al. proposed the Partial Information Decomposition
(PID) framework [4].

The PID framework facilitates the decomposition of joint mutual information
between multiple source variables and a target variable. Specifically, for a
random target variable $Y$ and a random source variables
$X={X_{1},X_{2},\cdots,X_{n}}$, the PID framework allows for the decomposition
of the information that $X$ provides about $Y$ into information atoms, such as
redundant, synergistic and unique information. These atoms represent the
partial information contributed by various subsets of $X$, individually or
jointly, providing a more nuanced understanding of the relationships between
the target and source variables.

Considering the simplest case of a system with three variables, one can employ
a Venn diagram to elucidate their interactions [4]. The unique information
$Un(Y:X_{1})$ from $X_{1}$ signifies the information that $X_{1}$ provides to
$Y$, which is not provided by $X_{2}$ and vice versa. In other words, unique
information refers to the contribution made by a specific source variable to
the target variable that is exclusive to that variable and not shared by other
source variables. Redundant information $Red(Y:X_{1},X_{2})$ represents the
common or overlapping information that $X_{1}$ and $X_{2}$ provide to $Y$.
Synergistic information $Syn(Y:X_{1},X_{2})$ captures the combined
contribution of $X_{1}$ and $X_{2}$ to $Y$, which cannot be obtained from
either variable individually.

Figure 2: Venn Diagram of PID.

######  Definition 1 (Redundant Information).

For an arbitrary multivariate system, we can select any variable as the target
variable $Y$ and the remaining variables as the source variables
${X_{1},\cdots,X_{n}}$. The redundant information
$Red(Y:{X_{1},\cdots,X_{n}})$ denotes the common or overlapping information
provided by the source variables [4], which is contained in each source [13].

Redundant information has the following properties [4]:

######  Axiom 1 (Symmetry of source variables).

$Red(Y:{X})$ is invariant to the permutation of X. For the source variables
$X_{i}$ and $X_{j}$ from $\\{X_{1},\cdots,X_{n}\\}$,$i,j\in\\{1\cdots n\\}$ ,
there is $Red(Y:X_{i},\cdots X_{j})=Red(Y:X_{j},\cdots X_{i})$.

######  Axiom 2 (Self-redundancy).

When there is only one source variable, the redundant information is
equivalent to the mutual information between the target variable $Y$ and the
source variable $X_{i}$, i.e. $Red(Y:X_{i})=I(Y:X_{i})$.

######  Axiom 3 (Monotonicity).

The redundancy should exhibit a monotonically decreasing behavior with the
inclusion of additional inputs, i.e. $Red(Y:X_{1},\cdots,X_{n})\leq
Red(Y:X_{1},\cdots,X_{n-1})$, where $n\in N$.

Despite numerous quantitative methods for information atoms in PID, a widely
accepted method still needs to be discovered, primarily due to negative
solutions. Such inconsistencies undermine the notion of information entropy as
a non-negative measure of uncertainty. To circumvent reliance on a specific
quantitative method, we employ classical mutual information and conditional
entropy for calculating the sum of the information entropy of certain
information atoms. Although this approach does not permit the precise
calculation of individual information atoms [4, 14], it ensures that the
framework remains independent of any specific PID calculation methods.
Consequently, when a particular PID calculation method computes the value of
one information atom, the information entropy of the remaining information
atoms is determined by the following Axiom:

######  Axiom 4 (Quantitative Computation).

In a three-variable system with a target variable $Y$ and source variables
$X_{i}$ and $X_{j}$, the following relationships hold:

$Un(Y:X_{i})=I(X_{i}:Y)-Red(Y:X_{i},X_{j})$

$Syn(Y:X_{i},X_{j})=I(Y:X_{i}|X_{j})-Un(Y:X_{i})=H(Y|X_{j})-H(Y|X_{i},X_{j})-Un(Y:X_{i})$

$Syn(Y:X_{i},X_{j})+Red(Y:X_{i},X_{j})+Un(Y:X_{i})+Un(Y:X_{j})=I(Y:X_{i},X_{j})$
[4]

Although several enlightening perspectives on PID have been proposed [4, 15,
16, 17, 18, 19], there is still no perfect quantitative definition. To make
our work not rely on any specific computational method, we need to explore
information decomposition and the properties of information atoms from a more
conceptual perspective. Given the high similarity between information
decomposition, especially the concept of redundant information, and the
concept of inclusion and overlapping, set theory may allow us to explore the
properties of PID more deeply.

###  2.3 A Set-theoretic Understanding of PID

Kolchinsky’s remarkable work [13] offers an understanding based on set theory.
Given that PID is inspired by an analogy between information theory and set
theory [14], the redundant information can be understood as information sets
that the sources provide to the target. More specifically, the definition of
set intersection $\cap\\{X_{i}\\}$ in set theory means the largest set that is
contained in all of the $X_{i}$, and these set-theoretic definitions can be
mapped into information-theoretic terms by treating “sets” as random
variables, “set size” as entropy, and “set inclusion” as an ordering relation
$\sqsubset$, which indicates when one random variable is more informative than
another.

Considering a set of sources variables $X_{1},...,X_{n}$ and a target $Y$, PID
aims to decompose $I(Y:X_{i},X_{j})$ and get $Red(Y:X_{1},\cdots,X_{n})$, the
total same information provided by all sources about the target, into a set of
non-negative terms. Therefore, redundant information can be viewed as the
"intersection" of the information contributed by different sources, leading to
the following definition:

######  Definition 2 (Set Intersection of Information [13] ).

For a variable-system, the redundant information from the source variables
$X_{1},\cdots,X_{n}$ to the target variable $Y$ is the information that all
source variables can provide to the target variable, the largest mutual
information between the target variable and a non-unique variable $Q$ that has
an ordering relation $\sqsubset$ with all source variables. That is

$\displaystyle Red(Y:X_{1},\cdots,X_{n})=I_{\cap}(X_{1},\cdots,X_{n}\to Y)$
$\displaystyle:=\sideset{}{}{\sup}_{Q}\\{I(Q:Y):Q\sqsubset X_{i},\forall
i\in\\{1\cdots n\\}\\}$

The ordering relation $\sqsubset$ is an analogy to the relation contained
$\subseteq$ in set theory, which is not specified but follows some
assumptions: i) Monotonicity of mutual information, $A\sqsubset B\Rightarrow
I(A:Y)\leq I(B:Y)$. ii) Reflexivity: $A\sqsubset A$ for all variable $A$. iii)
For all sources $X_{i}$, $O\sqsubset X_{i}\sqsubset(X_{1},\cdots,X_{n})$,
where $H(O)=0$ and $(X_{1},\cdots,X_{n})$ indicates all sources considered
jointly. One example of a partial order is Q $\sqsubset$ X if and only if
$H(Q|X)=0$. More derivative properties can be found in Kolchinsky’s work [13].

##  3 System Information Decomposition

In this section, we develop a mathematical framework of SID. The objective of
this framework is to decompose the information of all variables within a
system based on their interrelationships. By addressing the limitation of PID,
which focuses solely on a single target variable, we progress towards multi-
variable information decomposition for systems.

###  3.1 Extension of PID in a System Scenario

The PID method only decomposes joint mutual information between multiple
source variables and a specific target variable, as illustrated by the
outermost circle of the Venn diagram in Figure 2. We redesign the Venn diagram
to extend this method and encompass a system-wide perspective, as demonstrated
in Figure 3. The system comprises two source variables,$X_{1}$ and $X_{2}$,
and one target variable, $Y$, represented by the three intersecting circles.

The area size within the figure signifies the information entropy of the
variables or information atoms, and the central area denotes the joint mutual
information, encompassing redundant, unique from $X_{1}$, unique from $X_{2}$,
and synergistic information. This arrangement aligns with the Venn diagram
framework of PID.

Figure 3: Venn diagram from different perspectives of PID.

To enhance the comprehensiveness of the framework, it is necessary to
elucidate the unexplored section of the updated Venn diagram 3. In addition to
the four sections of joint mutual information, the information entropy of the
target variable $Y$ contains an unaccounted-for area. According to Shannon’s
formula, this area corresponds to the joint conditional entropy of the source
variables to the target variable $H(Y|X_{1},X_{2})$, which also characterizes
the interrelationships between the target variable and the source variables.
In the SID framework, numerous joint conditional entropy exist, including one
that stands out: the joint conditional entropy originating from all variables
except the target variable. To optimize the usefulness of the SID framework,
we define this specific joint conditional entropy as the target variable’s
external information ($Ext$). The definition is grounded in the philosophical
assumption that everything is interconnected. Since joint conditional entropy
implies the uncertainty that cannot be eliminated by the internal variables of
the system, the variables capable of providing this information must exist
outside the system. To some extent, external information can emphasize the
relationship between the target variable and the entire system rather than
just a simple relationship with other variables. Therefore, we also consider
it a kind of information atom within the SID framework.

######  Definition 3 (External Information).

For a system containing variables $Y$ and $\\{X_{1},\cdots,X_{n}\\}$, the
external information $Ext(Y)$ is defined as
$Ext(Y)=H(Y|X_{1},X_{2},\cdots,X_{n})$.

Thus, we have been able to decompose the target variable’s entropy into a
finite number of non-repeated information atoms according to the relationship
between it and the other variables in the system. Furthermore, we can apply
this information decomposition method to each variable in the system to
decompose the entire information entropy of the system, which results in a
preliminary version of the SID. For the convenience of expression, we use
$Un_{i-j}$, $Syn_{ij-k}$, and $Red_{ij-k}$ to represent $Un(X_{j},X_{i})$,
$Syn(X_{k}:X_{i},X_{j})$, and $Red(X_{k}:X_{i},X_{j})$ respectively. A Venn
diagram for a three-variable system is shown in Figure 4:

Figure 4: Venn diagram of SID’s Preliminary version.

###  3.2 Properties of Information Atoms

Although the preliminary version of SID can decompose all variables in a
system, the decomposition of each variable is carried out separately, and the
description of information atoms is directional (from source variables to the
target variable). For instance, the unique information provided by $X_{1}$ to
$X_{3}$ in Fig. 4 is not directly related to the unique information provided
by $X_{3}$ to $X_{1}$. To make information atoms better reflect the
relationship among variables and unifies the Venn diagram of Shannon’s
framework2.1 and PID framework 2.2, it is necessary to further explore the
properties of information atoms within the SID framework. In this subsection,
we are going to prove the symmetry property of information atoms by
demonstrating that unique, redundant, and synergistic information atoms remain
stable when different variables are considered as target variables.

######  Theorem 1 (Symmetry of Redundant Information).

Let $X_{1},\cdots,X_{n}$ be the variables in a system. In SID, there is only
one redundant information $Red(X_{1},\cdots,X_{n})$, which implies that the
redundant information is equal irrespective of the chosen target variable.
Formally, we write
$Red(X_{1},\cdots,X_{n})=Red(X_{i}:{X_{1},\cdots,X_{n}}\setminus
X_{i}),\forall i\in\\{1\cdots n\\}$.

###### Proof.

Suppose we have a multivariate system containing a target variable $Y$ and
source variables $X_{1},\cdots,X_{n}$. For the convenience of expression, we
use $\mathcal{X}$ to represent all the source variables $X_{1},\cdots,X_{n}$.
The proof is to show that $Red(Y:\mathcal{X},Y)=Red(Y;\mathcal{X})$ and
$Red(U:\mathcal{X},Y)=Red(Y:\mathcal{X},Y)$, where $U$ is the union variable
of $Y$ and $\mathcal{X}$, such that $U=(\mathcal{X},Y)$. Then, we can
demonstrate that redundant information is equal regardless of which variable
is chosen as the target variable.

Step One, to prove $Red(Y:\mathcal{X},Y)=Red(Y;\mathcal{X}):$

By Definition 2,
$Red(Y:\mathcal{X},Y)=\sup_{Q_{j}}\\{I(Q_{j}:Y):Q_{j}\sqsubset
Y,Q_{j}\sqsubset X_{i},\forall i\in\\{1\cdots n\\}\\}$. According to the
Monotonicity property of redundant information (Axiom 3) that adding new
source variables will only impose stricter restrictions on top of existing
ones, and the Symmetry property of source variables (Axiom 1) that the order
in which restrictions are imposed will not affect the results, we can make
this optimization problem into two steps, such that:

$\sup_{Q_{j}}\\{I(Q_{j}:Y):Q_{j}\sqsubset Y,Q_{j}\sqsubset X_{i},\forall
i\in\\{1\cdots n\\}\\}$

$=\sup_{Q_{j},Q_{k}}\\{I(Q_{j}:Y):Q_{j}\sqsubset Y,Q_{j}\sqsubset
Q_{k},Q_{k}\sqsubset X_{i},\forall i\in\\{1\cdots n\\}\\}$

$=\sup_{Q_{k}}\\{\sup_{Q_{j}}\\{I(Q_{j}:Y):Q_{j}\sqsubset Y,Q_{j}\sqsubset
Q_{k}\\}:Q_{k}\sqsubset X_{i},\forall i\in\\{1\cdots n\\}\\}$

$=\sup_{Q_{k}}\\{\sup_{Q_{j}}\\{H(Q_{j}):Q_{j}\sqsubset Y,Q_{j}\sqsubset
Q_{k}\\}:Q_{k}\sqsubset X_{i},\forall i\in\\{1\cdots n\\}\\}$, since
$Q_{j}\sqsubset Y$

$=\sup_{Q_{k}}\\{I(Q_{k}:Y):Q_{k}\sqsubset X_{i},\forall i\in\\{1\cdots
n\\}\\}$, since $\sup_{Q_{j}}\\{H(Q_{j}):Q_{j}\sqsubset Y,Q_{j}\sqsubset
Q_{k}\\}=I(Q_{k}:Y)$.

Therefore, $Red(Y:\mathcal{X},Y)=Red(Y;\mathcal{X})$

Step Two, to prove $Red(U:\mathcal{X},Y)=Red(Y:\mathcal{X},Y)$: Building upon
the conclusion that $Red(Y:\mathcal{X},Y)=Red(Y:\mathcal{X})$, we can replace
the target variable with the union variable $U=(\mathcal{X},Y)$, which
combines the target variable $Y$ and the source variables $\mathcal{X}$. (The
entropy of the union variable $U$ can be expressed as
$H(U)=H(\mathcal{X},Y)$.)

Firstly, let’s employ the contradiction method by assuming that
$Red(U:\mathcal{X},Y)<Red(Y:\mathcal{X},Y)$. That means that
$\sup_{Q_{j}}\\{I(Q_{j}:U):Q_{j}\sqsubset Y,Q_{j}\sqsubset X_{i},\forall
i\in\\{1\cdots n\\}\\}<\sup_{Q_{k}}\\{I(Q_{k}:Y):Q_{k}\sqsubset
Y,Q_{k}\sqsubset X_{i},\forall i\in\\{1\cdots n\\}\\}$. Let $Q_{j}^{*}$ and
$Q_{k}^{*}$ that satisfies or infinitely approaches the above conditions
($I(Q_{j}^{*}:U)=\sup_{Q_{j}}\\{I(Q_{j}:U):Q_{j}\sqsubset Y,Q_{j}\sqsubset
X_{i},\forall i\in\\{1\cdots n\\}\\}-\varepsilon,\forall\varepsilon>0$, and
$Q_{k}^{*}$ can also be inferred similarly). Since $Y\sqsubset U$ from
$U=(\mathcal{X},Y)$, we have $I(Q_{k}^{*},Y)\leq I(Q_{k}^{*},U)$. Given that
$Q_{k}^{*}\sqsubset Y$ and $Q_{k}^{*}\sqsubset X_{i}$ (same with the
restrictions of $Q_{j}^{*}$), the mutual information $I(Q_{j}^{*},U)$ should
greater or equal to $I(Q_{k}^{*},Y)$, which lead to a contradiction.
Consequently, we can conclude that $Red(U:\mathcal{X},Y)\geq
Red(Y:\mathcal{X},Y)$.

Secondly, let’s also use the contradiction method by assuming that
$Red(U:\mathcal{X},Y)>Red(Y:\mathcal{X},Y)$. In this case,
$\sup_{Q_{j}}\\{I(Q_{j}:U):Q_{j}\sqsubset Y,Q_{j}\sqsubset X_{i},\forall
i\in\\{1\cdots n\\}\\}>\sup_{Q_{k}}\\{I(Q_{k}:Y):Q_{k}\sqsubset
Y,Q_{k}\sqsubset X_{i},\forall i\in\\{1\cdots n\\}\\}$. Let’s focus on the
$Q_{j}^{*}$ and $Q_{k}^{*}$ that satisfies or infinitely approaches the above
conditions. Since $Q_{j}^{*}\sqsubset Y$ and $Y\sqsubset U$ from
$U=(\mathcal{X},Y)(H(Y|U)=0)$, we have $I(Q_{j}^{*}:U)=I(Q_{j}^{*}:Y)$, which
lead to a contradiction ($I(Q_{j}^{*}:Y)>I(Q_{k}^{*}:Y)$ with the same
restriction on $Q_{j}^{*}$ and $Q_{k}^{*}$). Therefore, we obtain
$Red(U:\mathcal{X},Y)\leq Red(Y:\mathcal{X},Y)$.

Since we have both $Red(U:\mathcal{X},Y)\geq Red(Y:\mathcal{X},Y)$ and
$Red(U:\mathcal{X},Y)\leq Red(Y:\mathcal{X},Y)$,
$Red(U:\mathcal{X},Y)=Red(Y:\mathcal{X},Y)$ is proved.

In Summary: Since we have established that
$Red(Y:\mathcal{X},Y)=Red(Y:\mathcal{X})$, and
$Red(U:\mathcal{X},Y)=Red(Y:\mathcal{X},Y)$, we can conclude that for all
$X_{i}$ in $\\{\mathcal{X}\\}$, $Red(X_{i}:Y,\\{\mathcal{X}\\}\setminus
X_{i})=Red(Y:\\{\mathcal{X}\\})$. Therefore, Theorem 1 is proved, and we can
use $Red(X_{1},\cdots,X_{n})$ or $Red_{1\cdots n}$ denote the redundant
information within the system $\\{X_{1},\cdots,X_{n}\\}$. ∎

######  Theorem 2 (Symmetry of Unique Information).

Let $X_{1},\cdots,X_{n}$ be the variables in a system. In SID, the unique
information of any two variables relative to each other is equal, regardless
of which is chosen as the target variable. Formally, we write
$Un(X_{i}:X_{j})=Un(X_{j}:X_{i})$, $\forall i\neq j$ where
$i,j\in\\{1,\cdots,n\\}$.

###### Proof.

According to Axiom 4, unique information is a part of the information provided
by the source variable to the target variable, that is, mutual information
minus redundant information. In a three-variable system
$\\{X_{1},X_{2},X_{3}\\}$, we have
$Un(X_{i}:X_{j})+Red(X_{i}:X_{j},X_{k})=I(X_{i};X_{j})$, for all $i\neq
j\in\\{1,2,3\\}$. Since $I(X_{i}:X_{j})=I(X_{j}:X_{i})$ according to the
symmetry of Shannon’s formula [3], and
$Red(X_{i}:X_{j},X_{k})=Red(X_{j}:X_{i},X_{k})=Red(X_{i},X_{j},X_{k})$
according to Theorem 1, we have $Un(X_{i}:X_{j})=Un(X_{j}:X_{i})$. Therefore,
we can represent this information atom as $Un(X_{i},X_{j})$, or $Un_{i,j}$. ∎

######  Theorem 3 (Symmetry of Synergistic Information).

Let $X_{1},\cdots,X_{n}$ be the variables in a system. In SID, the synergistic
information of any group of variables is equal, regardless of which is chosen
as the target variable. Formally, we write
$Syn(X_{1},\cdots,X_{n})=Syn(X_{i}:\\{X_{1},\cdots,X_{n}\\}\setminus
X_{i}),\forall i\in\\{1\cdots n\\}$.

###### Proof.

According to Axiom 4, Theorem 1, Theorem 2, and the chain rule of Shannon
formula, for a three-variable system with ${X_{i},X_{j},X_{k}}$:

$\displaystyle Syn(X_{k}:X_{i},X_{j})$
$\displaystyle=H(X_{k}|X_{j})-H(X_{k}|X_{i},X_{j})-Un(X_{i},X_{k})$
$\displaystyle=(H(X_{j},X_{k})-H(X_{j}))-(H(X_{i},X_{j},X_{k})-H(X_{i},X_{j}))-Un(X_{i},X_{k})$
$\displaystyle=H(X_{j},X_{k})+H(X_{i},X_{j})-H(X_{j})-H(X_{i},X_{j},X_{k})-Un(X_{i},X_{k})$
$\displaystyle=(H(X_{i},X_{j})-H(X_{j}))-(H(X_{i},X_{j},X_{k})-H(X_{j},X_{k}))-Un(X_{i},X_{k})$
$\displaystyle=H(X_{i}|X_{j})-H(X_{i}|X_{j},X_{k})-Un(X_{i},X_{k})$
$\displaystyle=Syn(X_{i}:X_{j},X_{k})$

Therefore, we proved Theorem 3 and we can write synergistic information in the
form of $Syn(X_{1},\cdots,X_{n})$ or $Syn_{1\cdots n}$. ∎

Based on the Theorem 1 2 3 (the symmetry of information atoms), the SID
framework can be merged into the formal version in Figure 5. In the formal
version of SID, the concept of target variable is canceled, and all variables
are equally decomposed according to their relationship with other variables.
Specifically, redundant information and unique information are merged.
redundant information (atoms) in any group of variables and unique information
(atoms) between any two variables appear only shown one time in the Venn
diagram, while synergistic information (atoms) appear in each participating
variable with the same value, and each variable contains one external
information (atom). So far, we can give the formal definition of SID:

Figure 5: Venn diagram of SID’s Formal Version.

######  Definition 4 (System Information Decomposition Framework).

SID is a system decomposition framework based on information entropy, that can
divide the whole information entropy of a multivariate system into non-
overlapping information atoms according to the relationship among variables.
In this framework, redundant information represents the common or overlapping
information of all the variables; unique information represents information
that is only owned by two variables but not by others; and synergistic
information represents the information that can be known from any variable
only when the other variables are observed simultaneously.

In the SID framework, the Venn diagram unifies the Shannon’s framework2.1 and
PID framework 2.2. Considering that Venn diagrams cannot present the systems
with more than three variables on a two-dimensional plane, we only present the
simple case of three-variable system ($\\{X_{1},X_{2},X_{3}\\}$) in this
paper. For the presentation of SID in systems with more than three variables,
we will analyze it in the discussion section.

###  3.3 SID and Information Measure

In addition to Axiom 4 and Definition 3 for the relationship between SID and
mutual information, conditional entropy and joint conditional entropy, there
are still some important information measures that deserve our attention.

######  Corollary 1 (Joint Entropy Decomposition).

For any subsystem with 3 variables,
$H(X_{1},X_{2},X_{3})=Ext(X_{1})+Ext(X_{2})+Ext(X_{3})+Un(X_{1},X_{2})+Un(X_{1},X_{3})+Un(X_{2},X_{3})+2*Syn(X_{1},X_{2},X_{3})+Red(X_{1},X_{2},X_{3})$.

Based on Corollary 1, which can be easily proved by Axiom 4, we can have a
deeper understanding of information atoms, that is, any information atom can
be understood as some kind of information stored by $m$ variables, and at
least $n$ variables need to be known to obtain the information
($m>n,m,n\in\mathbb{Z}$). Specifically, the external information of the system
is owned by the variable independently, so $m=1$ and $n=1$; redundant
information is owned by all variables, so $m=number\>of\>variables$ and $n=1$;
unique information is owned by two variables, Therefore $m=2$ and $n=1$;
synergistic information is shared by all variables, so
$m=number\>of\>variables$ and $n=number\>of\>variables-1$. Therefore, the
joint entropy decomposition is the sum of each information atom multiplied by
its $m-n$ quantity. This perspective will deepen our understanding of the
essence of information atoms and facilitate our exploration of the joint
entropy decomposition of systems with more than three variables. Besides, this
phenomenon also reflects the differences between information measures and Venn
diagrams. Considering that Venn diagrams cannot fully reflect the nature of
information decomposition, alternative visualization solution will be
discussed in the discussion section.

######  Corollary 2 (Total Correlation Decomposition).

For any subsystem with 3 variables,
$TC(X_{1},X_{2},X_{3})=Un(X_{1},X_{2})+Un(X_{1},X_{3})+Un(X_{2},X_{3})+Syn(X_{1},X_{2},X_{3})+2*Red(X_{1},X_{2},X_{3})$.

######  Corollary 3 (Intersection Information Decomposition).

For any system with 3 variables, its Intersection Information
$CoI(X_{1},X_{2},X_{3})=Red(X_{1},X_{2},X_{3})-Syn(X_{1},X_{2},X_{3})$.

According to the calculation of
$CoI(X,Y,Z)=H(X_{1},X_{2},X_{3})+H(X_{1})+H(X_{2})+H(X_{3})-H(X_{1},X_{2})-H(X_{1},X_{3})-H(X_{2},X_{3})$,
$Col$ is symmetry and unique for a system, which also verifies the symmetry of
information atoms ($Syn$ and $Red$) to some extent.

##  4 Case Studies

In this section, through a series of case analyses, we elucidate the unique
properties of the SID framework and its capacity to uncover higher-order
relationships that surpass the capabilities of current information and
probability measures.

Without loss of generality, we can construct a case that includes both macro
and micro perspectives, which can not only analyze the properties of SID at
the macro level but also obtain "ground truth" through known micro properties.
First, we construct six uniformly distributed Boolean variables $a,b,c,d,e,f$,
ensuring that these variables are independent. We then create new variables by
performing XOR operations on the existing variables: let $g=c\oplus e$,
$h=d\oplus f$, $i=c\oplus f$, and $j=d\oplus e$, where $\oplus$ represents
XOR.

Next, we construct new macro variables by combining these micro variables: let
$X_{1}=(a,b,c,d)$, $X_{2}=(a,b,e,f)$, $X_{3}=(c,d,e,f)$, $X_{4}=(a,c,e,h)$,
$X_{5}=(a,b,g,h)$, $X_{6}=(a,b,i,j)$. The combination method involves simple
splicing; e.g., when $a=1$, $b=0$, $c=1$, $d=1$, $X_{1}$ is equal to $1011$.
Appendix A provides a concrete example that matches this design. As the micro-
level variables are independent of each other, this combination ensures that
the properties of the macro variables are a combination of the properties of
the micro variables.

Then, we fix $X_{1}$ and $X_{2}$ as constants and form different three-
variable systems (Cases 1-4) by adding $X_{3}$, $X_{4}$, $X_{5}$, and $X_{6}$
respectively, as shown in Table 1. After knowing the microscopic dynamics of
these cases, we can more intuitively analyze their characteristics under the
SID framework.

Case | Variables | Micro-component | Micro-relationship
---|---|---|---
1 |  $X_{1}$, $X_{2}$, $X_{3}$ |  $abcd$, $abef$, $cdef$ |  $abcdef$ are independent
2 |  $X_{1}$, $X_{2}$, $X_{4}$ |  $abcd$, $abef$, $aceh$ |  $abcdef$ are independent, $h=d\oplus f$
3 |  $X_{1}$, $X_{2}$, $X_{5}$ |  $abcd$, $abef$, $abgh$ |  $abcdef$ are independent, $g=c\oplus e$, $h=d\oplus f$
4 |  $X_{1}$, $X_{2}$, $X_{6}$ |  $abcd$, $abef$, $abij$ |  $abcdef$ are independent, $i=c\oplus f$, and $j=d\oplus e$
Table 1: Cases Construction.

It is worth noting that these four cases yield identical results under
existing probability theory and information theory measures. The system has 64
equally probable outcomes, each variable has 16 equally probable outcomes, the
total information amount in the system is 6, the pairwise mutual information
between variables is 2, and the conditional entropy is 2. Existing systems
analysis methods cannot identify the differences observed in these four
examples.

However, the four systems exhibit three distinct internal characteristics
under the SID framework. Since these examples comprise mutually independent
micro variables, we can intuitively map the micro variables to the information
atoms in each case. In Case 1, the micro variables $a,b$ provide 2-bit unique
information between $X_{1}$ and $X_{2}$ ($c,d$ correspond to $X_{1}$ and
$X_{3}$, $e,f$ correspond to $X_{2}$ and $X_{3}$). In Case 2, micro variable
$a$ provides 1-bit redundant information, while $b$, $c$, and $e$ provide
1-bit unique information between $X_{1}$ and $X_{2}$, $X_{1}$ and $X_{4}$,
$X_{2}$ and $X_{4}$ respectively. The XOR relationship between $d-f-h$
provides 1-bit synergistic information between variables. In Cases 3 and 4,
micro variables $a$ and $b$ provide 2-bit redundant information, and XOR
relationships of $c-e-g$, $d-f-h$, and $c-f-i$, $d-e-j$ provide 2-bit
synergistic information for the two cases, respectively. Figure 6 displays the
SID Venn diagrams for Cases 1–4.

(a) Case 1.

(b) Case 2.

(c) Case 3.

(d) Case 4.

Figure 6: SID Venn Diagrams for Cases 1-4. In Case 1, $a,b$ provide 2-bit
unique information between $X_{1}$ and $X_{2}$ ($c,d$ correspond to $X_{1}$
and $X_{3}$, $e,f$ correspond to $X_{2}$ and $X_{3}$). In Case 2, $a$ provides
1-bit redundant information, $b$, $c$, and $e$ provide 1-bit unique
information between $X_{1}$ and $X_{2}$, $X_{1}$ and $X_{4}$, $X_{2}$ and
$X_{4}$ respectively. The XOR relationship between $d-f-h$ provides 1-bit
synergistic information. In Cases 3 and 4, $a,b$ provide 2-bit redundant
information, XOR relationships of $c-e-g$, $d-f-h$, and $c-f-i$, $d-e-j$
provide 2-bit synergistic information for the two cases, respectively.

##  5 Calculation of SID

Although we have proposed the framework of SID and proved the symmetry of
information atoms, the problem of exact computation has not been fully
resolved. Therefore, in this section, we alternatively propose the properties
that the calculation method of the SID framework should satisfy, and accept
any method that can meet these properties. Additionally, we propose a direct
methods for some special cases and two novel methods for more general cases
and validate their accuracy and applicability through the examination of the
cases 4.

###  5.1 Properties of Calculation Methods for SID

######  Property 1 (Shannon’s formula).

The sum of certain information atoms should equal to the mutual information
and conditional information. For a three-variable system, it is Axiom 4.

The information atoms can be regarded as a finer-grained division of Shannon’s
information entropy calculation, so calculation methods such as information
entropy, mutual information, and conditional entropy can accurately calculate
the sum of some information atoms, which means that the SID’s calculation
should conform to the Shannon formula. It is worth noting that when the
specific PID calculation method calculates the value of one information atom,
the rest of the information atoms will also get the results according to Axiom
4. This means that the calculation method of SID only needs to focus on one
information atom in the system.

######  Property 2 (Computational Symmetry).

The results of SID calculation should satisfy Theorems 1, 2, and 3.

For the same system, the order of variables in the calculation method will not
affect the results. This ensures that the SID framework provides a consistent
decomposition of information entropy, regardless of the order of variables.
Specifically, for redundant information and synergistic information, changing
the order of any variable in the calculation method will not change the
result; for unique information, exchanging the positions of the two focused
variables or changing the order of the remaining variables will not change the
result.

######  Property 3 (Non-negativity of information atoms).

After applying SID, the value of any information atom is greater than or equal
to zero. This non-negativity property holds because information measures, the
degree of uncertainty are always non-negative as per the principles of
information theory.

Although the computational problem of information atoms has not been
completely solved yet, just like finding the Lyapunov function, for a specific
case, we can often use specific methods, analysis, and some intuition to get
the result. For example, a direct and rigorous method is to use properties 1
and 3.

######  Proposition 1 (Direct Method).

If certain mutual information or conditional entropy is zero, we can directly
draw the conclusion that: (1) the redundant information and the corresponding
unique information are zero if some mutual information is zero, or (2) the
synergistic information and the corresponding unique information are zero if
some conditional entropy is zero. Then, we can obtain the values of the
remaining information atoms.

For a more general scenario, we are going to give a calculation formula that
can be applied to most situations and a neural network method that can give
approximate values.

###  5.2 A Calculation Formula

Although we can calculate some cases through the Direct Method 1 or from the
perspective of case construction like previous case analysis 4, in order to
make the SID framework applicable in a wider range of scenarios, we need to
find a general solution for information atoms. After analyzing a large number
of known-result cases and combining some intuitions, we reveal the
correspondence between the values of information atoms and certain structures
on the data, which we called Synergistic Block and Unique Block. Based on this
correspondence, we propose an identification method for unique information and
synergistic information and further construct a formula for calculating
synergistic information that is applicable in most cases.

######  Definition 5 (Synergistic Block and Unique Block).

For a full probability table containing the values of all variables, if we fix
a certain value of a variable (let $X_{1}=x_{1}$), we can get the possible
values ($j$and $k$) of the remaining variables under this condition
($j\in\\{X_{2}|X_{1}=x_{1}\\}$ , $k\in\\{X_{3}|X_{1}=x_{1}\\}$). Then, mark
all these values ($j$ and $k$) of the remaining variables ($X_{2}$, $X_{3}$)
while the fixed variables take other values ($X_{2}=j|X_{1}\neq x_{1}$ ,
$X_{3}=k|X_{1}\neq x_{1}$). For all values of remaining variables where both
occur simultaneously, such that $X_{2}=j$ and $X_{3}=k$ when $X_{1}\neq
x_{1}$, we call it Synergistic Block. For all values of remaining variables
where only one occurs, we call it Unique Block, such that $X_{2}=j$ and
$X_{3}\neq k$ when $X_{1}\neq x_{1}$ for $X_{2}$, or $X_{2}\neq j$ and
$X_{3}=k$ when $X_{1}\neq x_{1}$ for $X_{3}$.

Take Table A.1 as an example, we fixed the value of $X_{1}=0000$, and marked
the values of all variables in this scenario in yellow. Then, we mark the
values where $X_{2}$ to $X_{6}$ still take the same value when $X_{1}\neq
0000$ as pink. Taking $X_{1}$, $X_{2}$ and $X_{4}$ as examples, we marked the
synergistic blocks in bold, and marked the unique blocks of $X_{2}$ and
$X_{3}$ in italics. Besides, although not as obvious as the previous two,
redundant information also has corresponding redundant blocks.

######  Proposition 2 (Information Atom Identification).

The synergistic information is greater than zero if and only if the
synergistic block exists. For a three-variable system
$\\{X_{1},X_{2},X_{3}\\}$, $Syn(X_{1},X_{2},X_{3})>0$ iff $P(X_{2}=j$,
$X_{3}=k$, $X_{1}\neq x_{1}$, $j\in\\{X_{2}|X_{1}=x_{1}\\}$,
$k\in\\{X_{3}|X_{1}=x_{1}\\})>0$. The unique information between two variables
is greater than zero if and only if fix any of them, the remaining variable
have unique block for a three-variable system. That is $Un(X_{1},X_{2})>0$ iff
$P(X_{2}\neq j$, $X_{3}=k$, $X_{1}\neq x_{1}$, $j\in\\{X_{2}|X_{1}=x_{1}\\}$,
$k\in\\{X_{3}|X_{1}=x_{1}\\})>0$.

Based on the Proposition 2, we construct a calculation formula that can
calculate synergistic information. The specific calculation method for
synergistic information for a three-variable system involving $X_{1}$,
$X_{2}$, and $X_{3}$ is as follows:

$\displaystyle Syn(X_{1},X_{2},X_{3})=(\sum P(x_{1},x_{2},x_{3})*$
$\displaystyle\log(\frac{P(X_{2}=x_{2},X_{3}=k,k\in\\{X_{3}|X_{1}=x_{1}\\})}{P(X_{2}=x_{2}|X_{1}=x_{1})}*\frac{P(X_{3}=x_{3},X_{2}=j,j\in\\{X_{2}|X_{1}=x_{1}\\})}{P(X_{3}=x_{3}|X_{1}=x_{1})}*$
$\displaystyle\frac{P(X_{1}=x_{1})}{P(X_{2}=j,X_{3}=k,j\in\\{X_{2}|X_{1}=x_{1}\\},k\in\\{X_{3}|X_{1}=x_{1}\\})}))-H(X_{1}|X_{2},X_{3})$
(1)

In the previous case 4, since the data is relatively uniform, fixing any value
of $X_{1}$ will have the same result, so we can quickly calculate the
synergistic information of the four cases by fixing $X_{1}=0000$. In these
cases, the $log$ part of the formula can be intuitively understood as $log$
(yellow + synergistic block / yellow), which is $\log(4/4)=0$ in case 1;
$\log(8/4)=1$ in case 2; $\log(16/4)=2$ in cases 3 and 4. Unique information
can also be calculated by a similar method like $log$(yellow + unique block /
yellow).

###  5.3 An Approximate Method by Neural Information Squeezer

Another possible method is to use a generalized form of neural information
squeezer (NIS, a machine learning framework by using invertible neural
networks proposed in Ref [20]) to numerically calculate the redundancy of the
system, and then to derive other information atoms.

Figure 7: A generalized form of the Neural Information Squeezer network (NIS,
see [20]) to calculate mutual information(a) and redundancy(b) of a trivariate
system ($X,Y,Z$). In (a), there are two invertible neural networks
($\psi,\phi$) which can play the roles of encoder and decoder, respectively.
The whole network accepts the input $X$ to predict $Y$, and the intermediate
variable $\hat{Y}_{X}$, which is the minimum low-dimensional representation of
$X$, can be used to calculate the mutual information between $X$ and $Y$. In
(b), two NIS networks are stacked together. The first one is just the network
in (a), and the intermediate variable $\hat{Y}_{X}$ is fed into the second NIS
network to predict $Z$. Then the intermediate variable,
$\hat{Z}_{\hat{Y}_{X}}$ which is the minimum low-dimensional representation of
$\hat{Y}_{X}$, can be used to calculate the redundancy of the system
$\\{X,Y,Z\\}$.

As shown in Figure 7(a), the NIS framework has two parts: an encoder and a
decoder. The encoder can accept any real vector variable with dimension $p$,
and it contains two operators: a bijector $\psi$ modeled by an invertible
neural network (see details in [20]) with dimension $p$ and a projector $\chi$
which can drop out the last $p-q$ dimensions from the variable $\psi_{p}(X)$
to form variable $U$. The remaining part ($\hat{Y}_{X}$) can be regarded as a
low-dimensional representation of $X$ which will be used to construct the
target $Y$ via another invertible neural network $\phi$ by mapping
$[V,\hat{Y}_{X}]$ into $\hat{Y}$, where $V\sim\mathcal{N}(0,I)$ is a
$p^{\prime}-q$ dimensional random noise with Gaussian distribution, where
$p^{\prime}$ is the dimension of $Y$. Then, we need to train the whole
framework to conform that (1) $\hat{Y}$ approximates the target variable $Y$,
and (2) $U$ follows a $p-q$ dimensional standard normal distribution. It can
be proven that the following proposition holds:

######  Proposition 3.

For any random variables $X$ with $p$ dimension and $Y$ with $p^{\prime}$
dimension, and suppose $p$ and $p^{\prime}$ are very large, then we can use
the framework of Figure 7(a) to predict $Y$ by squeezing the information
channel of $\hat{Y}_{X}$ as the minimum dimension $q^{*}$ but satisfying
$\hat{Y}\approx Y$ and $U\sim\mathcal{N}(0,I)$. Further, if we suppose
$H(X)>H(X|Y)>0$, then:

$H(\hat{Y}_{X})\approx I(X;Y),$ (2)

and

$H(U)\approx H(X|Y).$ (3)

We will provide the proof in the appendix. The reason why we require that the
numbers of dimensions of $X,Y$ are large is because the maximal $q$ for
accurate predictions may not be integer if $p,p^{\prime}$ are small.
Therefore, we can enlarge the dimensions by duplicating the vectors.

To calculate the redundancy for a system with three variables: $X,Y,Z$, we can
use the NIS network twice, as shown in Figure 7(b). The first NIS network is
to use the intermediate variable $\hat{Y}_{X}$, the dense low-dimensional
representation of $X$ with the minimum dimension $q$, to construct $Y$. Then,
the second NIS network is to use $\hat{Z}_{\hat{Y}_{X}}$, the minimal
dimensional dense low-dimensional representation of $\hat{Y}_{X}$ to construct
$Z$. After these two steps, the Shannon entropy of the intermediate variable
of $NIS_{2}$: $\hat{Z}_{\hat{Y}_{X}}$ can approach the redundancy. Thus, the
redundancy of the system can be calculated approximately in the following way:

$Red(X,Y,Z)\approx H(\hat{Z}_{\hat{Y}_{X}}).$ (4)

To verify that $Red(X,Y,Z)$ calculated in this way can be regarded as the
redundancy of the system, we need to prove that Equation 4 satisfies the
property of symmetry for all the permutations of $X,Y,Z$, i.e., the following
proposition:

######  Proposition 4.

For a system with three random variables $X,Y,Z$, without losing generality,
we suppose that the conditional information satisfy $H(X)>H(X|Y)>0$ ,
$H(X)>H(X|Z)>0$, and $H(Y)>H(Y|X)>0$, then the redundancy calculated in
Equation 4 is symmetric:

$Red(X,Y,Z)\approx Red(X,Z,Y).$ (5)

To be noticed that $Red(X,Z,Y)\approx H(\hat{Y}_{\hat{Z}_{X}})$ is different
from $Red(X,Y,Z)$ in the way that the order of the predictions from $X$ is $Z$
and then $Y$.

The proof of Theorem 4 is also provided in the appendix. With the calculation
of redundancy, we can easily calculate unique and synergistic information
atoms. Furthermore, we can extend the method to systems with more variables by
just stacking more NIS networks in the similar way as shown in Figure 7 (b).

However, there are two disadvantages to this method, one is that the
calculation is inaccurate and requires a large number of training epochs.
Second, the numbers of dimensions of all variables must be large enough such
that the independent information among the variables can be discarded by
dropping out the dimensions. Further studies are needed.

To verify that the NIS framework can calculate redundant information, we
conducted numerical experiments using Case3 as an example, as Figure8 shows,
where the mutual information between each pair of variables and the redundant
information is 2 bits.

In this experiment, variable $X_{1}$ is used as the input of NIS1 in the
framework, with $X_{2}$ predicted as the target $Y$, and the intermediate
variable $\hat{Y}_{X}$ is fed into NIS2 to predict $X_{3}$. In this
experiment, both inputs and targets were expanded to 64 dimensions by direct
replication of the original variables, and let the two intermediate variables
in the NIS maintain consistent dimensions, both denoted by $q$ . The minimum
dimension of $\hat{Y}_{X}$ and $\hat{Z}_{\hat{Y}_{X}}$ are selected by
monitoring the changes in the loss curves.

From the above results, it can be seen that when $q$, the dimension of
intermediate variable, is relatively large, the entropy of the intermediate
variable is quite accurate for mutual information or redundant information. As
the $q$ drops below a threshold, the loss signally increases, indicating that
the intermediate variable cannot capture all the mutual information and the
redundant information.

Figure 8: (a) The changes of $H(\hat{Y}_{X})$ in NIS1 under $q=4,2,1$
respectively; (b) The changes of $H(\hat{Z}_{\hat{Y}_{X}})$ in NIS2 under
$q=4,2,1$ respectively; (c) The changes of training loss in NIS1 under
$q=4,2,1$ respectively; (d) The changes of training loss in NIS2 under
$q=4,2,1$ respectively. The same experiments were conducted for the other
three cases, and the redundant information could be accurately calculated
under the NIS framework.

##  6 Discussion

###  6.1 SID and PID

As an information decomposition method compatible with PID’s conceptual
framework, SID has mainly completed two developments on the basis of it: i)
The scope of information decomposed is expanded from the mutual information of
the source variables and the target variable to the information of all
variables in the system; ii) After decomposing all information in the system,
SID show the symmetry of information atoms among different variables. Besides,
it is worth noting that SID is not based on any existing PID calculation
methods, but instead proposes a set of computational properties that should be
satisfied.

Based on these two changes, the biggest difference between SID and PID is the
analysis perspective: PID focuses on the directed pairwise (second-order)
relationship between the set of source variables and the target variable,
while SID focuses on all relationship among variables in the system, from
pairwise to undirected high-order relationship. This exhaustion of
relationships enables SID to pay attention to the relationships among the
source variables and the high-order symmetric relationships that PID ignores.
Take Case 4 as an example. From the perspective of PID, there are directional
redundant, synergistic and unique information from $X1$ and $X2$ to the target
variable, but the information interaction relationship between $X1$ and $X2$
is unknown. Also, it cannot be realized from PID that the synergistic
information provided by $X1,X2$ to the target variable is only a partial
understanding of the undirected synergistic effect among the three variables,
and this effect also occurs when $X1$ or $X2$ as target. In addition, on the
basis of being compatible with PID, SID adds more constraints, such as Theorem
1 2 3, so it provides more ways in the calculation of information
decomposition. For example, in Proposition 1, it can be inferred that the
redundant information is zero through the presence of variable pairs with zero
mutual information in the variable set, which is not satisfied in some
existing PID calculation methods.

To sum up, SID extents the analysis scope and reveals several essential
natures of information atoms on the basis of being compatible with the PID
framework, and greatly expands the application scenarios of information
decomposition, which will be discussed in the next few paragraphs.

###  6.2 SID and Higher-order Measurement

The holism-versus-reductionism debate persists in modern literature [21].
Those who hold a reductionist view believe that any system can be divided into
many subsystems, and we can fully understand the entire system by studying the
properties of the subsystems and their connections, which is also the research
philosophy followed by most disciplines [22]. But holism holds that the system
should be treated as a whole because the splitting of the system will
inevitably lose the understanding of some of its properties [23]. This
contradiction seems irreconcilable when we don’t discuss in detail how to
decompose the system.

However, the SID offers a perspective that can explain this conflict by
accounting for higher-order relationships in the system that are not captured
by previous measures. To better divide the different measures, we divide
information entropy into first-order measures, which reflect a certain
attribute of a single variable. Mutual information and conditional entropy, on
the other hand, can be divided into second-order measures, which capture some
aspects of pairwise relationships between variables [24]. Although among the
second-order measurement, information theory’s cross-entropy can measure the
information shared among multiple variables, it still captures linear
superpositions of second-order relationships, which provides limited insight
into multivariate interactions. But under the SID framework, redundant,
synergistic, and unique information can be regarded as three- or higher-order
measures, revealing a new dimension of multivariate relationships that is
entirely distinct from the first and second orders and facilitating a deeper
comprehension of higher-order system relationships. In the case analysis, the
internal structure of Case 1 aligns well with the results of the second-order
measures, and can be considered a reducible, decomposable system. Cases 2, 3,
and 4, however, have internal structures that cannot be captured by second-
order measures and are thus regarded by holism as systems that cannot be
decomposed and understood individually. To some extent, SID and the case
analysis offer an explanation that bridges the gap between holism and
reductionism; that is, some of the system properties that holism insists
cannot be understood separately might be explained by higher-order measures or
decomposition methods.

###  6.3 Potential Application

In addition to philosophical discussions, higher-order measures can also be
applied to many fields. A foreseeable application across many domains comes
from that SID deepens our understanding of data, measures, and information. In
the case studies 4, the data contains information about the construction of
the four variable systems, but the inner relationship of the system cannot be
captured by probability measures or existing information measures. That means
the incompleteness of measures may limit our ability to analyze existing
systems, even if we have obtained complete data. Therefore, conducting higher-
order information measures in the analysis of complex systems may offer
valuable insights, especially in the field where traditional information
measures fail to capture the relationship among systems. A worth exploring
direction the quantitative analysis Higher-order Networks [25]. Since SID can
provide a data-driven framework for identifying and analyzing of high-order
network structures, it may potentially impact the analysis and understanding
of complex systems across various domains [26]. For example, in studying
neural networks and brain connectivity [27], the SID framework can provide
further insights into the higher-order information flow between multiple
neurons or brain regions, which will allow us to directly generate higher-
order network models between neurons through the temporal data of multiple
neurons, and use this model to explain the implementation of specific
functions; in ecological [28], financial, or social systems, the quantitative
characterization of high-order relationships among multiple agents can assist
in the development of more accurate models and forecasts, as well as the
design of effective control methods. Also, this combination is also a two-way
promotion: Since Venn diagrams have limitation on presenting more than three
variable systems on a two-dimensional plane, hypergraphs in the field of
Higher-order Networks may be a better tool for visualizing SID frameworks.

Another field where SID may interact is Causal Science, since it is a field
for studying the intrinsic relationships between multiple variables, just like
the SID framework. One of the goals of causal science is to search for
invariance in the system. We hope that the revealed properties of the system
are independent of the distribution of the data. However, the results obtained
from SID can vary with changes in the data distribution. Therefore, adopting
the methods of causal science to reveal system invariance is one direction in
which SID can be improved. In addition, conditional independence plays an
important role in causal discovery and causal inference in multivariate
systems [29], while in the quantitative calculation of SID, conditional
independence also plays a similar role in eliminating the uncertainty of
higher-order relations, refer to the calculation method in the first way.
Therefore, studying the properties of conditional independence within the
framework of SID may provide a bridge between causal science and SID. The
benefits of this association are mutual: from the perspective of Pearl Causal
Hierarchy theory [30], SID is a research technique that utilizes observational
data, which is at the lowest rung of the causal ladder. Investigating whether
lifting the approach to higher rungs of causal ladder can yield deeper
insights into the system is an area worth exploring, for instance, by
incorporating causal graphs (DAGs) into SID methods, etc.

Apart from the above fields, SID may also has potential applications. Since
information atoms provide a more refined division of information entropy, when
the physical meaning of information atoms within the SID framework is
revealed, specific information atoms may also become indicators for some
optimization or learning problems; The symmetry property of synergistic
information in SID may provide inspiration for the information disclosure, an
important application of PID in information protection field. In summary, SID,
as a progress in the underlying measurement, may play a role in many
application scenarios, which is also the focus of our next stage of work.

###  6.4 Limitations and Future Works

In addition to the above-mentioned promising progress and expectations, there
are still several limitations worthy of attention. The first limitation is the
absence of a fully compatible quantitative method for the proposed framework,
which restricts the practical application of SID in addressing real-world
problems. As we continue to develop and refine the SID framework, it is a
priority to develop robust computation methods for calculating SID components
and consider how higher-order information measures can be integrated into
existing analytical approaches. Furthermore, the existing proofs of framework
properties and computational methods have only been established for three-
variable systems. Although extending current work to general multivariate
systems is not a formidable challenge, it contains many aspects of work, such
as how to present the decomposition results of multivariate systems on a two-
dimensional plane; how to optimize the calculation algorithm to avoid the
exponential calculation cost as the number of variables increases, which will
be considered in the next stage of research. For the above-mentioned and any
possible problems, we cordially invite other scholars who share an interest in
this field to collaborate on addressing the existing challenges of SID and
contribute to the model’s refinement.

##  7 Conclusion

In this study, we introduced the System Information Decomposition (SID)
framework, which offers novel insights for decomposing complex systems and
analyzing higher-order relationships while addressing the limitations of
existing information decomposition methods.By proving the symmetries of
information atoms and connecting them to higher-order relationships, we show
that the SID framework can provide insights and advance beyond existing
measures in understanding the internal interactions and dynamics of complex
systems.Furthermore, we explored the far-reaching implications that SID’s
unveiling of higher-order measures could have on the philosophical aspects of
systems research, higher-order networks, and causal science. Despite the fact
that current research on SID still faces challenges in terms of quantitative
calculations and multivariate analysis, we believe that continued
collaboration and exploration by the scientific community will help overcome
these obstacles.In conclusion, the SID framework signifies a promising new
direction for investigating complex systems and information decomposition. We
anticipate that the SID analysis framework will serve as a valuable tool
across an expanding array of fields in the future.

## Acknowledgments

We sincerely thank all the non-authors who played a crucial role in its
successful completion. Our heartfelt appreciation goes to the Swarma Club, an
open academic community for Complex Systems, where the Causal Emergence
reading club provided the foundation for the ideas presented in this paper. We
are also very grateful to Professor Duguid at UC Berkeley, whose course
steadied an author’s orientation towards understanding systems from an
information perspective, serving as the genesis of this paper. We are also
very grateful to the reviewers for their constructive comments, which have
improved the theoretical rigor and comprehensiveness of the paper.

## References

  * [1] Brian Castellani and Frederic William Hafferty.  Sociology and complexity science: a new field of inquiry.  Springer Science & Business Media, 2009.
  * [2] Cristoforo Sergio Bertuglia and Franco Vaio.  Nonlinearity, chaos, and complexity: the dynamics of natural and social systems.  Oxford University Press on Demand, 2005.
  * [3] Claude Elwood Shannon.  A mathematical theory of communication.  ACM SIGMOBILE mobile computing and communications review, 5(1):3–55, 2001.
  * [4] Paul L Williams and Randall D Beer.  Nonnegative decomposition of multivariate information.  arXiv preprint arXiv:1004.2515, 2010.
  * [5] Ryan G James, Blanca Daniella Mansante Ayala, Bahti Zakirov, and James P Crutchfield.  Modes of information flow.  arXiv preprint arXiv:1808.06723, 2018.
  * [6] Pedro AM Mediano, Fernando Rosas, Robin L Carhart-Harris, Anil K Seth, and Adam B Barrett.  Beyond integrated information: A taxonomy of information dynamics phenomena.  arXiv preprint arXiv:1909.02297, 2019.
  * [7] Fernando E Rosas, Pedro AM Mediano, Henrik J Jensen, Anil K Seth, Adam B Barrett, Robin L Carhart-Harris, and Daniel Bor.  Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data.  PLoS computational biology, 16(12):e1008289, 2020.
  * [8] Ryan G James, Nix Barnett, and James P Crutchfield.  Information flows? a critique of transfer entropies.  Physical review letters, 116(23):238701, 2016.
  * [9] Fernando E Rosas, Pedro AM Mediano, Borzoo Rassouli, and Adam B Barrett.  An operational information decomposition via synergistic disclosure.  Journal of Physics A: Mathematical and Theoretical, 53(48):485001, 2020.
  * [10] Borzoo Rassouli, Fernando E Rosas, and Deniz Gündüz.  Data disclosure under perfect sample privacy.  IEEE Transactions on Information Forensics and Security, 15:2012–2025, 2019.
  * [11] Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler.  The kuramoto model: A simple paradigm for synchronization phenomena.  Reviews of modern physics, 77(1):137, 2005.
  * [12] Barry A Cipra.  An introduction to the ising model.  The American Mathematical Monthly, 94(10):937–959, 1987.
  * [13] Artemy Kolchinsky.  A novel approach to the partial information decomposition.  Entropy, 24(3):403, 2022.
  * [14] Paul L Williams.  Information dynamics: Its theory and application to embodied cognitive systems.  PhD thesis, Indiana University, 2011.
  * [15] Virgil Griffith, Edwin KP Chong, Ryan G James, Christopher J Ellison, and James P Crutchfield.  Intersection information based on common randomness.  Entropy, 16(4):1985–2000, 2014.
  * [16] Robin AA Ince.  Measuring multivariate redundant information with pointwise common change in surprisal.  Entropy, 19(7):318, 2017.
  * [17] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, and Jürgen Jost.  Shared information—new insights and problems in decomposing information in complex systems.  In Proceedings of the European conference on complex systems 2012, pages 251–269. Springer, 2013.
  * [18] Malte Harder, Christoph Salge, and Daniel Polani.  Bivariate measure of redundant information.  Physical Review E, 87(1):012130, 2013.
  * [19] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay.  Quantifying unique information.  Entropy, 16(4):2161–2183, 2014.
  * [20] Jiang Zhang and Kaiwei Liu.  Neural information squeezer for causal emergence.  Entropy, 25(1):26, 2023.
  * [21] Melanie Mitchell.  Complexity: A guided tour.  Oxford university press, 2009.
  * [22] Richard Gallagher and Tim Appenzeller.  Beyond reductionism, 1999.
  * [23] Jan Christiaan Smuts.  Holism and evolution.  Macmillan, 1926.
  * [24] Thomas M Cover.  Elements of information theory.  John Wiley & Sons, 1999.
  * [25] Ginestra Bianconi.  Higher-order networks.  Cambridge University Press, 2021.
  * [26] Federico Battiston, Enrico Amico, Alain Barrat, Ginestra Bianconi, Guilherme Ferraz de Arruda, Benedetta Franceschiello, Iacopo Iacopini, Sonia Kéfi, Vito Latora, Yamir Moreno, et al.  The physics of higher-order interactions in complex systems.  Nature Physics, 17(10):1093–1098, 2021.
  * [27] Ed Bullmore and Olaf Sporns.  Complex brain networks: graph theoretical analysis of structural and functional systems.  Nature reviews neuroscience, 10(3):186–198, 2009.
  * [28] Simon A Levin.  Self-organization and the emergence of complexity in ecological systems.  Bioscience, 55(12):1075–1079, 2005.
  * [29] Judea Pearl.  Causality.  Cambridge university press, 2009.
  * [30] Judea Pearl and Dana Mackenzie.  The book of why: the new science of cause and effect.  Basic books, 2018.

##  Appendix A Appendix

###  A.1 Case Table

$X_{1}$ | $X_{2}$ | $X_{3}$ | $X_{4}$ | $X_{5}$ | $X_{6}$
---|---|---|---|---|---
$a$ | $b$ | $c$ | $d$ | $a$ | $b$ | $e$ | $f$ | $c$ | $d$ | $e$ | $f$ | $a$ | $c$ | $e$ | $h$ | $a$ | $b$ | $g$ | $h$ | $a$ | $b$ | $i$ | $j$
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0
0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1
0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1
0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1
0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1
0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0
0 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0
0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0
0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0
0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1
0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1
0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1
0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1
0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0
0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0
0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0
0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1
0 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1
0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1
0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 1
0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 0
0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0
0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0
0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 0
0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 1
0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1
0 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1
0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1
0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0
0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0
1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0
1 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0
1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 1
1 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1
1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1
1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 1
1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 0
1 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0
1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0
1 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 0
1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 1
1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1
1 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1
1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 1
1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0
1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0
1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0
1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0
1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1
1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1
1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1
1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0
1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0
1 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0
1 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0
1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1
1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1
1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 1
1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0
1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0

###  A.2 Proof of Propositions for Neural Information Squeezer Network

Here we provide mathematical proves for the two propositions of the neural
network framework to calculate mutual information and redundancy.

First, we rephrase the proposition 1 and then we give the proof here.

Proposition 1: For any random variables $X$ and $Y$, we can use the framework
of Figure 7(a) to predict $Y$ by squeezing the information channel of
$\hat{Y}_{X}$ as the minimum dimension but satisfying $\hat{Y}\approx Y$ and
$U\sim\mathcal{N}(0,I)$. And we suppose the conditional entropy $H(X|Y)>0$
holds, then:

$H(\hat{Y}_{X})\approx I(X;Y)$ (6)

###### Proof.

The whole structure of the alternative NIS network (Figure 7(a)) can be
regarded as the similar structure as in Ref [20], but the dynamics learner is
absent. However, we can understand the dynamic is a fixed identical mapping.
In this way, all the conclusions proved in [20] can be applied here. Thus, we
have:

$I(X;Y)\approx I(\hat{Y}_{X};\hat{Y}_{X})=H(\hat{Y}_{X})$ (7)

if all the neural networks are well trained. The first equation holds because
of Theorem 2 (information bottle-neck) and Theorem 3(mutual information of the
model will be closed to the data for a well trained framework) in [20], the
second holds when $q$ is minimized such that the information channel of
$\hat{Y}_{X}$ is squeezed as possible as we can and because of the property of
mutual information.

Further, because $U$ is an independent Gaussian noise, therefore:

$H(U)=H(\psi(X))-H(\hat{Y_{X}})\approx H(X)-I(X;Y)=H(X|Y)$ (8)

The approximated equation holds because $\psi$ is a bijector which can keep
the entropy unchanged, and Equation 7 holds. Therefore, we can prove
proposition 1. ∎

To calculate the redundancy for a system with three variables we can further
feed the variable of $\hat{Y}_{X}$ into another NIS network to predict $Z$,
and narrow down the information channel of the intermediate variable
$\hat{Z}_{\hat{Y}_{X}}$ to get the minimum dimension $q^{*^{\prime}}$ for
$\hat{Z}_{\hat{Y}_{X}}$, then its Shannon entropy can approach the redundancy,
and the redundancy satisfies the property of permutation symmetry for all the
variables. We can prove the following proposition:

Proposition 2: For a system with three random variables $X,Y,Z$, suppose the
conditional information $H(X|Y)>0,H(X|Z)>0$, then the redundancy calculated by
Equation 4 is symmetric, which means:

$Red(X,Y,Z)\approx Red(X,Z,Y)$ (9)

###### Proof.

If we accept the definition of Equation 4, then:

$Red(X,Y,Z)\approx
H(\hat{Z}_{\hat{Y}_{X}})=H(\hat{Y}_{X})-H(U_{\hat{Y}_{X}})=H(X)-H(X|Y)-H(\hat{Y}_{X}|Z),$
(10)

where $U_{\hat{Y}_{X}}$ is the discarded Gaussian noise to predict
$\hat{Y}_{Z}$.

In another way, we can use $X$ to predict $Z$, and the intermediate variable
$\hat{Z}_{X}$ can be used to predict $Y$, and the intermediate variable
$\hat{Y}_{\hat{Z}_{X}}$ can be used to approximate the redundancy which is
denoted as $Red(X,Z,Y)$. Therefore,

$Red(X,Z,Y)\approx H(X)-H(X|Z)-H(\hat{Z}_{X}|Y).$ (11)

Because the discarded noise variable $U_{\hat{Y}_{X}}$ in the process of
predicting $Y$ by $X$ is independent on all the variables, therefore:

$H(U_{\hat{Y}_{X}})=H(U_{\hat{Y}_{X}}|Z)=H(U_{\hat{Y}_{X}}|Y,Z)=H(X|Y,Z),$
(12)

Similarly, the discarded noise variable $U_{\hat{Z}_{\hat{Y}_{X}}}$ in the
process of predicting $Z$ by $\hat{Y}_{X}$ is also independent on all the
other variables, and $\psi(X)$ is the combination of $U_{\hat{Y}_{X}}$ and
$\hat{Y}_{X}$, thus:

$H(X|Y,Z)=H(U_{\hat{Y}_{X}}|Z)=H(X|Z)-H(\hat{Y}_{X}|Z).$ (13)

In the same way, we can obtain:

$H(X|Z,Y)=H(U_{\hat{Z}_{X}}|Y)=H(X|Y)-H(\hat{Z}_{X}|Y).$ (14)

Because $H(X|Y,Z)=H(X|Z)-H(\hat{Y}_{X}|Z)=H(X|Y,Z)=H(X|Y)-H(\hat{Z}_{X}|Y)$,
therefore:

$H(X|Z)+H(\hat{Z}_{X}|Y)=H(X|Y)+H(\hat{Y}_{X}|Z)$ (15)

and the Equation 10 and 11 lead to:

$Red(X,Y,Z)=Red(X,Z,Y).$ (16)

This equation is general for all the permutations of $X,Y$ and $Z$, thus, the
redundancy defined in the neural network NIS satisfies permutation symmetry.

∎