and thus the function $f_{\wedge}(a,b)$ is defined as,

\begin{equation}
\xi''(a\wedge b|\Gamma) = \frac{\xi''(a|\Gamma)\xi''(b|\Gamma)}{\xi''(a\vee b|\Gamma)}.
\end{equation}

This gives the sum rule in terms of a product and a corresponding product rule in terms of a ratio.  If one exponentiates the solution for the product rule from (<ref>) we find,

\begin{equation}
\xi''(a\wedge b|\Gamma) = \xi''(a|\Gamma)^{\log\xi''(b|\Gamma \wedge a)},
\end{equation}

which is certainly unappealing and not intuitive, however when $b = \neg a$, we recover $\xi''(a\wedge b|\Gamma) = \xi''_{\fals}$ as expected.  Since $\phi(\xi_{\fals}) = 0$, the value $\psi(\xi_{\fals}) = \exp[\phi(\xi_{\fals})] = 1$.  Thus in this logarithmic scaling, total disbelief is given the value $\xi''_{\fals} = 1$.  Complete certainty on the other hand corresponds to $\xi''_{\tru} = \psi(\xi_{\tru}) = \exp[\phi(\xi_{\tru})]$, which has not yet been specified.  Given that the image of $\phi$ is now restricted to the positive reals, $\mathrm{im}(\phi) = \mathbb{R}_+$, one can always regraduate the solutions so that $\phi(\xi_{\tru}) = 1$ which sends the image to,

\begin{equation}
\mathrm{im}(\phi) = [0,1].
\end{equation}

In this scale, $\mathrm{im}(\psi) = [1,e]$.

Both solution (<ref>) and (<ref>) are simply the result of the associativity constraint, which both $\wedge$ and $\vee$ satisfy.  However, once one determines the form of either $\{f_{\vee},f_{\neg}\}$ or $\{f_{\wedge},f_{\neg}\}$, the remaining function is constrained through the relationships in (<ref>) and (<ref>) as well as with the definitions of the lower and upper bounds given from (<ref>) and (<ref>).

Remarks on the results —
We have identified that the rules for manipulating degrees of belief are the rules for probabilities.  Hence, we make the identification $\xi \rightarrow P$ and $\xi_{\tru}\rightarrow P_{\tru} = 1$ so that the sum and product rules for all $a|\Gamma, b|\Gamma \in \tilde{\mathcal{A}}$ are written in the standard notation,

\begin{align}
P(a\vee b|\Gamma) &= P(a|\Gamma) + P(b|\Gamma) - P(ab|\Gamma)\label{sumrule}\\
P(ab|\Gamma) &= P(a)P(b|\Gamma a) = P(b)P(a|\Gamma b),\label{productrule}
\end{align}

where for ease of notation we have adopted that conjunction is written as a juxtaposition, $a\wedge b \rightarrow ab$.  The negation is written,

\begin{equation}
P(\neg a|\Gamma) = 1 - P(a|\Gamma).
\end{equation}

The space of probabilities is written $\Xi \rightarrow \mathbf{P}$, which is the set of maps from $\tilde{\mathcal{A}}$ to the interval $[0,1]$ which are consistent with the properties of $f_{\vee},f_{\wedge}$ and $f_{\neg}$.  Our inductive inference framework now consists of the web of beliefs $\mathbf{P}(\tilde{\mathcal{A}})$ constrained according to the rationale $\mathcal{R}$ which contains the representations of $f_{\neg}, f_{\wedge}$ and $f_{\vee}$.

Several authors have argued that the Cox derivation given above cannot apply generally to all universes of discourse.  Specifically, Halpern <cit.> claims that it fails for sets $\tilde{\mathcal{A}}$ which are finite.  His argument is that, for special cases in which the space $\Xi$ is discrete, one cannot assume that $\Xi = [\xi_{\fals},\xi_{\tru}] \subseteq \mathbb{R}$ is an interval.  Thus, the results from Cox and more importantly from Aczél do not hold.  This is certainly true since the space $\Xi$ must be continuous according to the equivalence in theorem (<ref>) to hold, however there is no requirement that discrete spaces $\tilde{\mathcal{A}}$ must be represented by discrete degrees of belief $\Xi$.  Assuming that $\Xi$ is continuous does not preclude it from representing degrees of belief for discrete spaces $\tilde{\mathcal{A}}$.  In other words, continuity in $\Xi$ does not require that $\Xi(\tilde{\mathcal{A}})$ be surjective in $\Xi$.

Another argument, that the results of Cox do not have universal applicability, comes from Colyvan <cit.>.  The author correctly points out the subtleties of calling Cox's results an extension of propositional logic, specifically the fact that some statements of interest are not propositions at all, like the situation of incomplete arguments from chapter one.  He also suggests that Cox's results cannot be general, since they rely on preservation of the law of excluded middle, which some logics abandon.  Neither of these objections are problems for us since we have already dealt with the problem of incomplete information in the definition of $\tilde{\mathcal{A}}$.  Colyvan's objection only goes to show the uselessness of the intuitionist program, since the rules of probability are already known to work in the real world.

Consistency between $f_{\vee}$, $f_{\wedge}$ and $f_{\neg}$ —
It is easy to check that the relationships between the connectives $\vee$, $\wedge$ and $\neg$ are satisfied by the regraduated solutions (<ref>) and (<ref>).

\begin{align}
P(a \vee b) &= P(\neg(\neg a \neg b))\nonumber\\
&= 1 - P(\neg a\neg b) = 1 - P(\neg a)P(\neg b|\neg a)\nonumber\\
&= 1 - [P(\neg a)P(\neg b|\neg a) + P(a)P(\neg b|a)] + P(a)P(\neg b|a)\nonumber\\
&= 1 - P(\neg b) + P(a)(1 - P(b|a))\nonumber\\
&= P(a) + P(b) - P(a)P(b|a).
\end{align}

Likewise we have,

\begin{align}
P(ab) &= P(\neg(\neg a \vee \neg b))\nonumber\\
&= 1 - P(\neg a \vee \neg b) = 1 - [P(\neg a) + P(\neg b) - P(\neg a \neg b)]\nonumber\\
&= 1 - P(\neg a) - P(\neg b) + [P(\neg a)P(\neg b|\neg a) + P(a)P(\neg b|a)] - P(a)P(\neg b|a)\nonumber\\
&= P(a)(1 - P(\neg b|a)) = P(ab).
\end{align}

From here one can form the degree of belief for any $n$-ary proposition from the fact that $\wedge$, $\vee$ and $\neg$ form a functionally complete set.

Modal degrees of belief? —
Now that we've determined a set of representations for the connectives $\Omega$, can we do the same for the modal operators?  Essentially we would like to determine if some representation of the form $f_{\square}$ and $f_{\diamondsuit}$ can make sense in our system.  The existence of such functions must be compatible with the other representations $\{f_{\Omega}\}$ and must obey the simple axioms,

\begin{align}
P(\square a) = f_{\square}(P(a)) &= f_{\neg}(f_{\diamondsuit}(f_{\neg}(P(a))))\nonumber\\
&= 1 - f_{\diamondsuit}(1 - P(a)),
\end{align}

and the dual,

\begin{align}
P(\diamondsuit a) = f_{\diamondsuit}(P(a)) &= f_{\neg}(f_{\square}(f_{\neg}(P(a))))\nonumber\\
&= 1 - f_{\square}(1 - P(a)).
\end{align}

Depending on the axioms that one includes in $\mathcal{I}_{\square,\diamondsuit}$, one could potentially constrain the functional form of $f_{\square}$ and $f_{\diamondsuit}$ further.  For example, if one adopts axioms $\mathbf{NC}$ and $\mathbf{CN}$ so that $\square (a \wedge b) \Leftrightarrow (\square a)\wedge (\square b)$, then the form of $f_{\square}$ is constrained to obey,

\begin{align}
P(\square(a \wedge b)) &= f_{\square}(P(a \wedge b))\nonumber\\
&= f_{\square}(P(a))f_{\square}(P(b|\square a))
\end{align}

We will not pursue these ideas any further in this thesis and instead suggest that it be considered for future work.

Degrees of beliefs for predicates? —
An obvious question at this point is wether we can assign a meaningful degree of belief to some predicate $A(x)$.  Certainly whenever $x$ is bound by quantification the term $A(x)$ just becomes a standard statement, e.g. something like $P(\exists xA(x))$ is well defined.  But what about the statement $P(A(x))$ where $x$ remains free?  Such a degree of belief is not well defined, and hence the notation $P(A(x))$ is not meaningful.  What we can do however, is adopt a different notation for predicates which agrees with the standard meaning in probability.

Whenever an argument is a predicate in which at least one of the variables remains free, we write a lowercase $p$, so that $p(A(x))$ refers to a probability distribution function over the predicate $A(x)$.  Often times we take a shorthand notation and simply write the distribution as $p(x)$, leaving out the fact that $x$ refers to a predicate $A(x)$.  It is often the case that this notation can obscure meaning.  When we write something like $p(x)$, we are not really writing a probability but are instead referring to a probability valued function over the variable $x$[The most common example of this situation is whenever $x$ is some subset of $\mathbb{R}$, i.e. $x \in X \subseteq \mathbb{R}$.  For example, $x$ could refer to the position of a particle in one dimension.  Then, the predicate $A(x)$ is identified as $A(x) = $“the particle is at position $x$.”  Because the universe of discourse is typically mentioned ahead of time, there is seldom any confusion when we write $p(x)$ to refer to the predicate $p(A(x))$.].

Whenever a value of $x$ is specified, so that $x = x_0$, we identify,

\begin{equation}
p(x=x_0) = P(x_0) = P(A(x_0)).
\end{equation}

When $x$ is continuous, then the function $p(x)$ is called a probability density function, indeed because it transforms as a density.  The probability associated with some continuous interval $\Delta x \subset X$ is given by the integral,

\begin{equation}
P(\Delta x) = \int_{\Delta x}dx\, p(x),
\end{equation}

where $p(x)dx$ is the probability assigned to the infinitesimal volume $dx$.  This is nothing more than a sum of all the probabilities $P(a)$ where $a \in \Delta x$.  For predicates of several variables, we write the free variables separated by a comma so that $p(A(x,y))=p(x,y)$.  Since the sum and product rule are valid for any statement, they can be written in terms of distributions,

\begin{align}
p(x,y) &= p(x)p(y|x) = p(y)p(x|y),\\
p(x\vee y) &= p(x) + p(y) - p(x,y).
\end{align}

§.§ Inductive inference rules $\mathcal{Z}$

We will briefly survey some important inference rules.  Inductive inference rules can either be direct consequences of the axioms, or exist as rules of thumb.  We will mainly focus on the former case and leave the latter to another chapter.

Bayes theorem —
One of the most important results of probability theory is that of Bayes' theorem <cit.>, which was arguably first identified by Laplace.  Consider the commutativity of $\wedge$ so that the product rule can be written in two equivalent ways,

\begin{equation}
p(a b|\Gamma) = p(a|\Gamma)p(b|\Gamma a) = p(b|\Gamma)p(a|\Gamma b).
\end{equation}

Thus, it must be the case that,

\begin{equation}
p(a|\Gamma b) = \frac{p(a|\Gamma)p(b|\Gamma  a)}{p(b|\Gamma)} \quad \mathrm{and} \quad p(b|\Gamma  a) = \frac{p(b|\Gamma)p(a|\Gamma  b)}{p(a|\Gamma)}.\label{Bayes}
\end{equation}

This is an incredibly powerful statement, for several reasons.  First, this demonstrates that the web of beliefs $\mathbf{P}(\tilde{\mathcal{A}})$ is not just constrained to be consistent with $\wedge,\vee$ and $\neg$, but that once one determines $p(a|\Gamma)$ and $p(b|\Gamma)$, the conditional degrees of belief $p(a|\Gamma b)$ and $p(b|\Gamma a)$ cannot be assigned independently.  Because of this, one is able to make inferences about statements $a|\Gamma b$ when $b|\Gamma a$ is known and vice versa.

One can write the conditional $p(a|\Gamma b)$ as a function $f_{\phi}$ of the other degrees of belief,

\begin{equation}
p(a|\Gamma b) \stackrel{\mathrm{def}}{=} f_{\phi}\left(\frac{}{}p(a|\Gamma),p(b|\Gamma),p(b|\Gamma a)\right),
\end{equation}

or equivalently in terms of the conjunction,

\begin{equation}
p(a|\Gamma b) \stackrel{\mathrm{def}}{=} f_{\phi}'\left(\frac{}{}p(ab|\Gamma),p(a|\Gamma)\right),
\end{equation}

so that either $f_{\phi}$ or $f_{\phi}'$ represent the application of Bayes theorem.

Marginalization —
Sometimes one would like to conduct a joint inference on $a|\Gamma$ and $b|\Gamma$, when not much is known about one of the statements.  Consider that we have the joint distribution $p(ab|\Gamma)$, but we have little information about $b|\Gamma$. A convenient procedure then, is to marginalize over $b|\Gamma$ by considering instead the probability $p(a[b\vee \neg b]|\Gamma)$,

\begin{align}
p(a|\Gamma) = p(a[b\vee \neg b]|\Gamma) &= p(ab|\Gamma) + p(a\neg b|\Gamma)\nonumber\\
&= \sum_b p(a|\Gamma)p(b|\Gamma a).\label{totalprob}
\end{align}

The distribution $p(a|\Gamma)$ is called the marginal and equation (<ref>) is often called the law of total probability.  When the universe of discourse $X\times Y$ is continuous, the marginal is an integral,

\begin{equation}
p(x) = \int dy\, p(x,y) = \int dy\, p(y)p(x|y).
\end{equation}

Expected values —
An extremely useful concept is that of the expected value.  For a discrete universe of discourse $A$, it is simply the weighted average over all $a \in A$,

\begin{equation}
\langle a\rangle \stackrel{\mathrm{def}}{=} \sum_{a\in A} a P(a).
\end{equation}

The expected value is sometimes written $\mathbb{E}[a] = \langle a \rangle \stackrel{\mathrm{def}}{=} \mu_{a}$, which is also called the mean.  Typically when we use the words expected value we are referring to the weighted average over each element of the subject matter, however the words expected value can apply more generally for any power of $a$,

\begin{equation}
\langle a^n\rangle = \sum_{a\in X}a^nP(a).
\end{equation}

More general still, one can define the expected value for any function of $a$,

\begin{equation}
\langle f(a) \rangle = \sum_{a\in A} f(a)P(a).
\end{equation}

For continuous universes of discourse $X$ the sum gets replaced by an integral,

\begin{equation}
\langle f(x) \rangle = \int_{X} dx\, f(x)p(x).
\end{equation}

Bayes' rule —
Everything we have discussed so far represents a calculus for manipulating beliefs when no new information is present.  Now we ask the question of how to update our beliefs once new information becomes available in the form of data.  We can formulate this problem in terms of deciding how one should assign a new web of beliefs $\mathbf{P}(\tilde{\mathcal{A}})$ given some new information about $\tilde{\mathcal{A}}$.  We will represent a generic change in the web of beliefs by some function $\star:\mathbf{Q}\rightarrow \mathbf{P}$ so that $\mathbf{Q}(\tilde{\mathcal{A}})\xrightarrow{\star}\mathbf{P}(\tilde{\mathcal{A}})$.

Consider that we have some prior information about the relationship between two statements $a|\Gamma$ and $b|\Gamma$, whose joint distribution is given by,

\begin{equation}
q(ab|\Gamma) = q(a|\Gamma)q(b|\Gamma a) \in \mathbf{Q},
\end{equation}

which we call the prior.  Given new information about $b|\Gamma$, we would like to determine a suitable posterior $p(a|\Gamma) \in \mathbf{P}$.  It is important to emphasize at this point that the universe of discourse does not consist of just the statements $a|\Gamma$ or $b|\Gamma$, but of both.  This will be important later when we construct a generalization to Bayes' rule.

Consider now that we have collected new information about $b|\Gamma$, such that the value of $b|\Gamma$ is known, i.e. $b|\Gamma \xrightarrow{\star}b'|\Gamma$ so that the posterior web of beliefs is constrained to satisfy,

\begin{equation}
q(b|\Gamma) \xrightarrow{\star} p(b|\Gamma) = \sum_{a}p(ab|\Gamma) = \delta_{bb'}.
\end{equation}

This is not enough to constrain the form of $p(a|\Gamma) \in \mathbf{P}(\tilde{\mathcal{A}})$, since many posteriors will satisfy this rule and be internally consistent with respect to the rationale.  In order to constrain the form of $p(a|\Gamma)$, one needs to invoke some new principle <cit.> inspired by parsimony,
[Principle of Minimal Updating]
The web of beliefs $\mathbf{Q}(\tilde{\mathcal{A}})$ should only be revised to the extent required by the new information.

Like we have said, such a constraint is not necessary for an updating scheme to be consistent, but is merely imposed to satisfy some ethical desire for how we should conduct inference <cit.>.  In the case of the joint posterior distribution,

\begin{equation}
p(ab|\Gamma) = p(a|\Gamma)p(b|\Gamma a) = p(a|\Gamma b)\delta_{bb'},
\end{equation}

the conditional $p(a|\Gamma b)$ is at this point still arbitrary.  If we invoke the principle of minimal updating (PMU), then nothing requires us to change the distribution $p(a|\Gamma b)$ since the change is already invoked in $\delta_{bb'}$.  Thus we identify,

\begin{equation}
p(a|\Gamma b) = q(a|\Gamma b),
\end{equation}

so that the web of posterior beliefs $\mathbf{P}(\tilde{\mathcal{A}})$ is given by,

\begin{equation}
p(ab'|\Gamma) = q(a|\Gamma b')\delta_{bb'}
\end{equation}

The posterior distribution $p(a|\Gamma)$ is then given by marginalizing over $b|\Gamma$,

\begin{equation}
p(a|\Gamma) = \sum_{b}q(a|\Gamma b')\delta_{bb'} = q(a|\Gamma b').
\end{equation}

This identifies the updated posterior distribution $p(a|\Gamma)$ as the prior conditional distribution $q(a|\Gamma b')$, which is called Bayes' rule and is often written as,

\begin{equation}
q(a|\Gamma) \xrightarrow{\star} p(a|\Gamma) = q(a|\Gamma)\frac{q(b'|\Gamma a)}{q(b'|\Gamma)}.\label{bayesrule}
\end{equation}

Typically the conditional distribution $q(b'|\Gamma a)$ is called the likelihood and the factor,

\begin{equation}
q(b'|\Gamma) = \sum_{a}q(ab'|\Gamma),
\end{equation}

is called the evidence.  Some may point out that (<ref>) is simply the same thing as Bayes' theorem in (<ref>), however the theorem refers only to a consequence of the product rule in $\mathbf{Q}(\tilde{\mathcal{A}})$.  The action of assigning to the degree of belief $p(a|\Gamma)$ the conditional $q(a|\Gamma b)$ is something entirely different since we changing the web $\mathbf{Q}(\tilde{\mathcal{A}}) \xrightarrow{\star}\mathbf{P}(\tilde{\mathcal{A}})$.

§.§ Other approaches

There exist a plethora of other approaches for constructing the rules of probability in the literature.  We have mentioned several already that have been inspired by Cox, such as Caticha <cit.>, Fine <cit.>, Jaynes <cit.>, Knuth <cit.>, Paris <cit.>, Skilling <cit.>, Smith and Erickson <cit.>, Tribus <cit.>, Van Horn <cit.> and Vanslette <cit.>.  We will briefly discuss some of the other popular approaches, such as the subjective and frequentist approaches and their similarities, as well as their differences, with the approach in this thesis.

De Finetti —
My thesis, paradoxically, and a little provocatively, but nonetheless genuinely, is simply this: PROBABILITY DOES NOT EXIST. The abandonment of superstitious beliefs about the existence of Phlogiston, the Cosmic Ether, Absolute Space and Time,..., or Fairies and Witches, was an essential step along the road to scientific thinking. Probability, too, if regarded as something endowed with some kind of objective existence, is no less a misleading misconception, an illusory attempt to exteriorize or materialize our true probabilistic beliefs.Bruno de Finetti, 1970
Bruno de Finetti championed the idea that probabilities are inherently subjective and do not exist in any objective sense.  This idea is the overarching theme of what is called the subjective approach to probability theory[Perhaps the main difficulty with de Finetti's approach is the confusion over the dichotomies  subjective vs. objective and ontic vs. epistemic.  The problem is that often times when one says “subjective” (personal) what they really mean is “epistemic” (having to do with knowledge).  While we take the view that probabilities are inherently epistemic, they can still be either subjective or objective depending on the subject matter.  For example, as we will see in Chapters six, seven and eight, in quantum mechanics probabilities play a central role and are entirely epistemic quantities, however they are not subjective.  The probabilities which appear in QM are objective in the sense that they are constrained to agree with the results of experiment.].  De Finetti's idea was that probabilities are assignments of personal belief which represent how one would bet on a particular outcome of some event given their current information.  In this way, probabilities are intimately tied to actions, which is quite different from what we have developed in this chapter.  In our treatment, the probability calculus determines what one ought to believe, but does not suggest any method for determining what one ought to do with those beliefs.  Those ideas belong to a different subject altogether, such as decision theory or game theory.

De Finetti's contemporaries, Ramsey <cit.> and Savage <cit.>, developed a similar approach in which the specific scheme of betting with money was replaced with a more general concept of utility.  This idea later culminated in utility theories developed by von Neumann and Morgenstern <cit.>.  In this scheme, one is required to specify a utility function $u:X\rightarrow\mathbb{R}$, where $X$ is the subject matter.  Depending on the treatment, the utility function is constrained to obey a set of axioms <cit.>.  The central quantity of interest is the expected utility,

\begin{equation}
\langle u(x) \rangle = \sum_{x\in X}u(x)p(x).\label{exputil}
\end{equation}

As was shown by von Neumann and Morgenstern <cit.>, by assuming that the function $u(x)$ obeys certain properties, such as transitivity and continuity, then a rational agent will make decisions which maximize (<ref>).

The problem with utilitarian approaches is that they single out universes of discourse which only apply to the behavior of the rational agent.  Utilitarianism is certainly useful for the stock market and even for the design of physical experiments, but it is not useful for physics as a whole.  Even so, the idea that probabilities are inherently subjective was an important step for bringing inference into the 20th century.  One use of the de Finetti approach in physics is quantum Bayesianism, or QBism <cit.>.

Kolmogorov —
Perhaps the most popular approach to probability theory is the one developed by Kolmogorov <cit.> in which probabilities are measures associated to some set $X$.  The formalism was developed as a means for assigning probabilities, or measures, to volumes in spaces such as $\mathbb{R}^n$.  A measure is a function $\mu$ from a $\sigma$-algebra over $X$ to the extended real numbers $\mathbb{R}\cup[-\infty,\infty]$.  A $\sigma$-algebra for $X$ is a subset of the power set $\Sigma \subseteq \mathscr{P}(X)$ which has the following properties,
Let $X$ be a set.  A collection of subsets $\Sigma \subseteq \mathscr{P}(X)$ is called a $\sigma$-algebra if,

* The entire set $X$ and the empty set $\emptyset$ are in $\Sigma$,

\begin{equation}
X,\emptyset \in \Sigma.
\end{equation}

* $\Sigma$ is closed under complementation,

\begin{equation}
\forall A \in \Sigma : X\backslash A \in \Sigma.\label{sigma2}
\end{equation}

* $\Sigma$ is closed under countable unions.

\begin{equation}
\forall n \in \mathbb{N} : \forall A_1,\dots,A_n \in \Sigma : \bigcup_{k=1}^nA_k \in \Sigma.\label{sigma3}
\end{equation}

Due to De Morgans laws axioms (<ref>) and (<ref>) are equivalent to,

\begin{equation}
\forall n \in \mathbb{N} : \forall A_1,\dots,A_n \in \Sigma : \bigcap_{k=1}^nA_k \in \Sigma.
\end{equation}

These axioms demonstrate that a $\sigma$-algebra is essentially an algebra of sets with the additional requirement that countable unions be in $\Sigma$.  In special cases, $\sigma$-algebras $(\Sigma,\cup,\cap,\backslash)$ are isomorphic to the Boolean algebra $(\mathcal{A},\vee,\wedge,\neg)$ whenever $\mathcal{A}$ is finite[There are some subtleties with the differences between $\sigma$-algebras and Boolean algebras.  Due to Stone's representation theorem, Boolean algebras are isomorphic to some field of sets, which may only require closure under finite unions, i.e. given a finite sequence of subsets $A_i,\dots,A_n$ of the algebra $X$, we have $\forall n < \infty : \bigcup_{i=1}^n A_i \in X$.  Any $\sigma$-algebra however, is completed with respect to the union, so that it is closed under a countable number of unions (axiom (<ref>)).  The reason for axiom (<ref>) actually comes from a different direction that is related to the Banach-Tarski paradox <cit.>.  In order to assign a meaningful volume to $n$-dimensional spaces, Banach and Tarski showed that one of four assumptions must be true, either

* The volume of a set could change under rotation.

* One must abandon the axiom of choice.

* The volume of the union of two disjoint sets is not necessarily equal to the sum of their volumes.

* There are some sets which are non-measureable.

Obviously solutions (1) and (3) are undesirable.  The opted for solution has been number (4), for which we need to develop the notion of a measurable set.  Axiom (<ref>) is explicitly included to avoid the problems of the Banach-Tarski paradox.].  If the Boolean algebra $(\mathcal{A},\wedge,\vee,\neg)$ is complete and the set $\mathcal{A}$ is countable, then the properties of $(\mathcal{A},\wedge,\vee,\neg)$ are stronger than the axioms in [sigmaalgebra]1.1.  In this sense, a $\sigma$-algebra is a sort of in-between for these two situations.

A space $X$ together with a sigma algebra $\Sigma$ is called a measurable space $\langle X,\Sigma\rangle$.  A measure $\mu$ on $\Sigma$ is then defined as,

Let $\Sigma$ be a $\sigma$-algebra over $X$.  A function $\mu:\Sigma\rightarrow\mathbb{R}\cup[-\infty,\infty]$ is called a measure if,

* The function $\mu$ is non-negative,

\begin{equation}
\forall A \in \Sigma : \mu(A) \geq 0.
\end{equation}

* The empty set has zero measure,

\begin{equation}
\mu(\emptyset) = 0.
\end{equation}

* For all countable collections $\{A_k\}_{k=1}^{n}$, the function $\mu$ is countably additive over disjoint subsets,

\begin{equation}
\forall n \in \mathbb{N} : \forall \{A_k\}_{k=1}^{n} : \forall k \neq \ell A_k\cap A_{\ell} = \emptyset : \mu\left(\bigcup_{k=1}^{n}A_k\right) = \sum_{k=1}^n\mu(A_k).
\end{equation}

Finally, whenever $\mu(X) = 1$ we say that $\mu$ is a probability measure over $\Sigma$.  While the axioms for $\sigma$-algebras and measures are reasonable, they have the problem of lacking an interpretation.  The set $X$ could be anything, such as an interval $[a,b] \subset \mathbb{R}$.  By assigning a probability measure to subsets of $[c,d] \subset [a,b]$, what exactly is the meaning of $\mu([c,d])$?  What is $\mu([c,d])$ the probability of?  By lacking an interpretation, this approach can lead to confusion.

There is also the additional problem of not having a definition of a conditional probability a priori.  It has to be put in by hand after the fact by defining,

\begin{equation}
\mu(a|b) \stackrel{\mathrm{def}}{=}\frac{\mu(ab)}{\mu(b)}.
\end{equation}

What's worse about this situation is that the Kolmogorov approach is ill-equipped to handle an updating procedure.  Because there is no interpretation as to what the measure $\mu(a|b)$ is supposed to represent, the procedure in [bayesrules]Section 1.6 is not well defined.

In our approach we fix the problem of the interpretation from the beginning.  The subject matter is not some generic set $X$, but the set of statements $\tilde{\mathcal{A}}$.  Some may argue that this is less general than Kolmogorov because we do not allow the subject matter to be arbitrary sets $X$, but so be it.  The problem with Kolmogorov is that it is too general and completely lacks an interpretation, which has the potential to allow fallacious reasoning.

Frequentism —
In the long run we shall all be dead.Keynes, 1923
There are many “interpretations” of probability theory which claim that probabilities are frequencies, or expectations of chance given some large ensemble of repeated events.  The appeal of this interpretation is that one can determine frequencies of random events by simply counting a number of positive occurrences in the ensemble.  Such an assignment of probability is independent of any agent, be they ideally rational or not, and hence one feels that they have constructed some objective notion of probability.

There are several philosophical and practical objections to this idea.  The biggest problem is that frequentism relies on there being a large number of repeated trials, or that the situation in question be repeatable in principle.  Thus, inferences about which there are no such ensembles cannot be defined in the frequentist interpretation.  A question such as, “what is the probability that there is life on Mars?” is not even a legitimate question, since there is only one Mars and not an ensemble.  To combat these issues a frequentist may simply agree, that such a question is nonsensical, and that probabilities are only defined in the presence of repeatable events.  Such an admission greatly reduces the power of any inference framework, which is undesirable.  On the contrary, they may attempt to justify such questions on the basis of the possible worlds semantics so that one constructs a virtual ensemble, but this destroys any notion of objectivity that one may have gained from the interpretation in the first place.

The frequentist interpretation also tends to be rather vague, despite its claim of being objective.  One source of vagueness is the meaning of the word random.  The popular choice for defining random variables is to define a probability space $P$, which is a triple $P = \{\Omega, \mathcal{F}, \mu\}$, where $\Omega$ is a set of mutually exclusive and exhaustive outcomes and $\mathcal{F} \subseteq \mathscr{P}(\Omega)$ is a collection of subsets of $\Omega$ called the event space.  The event space $\mathcal{F}$ is a [sigmaalgebra]$\sigma$-algebra generated by $\Omega$ and hence is equivalent to a Boolean algebra in special cases.  One then defines a random variable $X:(\Omega,\Sigma)\rightarrow (E,T)$ as a measureable function[A measureable function $f:X\rightarrow Y$ between two measurable spaces $(X,\Sigma)$ and $(Y,T)$ preserves the structure of the underlying $\sigma$-algebras, i.e. the preimage of any $B\in T$ is in $\Sigma$.] from the measurable space $(\Omega,\Sigma)$ to some other measureable space $(E,T)$.  Then, the measure associated to whether the function $X$ assigns values to some subset $S \subseteq E$ in the target is defined by,

\begin{equation}
\mu\left(X \in S\right) \stackrel{\mathrm{def}}{=} \mu\left(\left\{\omega \in \Omega\,\middle|\, X(\omega) \in S\right\}\right).
\end{equation}

In this sense, $X$ is just a function which groups the elements of $\Omega$ into particular subsets.  An analogous construction in $(\tilde{\mathcal{A}},\wedge,\vee,\neg)$ would be a homomorphism $\mathcal{X}:\tilde{\mathcal{A}}\rightarrow\tilde{\mathcal{B}}$ to another Boolean algebra $(\tilde{\mathcal{B}},\wedge,\vee,\neg)$ such that for every element $b \in \tilde{\mathcal{B}}$, the preimage $\mathcal{X}^{-1}(b) \in \tilde{\mathcal{A}}$.  While all of this is perfectly well defined, it has already obscured the subject matter enough to make the effort seem not worthwhile.

One of the biggest proponents of the frequentist view was Fisher <cit.>, who originally started out as a Bayesian but was unable to deal with the problem of the assignment of the prior <cit.>.  Fisher then became obsessed with trying to remove all subjectivity from inductive inference, which is practically impossible.  No matter what one does, there will always be some subjective component left over in any inference where one has incomplete information.  Savage refers to Fisher's attempt as “a bold attempt to make the Bayesian omelet without breaking the Bayesian egg” <cit.>.

Quantum probabilities? —
There have been attempts at arguing that classical inductive inference is incomplete, since quantum theory seems to demonstrate a violation of the sum and product rules.  As was shown in <cit.>, these arguments are simply a misunderstanding of the problem.  We will demonstrate the results of the argument here.

The prototypical example for why classical probability theory is incompatible with quantum mechanics will make use of the superposition principle and the role of interference effects.  Consider the gedanken two slit experiment in which a particle $q$ is prepared at a source $s$ with some velocity in the direction of two slits $a$ and $b$.  The particle then has some probability of going through either slit $a$ or slit $b$ and then landing somewhere on the screen on the other side at location $x$,

A two slit experiment where a particle is prepared at source $s$ and then passes through either slit $a$ or slit $b$ and arrives at some location on the screen.

In quantum mechanics, the states that describe the particle going through slit $a$ and $b$ are represented by the wave functions, $\psi_a$ and $\psi_b$ respectively.  The state that describes the particle arriving at $x$ is then given by the superposition $\psi_{ab} \propto \psi_a + \psi_b$, where the probability of detection at $x$ is,

\begin{equation}
p_{ab}(x) = |\psi_{ab}|^2 \propto |\psi_a + \psi_b|^2 = |\psi_a|^2 + |\psi_b|^2 + 2\mathrm{Re}[\psi_a^*\psi_b].
\end{equation}

The term $|\psi_a|^2$ corresponds to the probability of detection at $x$ when the particle passes through slit $a$ and when slit $b$ is closed.  Likewise $|\psi_b|^2$ is the probability that the particle passes through slit $b$ when slit $a$ is closed.  Since the particle must pass through either slit $a$ or slit $b$, it looks like quantum mechanics violates the sum rule since for these mutually exclusive events we find,

\begin{equation}
p_{ab}(x) \neq |\psi_a|^2 + |\psi_b|^2.\label{wrongq}
\end{equation}

The problem, which was originally suggested by Feynman <cit.>, is actually more subtle.  A counter argument was given by Koopman <cit.>, as well as in Ballentine <cit.> and Caticha <cit.>.  Following Caticha, consider first that we enumerate the various statements in the problem as follows,

* $s = $“particle $q$ is generated at the source $s$.

* $a = $“slit $a$ is open” and $\neg a = $“slit $a$ is closed.”

* $b = $“slit $b$ is open” and $\neg b = $“slit $b$ is closed.”

* $\alpha = $“particle $q$ goes through slit $a$.”

* $\beta = $“particle $q$ goes through slit $b$.”

* $x = $“particle $q$ is detected at $x$.”

The statement of interest is then, $x | s a b$, or,
“The particle $q$ is detected at $x$, given that it was generated at source $s$ and slit $a$ and slit $b$ are both open.”

A conjunction of interest will be $[\alpha \vee \beta] x|s a  b$, whose probability can be determined as,

\begin{equation}
p([\alpha \vee \beta] x|s a  b) = p(\alpha  x|s a  b) + p(\beta  x|s a  b) - p(\alpha  \beta  x|s a  b).
\end{equation}

Since the particle cannot go through both slits at the same time, the conjunction vanishes,

\begin{equation}
p(\alpha\beta x|s a  b) = 0.
\end{equation}

Thus we have,

\begin{equation}
p([\alpha \vee \beta] x|s a  b) = p(\alpha  x|s a  b) + p(\beta  x|s a  b).
\end{equation}

Likewise, we can also write $p([\alpha\vee\beta] x|s a  b)$ using the product rule,

\begin{equation}
p([\alpha\vee\beta] x|s a  b) = p(x|s a  b)p([\alpha \vee \beta]|s  a  b  x).
\end{equation}

However, since the particle must go through either slit $a$ or $b$ if it was detected at $x$, then the second term $p([\alpha \vee \beta]|s  a  b  x) = $.  Thus we have,

\begin{equation}
p(x|s a  b) = p(\alpha  x|s a  b) + p(\beta  x|s a  b).\label{rightq}
\end{equation}

Finally, the corresponding equation for (<ref>) is,

\begin{equation}
p(x|s a  b) \neq p(\alpha x|s a  \neg b) + p(\beta x|s \neg a  b).
\end{equation}

But, as we have shown in (<ref>), this discrepancy is perfectly consistent with the results of probability theory, hence there is no contradiction.  However, one may be tempted to use the product rule in (<ref>) so that we have,

\begin{equation}
p(x|s a  b) = p(\alpha|s a  b)p(x|s a  b  \alpha) + p(\beta|s a  b)p(s a  b  \beta).\label{rightq2}
\end{equation}

Our classical intuition tells us that, if the particle passes through slit $a$, then the fact that slit $b$ is open makes no difference and hence one can make the substitution,

\begin{equation}
p(x|s a  b  \alpha) \rightarrow p(x|s a  \neg b  \alpha),
\end{equation}

and likewise for the second term in (<ref>).  Thus, one would be led to the inference,

\begin{align}
p(x|s a  b) &= p(\alpha|s a  \neg b)p(x|s a  \neg b  \alpha) + p(\beta|s \neg a  b)p(s \neg a  b  \beta)\nonumber\\
&= p(\alpha x|s a  \neg b) + p(\beta x|s \neg a  b),\label{wrong2}
\end{align}

which is incorrect.  This demonstrates that our classical intuition about whether distant objects can influence quantum systems is wrong, quantum mechanics can be non-local.

§.§ Summary

In this chapter we constructed a theory of inference by designing the rules for probabilities as degrees of rational belief.  By constraining representations ([section321]Section 3.2.1) of elements of the Boolean algebra over extended proposition space ([section26]Section 2.6) to adhere to certain criteria, we arrived at the sum and product rules for probabilities.  We introduced the principle of minimal updating ([section36]Section 3.6) in order to derive Bayes' rule.  In the next chapter we will generalize Bayes' rule to situations in which one wishes to update their beliefs in the presence of constraints.

§ ENTROPIC INFERENCE

The theory of probability, if it is to be useful at all, demands a theory for updating probabilities.Ariel Caticha, 2020
At the end of the previous chapter, we had constructed an inductive inference for dealing with states of partial knowledge and a calculus for manipulating them in a way consistent with some rationale, $\mathcal{R}$.  This answered the first of the two questions posed in the Introduction, the second of which – how does one update their beliefs in the presence of new information – we will address in this chapter.  As the quotation above suggets, any inference framework would be remiss if it didn't also contain a method for updating ones beliefs whenever new information becomes available.

The rationale for inductive inference was developed to express several desirable properties including Universal Applicability, Consistency, Practicality and Parsimony.  We proceed in much the same way by invoking a similar rationale, $\mathcal{R}_{S}$, for the purposes of updating.  We have already seen some instances of this with the Principle of Minimal Updating which was imposed in the derivation of Bayes' rule at the end of chapter two.  As systematic as Bayes' rule is, it only provides a way for updating beliefs when new information comes in the form of data.  This raises the central question of this chapter: How should we update our beliefs when new information does not come in the form of data?  Before we can address that question, we must first clarify what is meant after all by information.

The tool for updating that we will construct is commonly referred to as the Maximum Entropy method (ME) <cit.> which has its roots in MaxEnt <cit.>.  Jaynes developed MaxEnt for the purposes of assigning priors from arbitrary constraints <cit.>.  While this helps to identify constraints as being a type of information different from data, MaxEnt is ultimately not a tool for updating, hence it cannot take into account arbitrary priors <cit.>.  On the other hand, Bayes' rule allows for arbitrary priors but does not allow for arbitrary constraints.  The goal of constructing the ME method will be to unify both Bayes' rule and MaxEnt into a complete method for inductive inference – which we call entropic inference – so that one can update arbitrary priors on the basis of arbitrary constraints.  This is done by generalizing the PMU.

In most circles, the words entropy and information are invoked in the context of Information Theory <cit.>.  Originally developed by Shannon <cit.>, Information Theory is mainly concerned with sending bits of information across communication channels and analysing the degrading effects of noise.  Through the use of various axioms, Shannon constructed his entropy as a means of quantifying an amount of information that is missing from a probability distribution.  While certainly useful in the context of communications theory, the Shannon entropy – which was later adopted in the context of MaxEnt – can only refer to discrete variables; for continuous variables it is undefined.  One can get around this problem by introducing an invariant measure into the definition so that the entropy takes the form of the Kullback-Leibler divergence[The KL divergence was actually first written down by Gibbs <cit.>.] (KL).  Despite this, the Shannon entropy was constructed using axioms which have nothing to do with updating beliefs.  As pointed out by Caticha <cit.>, this presents a serious problem since one could have introduced different axioms which would lead to different entropies (such as the Renyi <cit.> or Tsallis <cit.> types).  Given all of the possibilities, which functional should we choose?

The revolution in approaches to using entropy – and the answer to the previous question – begins with a paper by Shore and Johnson <cit.> in which they derive the functional form of the relative entropy for the purposes of updating.  Their approach was to define a set of axioms that constrain the updating procedure, which they then use to determine the functional form of the entropy via eliminative induction.  This approach of axiomitizing the updating process, rather than the measure itself, gives incredible clarity to the subject of inference – while previous justifications for using entropy relied on discussing the properties of the functional, Shore and Johnson instead construct the entropy as the tool for incorporating new information.  This point is made clear in the introduction to their paper,

“Most [approaches to entropy] are based on a formal description of what is required of an information measure; none are based on a formal description of what is required of a method for taking information into account.” (Shore and Johnson, 1980)

While Shore and Johnson's approach provided a systematic way of constructing an entropy functional for the purposes of updating, they had not addressed the question of why one should use a variational principle in the first place, i.e. why the max in MaxEnt.  The explanation for this came in Skilling's paper “The Axioms of Maximum Entropy” <cit.> in which he makes explicit the desire for a tool which ranks posterior distributions with respect to some order of preference.  In <cit.> he states[In his paper <cit.>, Skilling was mainly concerned with reconstructing images with incomplete data.],

“We aim to provide an image [posterior distribution] which is the `best' according to an agreed criterion. This involves setting up a ranking procedure which determines which of two images is `better'. To avoid circularity, and to ensure that there is always some image which is not `bettered' by any other, we impose the transitivity requirement,
(f better than g) and (g better than h) $\Rightarrow$ (f better than h).” (Skilling, 1988)

Once we specify that what we desire is a tool for ranking probabilities – which in turn is the relative entropy – then maximizing it to find the best posterior is completely natural, intuitive and no longer requires any ad hoc justification.

Another set of advancements for entropic methods of inference comes from Caticha <cit.>.  His contributions to the subject include answering questions such as, what constitutes information and are entropic and Bayesian methods compatible, both of which we will discuss in this chapter.  Despite showing that Bayes' rule is a special case of the ME method, his work on the subject <cit.> is perhaps still largely unappreciated.  Before he and his student A. Giffin published their results, it was largely believed that Bayesian and entropic methods referred to two different and unconnected (or even incompatible) ideas.

Caticha's original paper <cit.> introduced the Principle of Minimal Updating (PMU) which we had used for constraining Bayes' rule in the previous chapter.  The idea, which is central to the Bayesian philosophy, allowed Caticha to design the relative entropy using only three design criteria, whereas previous approaches (such as Shore and Johnson, Skilling) required at least four.  As we already mentioned, the design derivation in this thesis follows closely the work of Caticha by adopting the PMU and the same design criteria found in <cit.>.

Apart from answering the question of how we update from a prior to a posterior, we also wish to construct a tool for the purposes of ranking joint distributions with respect to their correlations.  Correlations are a central component to any study involving inference, so the desire for quantitative measures is felt universally.  Such measures are plentiful in information theory, however here our goal is to design one according to first principles.  A design derivation for correlation functionals had not been attempted before K. Vanslette and myself published our results on the matter in the journal Entropy <cit.>.  We will follow the design derivation of that paper in this thesis.

As we will see, the type of correlation functional $I[\mathbf{P}(\tilde{\mathcal{A}})]$ one arrives at depends on a choice of the splits within the proposition space $\tilde{\mathcal{A}}$, and thus the functional we seek is $I[\mathbf{P}(\tilde{\mathcal{A}})]\rightarrow I[\mathbf{P}(\tilde{\mathcal{A}}),\tilde{\mathcal{A}}]$.  For example, if one has a proposition space $\tilde{\mathcal{A}} = \tilde{\mathcal{A}}_1\times\dots\times\tilde{\mathcal{A}}_N$, consisting of $N$ variables, then one must specify which correlations the functional $I[\mathbf{P}(\tilde{\mathcal{A}}),\tilde{\mathcal{A}}]$ should quantify.  Do we wish to quantify how the variable $\tilde{\mathcal{A}}_1$ is correlated with the other $N-1$ variables?  Or do we want to study the correlations between all of the variables?  In our design derivation, each of these questions represent the extremal cases of the family of quantifiers $I[\mathbf{P}(\tilde{\mathcal{A}}),\tilde{\mathcal{A}}]$, the former being a bi-partite correlation (or mutual information) functional and the latter being a total correlation functional.

In the main design derivation we will focus on the the case of total correlation which is designed to quantify the correlations between every variable subspace $\tilde{\mathcal{A}}_i$ in a set of variables $\tilde{\mathcal{A}} = \tilde{\mathcal{A}}_1\times\dots\times\tilde{\mathcal{A}}_N$.  We suggest a set of design criteria (DC) for the purpose of designing such a tool. These DC are guided by the Principle of Constant Correlations (PCC), which states that “the amount of correlations in $\mathbf{P}(\tilde{\mathcal{A}})$ should not change unless required by the transformation, $(\mathbf{P}(\tilde{\mathcal{A}}),\tilde{\mathcal{A}})\stackrel{*}{\rightarrow}(\mathbf{P}(\tilde{\mathcal{A}})',\tilde{\mathcal{A}}')$.” This implies our design derivation requires us to study equivalence classes of $[\mathbf{P}(\tilde{\mathcal{A}})]$ within statistical manifolds $\mathbf{\Delta}$ under the various transformations of distributions $\mathbf{P}(\tilde{\mathcal{A}})$ that are typically performed in inference tasks. We will find, according to our design criteria, that the global quantifier of correlations we desire in this special case is equivalent to the total correlation <cit.>.

Once one arrives at the TC as the solution to the design problem, one can then form special cases such as the mutual information <cit.> or, as we will call them, any $n$-partite information NPI which measures the correlations shared between generic $n$-partitions of the proposition space.  The NPI can also be derived using the same principles as the TC except with one modification, as we will discuss in [npartite]Section V.

The special case of NPI when $n=2$ is the bipartite (or mutual) information, which quantifies the amount of correlations present between two subsets of some proposition space $\tilde{\mathcal{A}}$.  Mutual information (MI) as a measure of correlation has a long history, beginning with Shannon's seminal work on communication theory <cit.> in which he first defines it.  While Shannon provided arguments for the functional form of his entropy <cit.>, he did not provide an axiomatization of MI.

Information is what information does, it changes your mind.Ariel Caticha, 2020
Information can mean many different things depending on the context in which it is used.  For example, the subject of Information Theory considers information to be something that is transmitted from a source to some destination through a channel and this information can be measured, i.e. we can quantify an amount of information.  Information theory however has little to do with the beliefs of rational agents, much less with updating them.  Hence, this view of information is unsatisfactory for our purposes.

We also often hear that information is “physical,” that physical systems “carry” information.  This view was likely inspired by information theory in the interests of understanding the limitations of computation.  Questions such as, what is the minimum amount of energy required to store a single bit of information, or, how much energy is expended when we throw away information, suggest that a piece of information must, in the very least, be represented by something physical.  This idea was pushed further by Landauer who first argued it in his 1991 article “Information is physical” <cit.> that has inspired many others in the field.   He later wrote <cit.>,

“Information is not an abstract entity but exists only through a physical representation, thus tying it to all the restrictions and possibilities of our real physical universe.” (Landauer, 1999)

Essentially, all information must somehow be coupled to the physical world in order to be transmitted from one agent to another, whether it be words inscribed on rock or the electromagnetic waves sent through an ethernet cable.  Therefore, the distinction between some deeper understanding of what constitutes information and its physical representation become unimportant – one simply adopts the pragmatic view that information is physical itself so that no further abstraction is needed.

John Wheeler suggested a similar idea in his paper “Information, Physics, Quantum: The Search for Links,” in which he coined the term “it from bit” <cit.>.  The idea of “it from bit” is that every constituent of existence can be described by a series of yes/no questions, such as is the position of the particle `x'.  By answering either “yes” or “no” to such a question, one is associating a single classical bit of information to the description of a physical object.  Consequently, every physical entity is defined by a set of unique answers of “yes” and “no.”  While this topic is certainly interesting, the idea that information is physical turns out to be too restrictive for the purposes of inference.

Here we adopt the pragmatic view that information is an epistemic quantity.  In this sense, information is what causes us to change our beliefs – it is what we seek when we ask a question.  Indeed, since the inductive inference framework was developed on the basis of reasoning with incomplete information, it is only natural to seek more complete information, with the goal of improving our ability to reason.  Following Caticha <cit.>, we state the defining characteristics of information,

“Information is what affects and therefore constraints rational beliefs.  Information is what forces a change in beliefs.” (Caticha, 2020)

In this epistemic definition, there is no notion of an amount of information that can be measured.  New information is simply a set of constraints on our beliefs and the constraints specify what we should believe given that we were ideally rational.

There is an instructive analogy of our definition of information with that of physics, which was presented in <cit.>.  In classical mechanics, a system is defined by its quantity of motion, its momentum.  The systems state of motion changes whenever it is acted upon by forces – an initial state $\vec{p}_i$ changes to some final state $\vec{p}_f$ under the influence of some external force $\vec{F}_{\mathrm{ext.}}$.  In inference a state is defined by degrees of belief, a set of probabilities $\mathbf{P}(\tilde{\mathcal{A}})$.  Information then, is the force which causes a change from some initial set of beliefs, $\mathbf{Q}(\tilde{\mathcal{A}})$, to a final set of beliefs, $\mathbf{P}(\tilde{\mathcal{A}})$.  In this sense, updating beliefs is a form of dynamics.  A generalization to this dynamics is discussed in chapter four, and the application of its generalization to physics as an Entropic Dynamics is discussed in part two of this thesis.

Correlations? —
The term correlation may be defined colloquially as being a relation between two or more “things".
While we have a sense of what correlations are, how do we quantify this notion more precisely?  If correlations have to do with “things” in the real world, are correlations themselves “real?”  Can correlations be “physical?”  In the context of inference, correlations are broadly defined as being statistical relationships between propositions.  In this thesis we adopt the view that whatever correlations may be, their effect is to influence our beliefs about the natural world.  In this sense, correlations are just a particular type of information.  More specifically,

Correlations are the pieces of information that constrain conditional beliefs.

The design for correlation quantifiers can be motivated by first examining the more intuitive extreme cases.  Perhaps the most intuitive situation occurs when two systems are actually independent – the joint system contains no correlations whatsoever.  We would expect in this case that our joint beliefs would factor, $p(a,b) = p(a)p(b)$, since knowing either the value of $a$ or $b$ tells us nothing about the other.  The other extreme happens when $a$ and $b$ are completely determined by each other – the joint system is maximally correlated.  In this case, we would expect our joint beliefs to behave as $p(a,b) = p(a)\delta(b - f_a(a)) = p(b)p(a - f_b(b))$ where the delta functions signify that $b$ is some function of $a$, $b = f_a(a)$, and that this function is invertible, $f_a = f_b^{-1}$.  While we understand both of these edge cases, it is not entirely clear how one quantifies anything that happens in the middle.  If we are given two joint distributions $p_1(a,b) = p_1(a)p_1(b|a)$ and $p_2(a,b) = p_2(a)p_2(b|a)$, how can we decide if one represents a more correlated state than the other?

Quantifying correlations as part of entropic inference is fundamentally different from the procedure of updating.  While they will both involve objects which take the form of relative entropies, the correlation quantifiers are not designed as tools for updating.  They serve the weaker function of simply providing a ranking of joint distributions with respect to their correlations.  There is no variational principle which guides the construction of the correlation quantifiers, and hence more details are needed in order to construct them.

In Section 2 we will outline the basic principles for constructing a tool for updating, as well as a tool for ranking correlations.  In Section 3 we will enumerate various design criteria inspired by these principles and then go on to demonstrate the solution to the updating problem.  We will also breifly discuss the special case of Bayes' rule from the ME method.  Section 4 will be similar to Section 3 in that we will specify design criteria for the purposes of deriving a set of correlation quantifiers.  We conclude with Section 5 which discusses various results surrounding sufficiency that were developed in <cit.>.

§.§ The design of entropic inference

Now that we have settled the question as to what constitutes information, we need to determine a systematic way of incorporating that new information into our beliefs – we must answer the question of how does one update their beliefs in the presence of new information?  In the spirit of chapter two, we will define a set of reasonable criteria that our entropic inference must adhere to.  This defines a rationale for entropic inference, which we label $\mathcal{R}_S$.

Since entropic inference is concerened with updating beliefs, we can take some suggestions for criteria of rationality from the rationale of inductive inference $\mathcal{R}$.  For one, since the design of inductive inference is agnostic to the underlying subject matter for which it is used, so too should be the method of updating.  The updating scheme must apply to every situation, otherwise how would one know which updating scheme to use?

The method for updating beliefs should be independent of the subject matter, it must have universal applicability.

The criterion of universal applicability has profound implications.  If our method was required to behave differently if we were updating beliefs about the stock market as opposed to physics, then we would have broken the rationale set forth in the inductive inference framework, which is unacceptable.  This means that our beliefs in the inner workings of the natural world must be subject to the same dynamical rules as our beliefs in anything else.

Influenced by the discussion of section ... in chapter two, we adopt a principle of parsimony.  Specifically, we adopt the view that past information is valuable and that one should not throw away what was once learned unless the new information renders it obsolete.  This view is expressed in a slightly altered version of the principle of minimal updating that was given in chapter two.

[Parsimony: Minimal Updating]
Beliefs should be updated only to the extent required by the new information.

The principle of minimal updating (PMU) automatically imposes a level of objectivity into the inference framework.  For example, consider the special case for when there is no new information available.  The PMU suggests that any ideally rational agent (Ira) should not change their beliefs.  To do the opposite would violate common sense – rational agents should not change their minds arbitrarily.  Another objective component of the PMU derives from the same situation.  Because we are stipulating when an Ira is not supposed to update their beliefs, the act of not updating is unique, i.e. there are many ways to change something, but only one way to keep it the same.

Like with the rationale for inductive inference, we impose a principle of consistency.  This principle is inspired by one of the central aspects of scientific inquiry, namely independence.  A basic requirement for any scientific model to be successful is that all relevant variables are taken into account and that whatever was left out, the rest of the universe, does not matter.  In other words, in order to do science we must be able to understand parts of the universe without understanding its entirety.  The central idea is that we can focus on a particular system of interest while neglecting everything else because the two systems are statistically independent, i.e. they are uncorrelated.  Following Caticha <cit.>, we adopt the requirement that,

“Whenever two systems are a priori believed to
be independent and we receive information about one it should not matter if
the other is included in the analysis or not.” (Caticha, 2014)

This idea can be stated in the form of a consistency requirement.
[Consistency: Independence]
Whenever systems are known to be independent, it should not matter whether the analysis treats them jointly or separately.

In other words, if there exist multiple ways to conduct the same inference, the results better agree.  This sentiment was declared in Shore and Johnson <cit.> whose axioms were based on a fundamental principle of consistency,

“Since the maximum entropy principle is asserted as a general method of inductive inference, it is reasonable to require that different ways of using it to take the same information into account should lead to consistent results.” (Shore and Johnson, 1980)

These three principles, Universality, Parismony and Consistency, form the basis of our rationale $\mathcal{R}_S$ which we will use to constrain our design of entropic inference.

Concerning correlations — To design our correlation quantifiers, we must also decide on a set of criteria in the form of a rationale, $\mathcal{R}_I$.  Certainly we would expect any tool for quantifying correlations to be of universal applicability, so that the Universality principle (<ref>) is also included in $\mathcal{R}_I$.  It is also reasonable to expect that the Consistency principle be included in $\mathcal{R}_I$ since properties of independence play a crucial role in the study of correlations.

Since correlation quantifiers are not tools for updating, their design does not warrant the PMU.  Instead of focusing on what happens when we update our beliefs, we can consider a broader class of transformations on probability distributions that was discussed in <cit.>.  These include coordinate transformations, marginalization, products and entropic updating as a special case.  In certain special cases of these transformations, we expect correlations to remain unchanged, and hence we impose that they are not.  This notion can be summarized in the following principle which is similar to the PMU,

[Constant Correlations]
The amount of correlations present in a state of knowledge should not change unless required by a particular transformation.

Much like the PMU, the Principle of Constant Correlations (PCC) imposes a level of objectivity in the design of correlation quantifiers since it instructs us when not to make changes to our beliefs about correlations.  In one sense, it is more general than the PMU since it concerns transformations which are of a more general form than entropic updating.  On the other hand, correlations occupy only a subset of the kind of information that is present in any set of beliefs and in this way it refers to a special case.

In conclusion, the rationale $\mathcal{R}_I$ is identical to $\mathcal{R}_S$ except for swapping the PMU with the PCC.  This is expected, since each tool is designed with a different purpose in mind.

The universe of discourse —
The arena of any inference task consists of two ingredients, the first of which is the subject matter, or what is often called the universe of discourse.  This refers to the actual propositions that one is interested in making inferences about.  Propositions tend to come in two classes, either discrete or continuous.  Discrete proposition spaces will be denoted by caligraphic uppercase latin letters, $\mathcal{X}$, and the individual propositions will be lowercase latin letters $x_i \in \mathcal{X}$ indexed by some variable $i = \{1,\dots,|\mathcal{X}|\}$, where $|\mathcal{X}|$ is the number of distinct propositions in $\mathcal{X}$.

From this chapter onward we will generally be interested in universes of discourse which are represented by continuous spaces $\mathbf{X}$, such as the position of a particle in $\mathbb{R}^3$.  Thus, the spaces of statements $\tilde{\mathcal{A}}$ as well as the web of beliefs $\mathbf{P}(\tilde{\mathcal{A}})$ become uncountable sets.  In accordance with the notation of Chapter 2 Section ..., we will denote probability distributions over $\mathbf{X}$ as $p(x)$ and priors as $q(x)$.  For generic distributions we will use the Greek $\rho$ for posteriors and $\varphi$ for priors.

The second ingredient that one needs to define for general inference tasks is the space of models, or the space of probability distributions which one wishes to assign to the underlying proposition space.  These spaces can often be given the structure of a manifold, which in the literature is called a statistical manifold <cit.>[We will discuss in more detail the structure of statistical manifolds in the next chapter, but for now it suffices to think of $\sm$ as simply a set of distributions.].A statistical manifold $\sm$, is a manifold in which each point $p(x) \in \sm$ is an entire probability distribution, i.e. $\sm$ is a space of maps from subsets of $\mathbf{X}$ to the interval $[0,1]$, $p(x):\mathcal{P}(\mathbf{X})\rightarrow[0,1]$.  The notation $\mathcal{P}(\mathbf{X})$ denotes the power set of $\mathbf{X}$, which is the set of all subsets of $\mathbf{X}$, and has cadinality equal to $|\mathcal{P}(\mathbf{X})| = 2^{|\mathbf{X}|}$.

The correlations present in any distribution $p(x)$ necessarily depend on the conditional relationships between various propositions.  For instance, consider the binary case of just two proposition spaces $\mathbf{X}$ and $\mathbf{Y}$, so that the joint distribution factors,

\begin{equation}
p(x,y) = p(x)p(y|x) = p(y)p(x|y).\label{firstjoint}
\end{equation}

The correlations present in $p(x,y)$ will necessarily depend on the form of $p(x|y)$ and $p(y|x)$ since the conditional relationships tell us how one variable is statistically dependent on the other.  As we will see, the correlations defined in the above eq. are quantified by the mutual information.  For situations of many variables however, the global correlations are defined by the total correlation, which we will design first.  All other measures which break up the joint space into conditional distributions (including (<ref>)) are special cases of the total correlation.

A tool for updating —
We wish to design a tool for the purposes of updating an entire probability distribution $\varphi\xrightarrow{\star} \rho$ when new information comes in the form of constraints.  This will be done through the process of eliminative induction, which was discussed in the Introduction of Part I.  The goal is to construct a tool that will allow us to systematically search for a preferred posterior distribution.  The procedure, which was first proposed by Skilling <cit.>, is quite simple: first, rank the possible posterior distributions according to some order of preference, and then pick the one which ranks highest.

While simple, the design specification suggested by Skilling is highly constraining.  For one, it implies that our preferences in posterior distributions should be transitive, i.e. if distribution $\rho_1$ is preferred to distribution $\rho_2$ and distribution $\rho_2$ is preferred to $\rho_3$ then $\rho_1$ must also be preferred to $\rho_3$.  Similar to the construction of probabilities as degrees of belief, we implement a ranking by assigning a real number to each posterior distribution $S[p]$ in such a way that if $\rho_1$ is preferred to $\rho_2$ then $S[\rho_1] > S[\rho_2]$.

As pointed out by Caticha <cit.>, the choice to rank distributions implies that the updating scheme will take the form of a variational principle.  This principle, which we call the Method of Maximum Entropy (ME), allows us to determine the preferred posterior distribution $\rho$, by maximizing the entropy functional $S[\rho]$.  Since the ranking of posteriors must be done with respect to a prior $\varphi$, the desired entropy functional must depend also on the prior, $S[\rho] \rightarrow S[\rho,\varphi]$.  In this sense, the entropy functional ranks the posterior $\rho$ relative to the prior $\varphi$, and hence $S[\rho,\varphi]$ is typically called the relative entropy.

Let $\varphi$ be a web of prior beliefs and $\rho$ be a posterior web.  The order of preference given to $\rho$ is quantified by the real number,

\begin{align}
S:\Delta\times\Delta &\rightarrow \mathbb{R}\nonumber\\
(\rho,\varphi) &\mapsto S[\rho,\varphi],
\end{align}

which is called the relative entropy.

To find the desired entropy functional, we use the process of eliminative induction by specifying certain design criteria.  These design criteria (DC) specify the behavior of the ME method in certain special cases in which we know the answer.  Amazingly, by specifying a small set of DC one can arrive at a general functional form for the entropy.  This procedure has been exercized many times in the past <cit.> and each derivation has become increasingly simpler.  Shore and Johnson <cit.> and Skilling <cit.> each give four design critera.  Caticha <cit.> reduced the required DC to three, while Vanslette <cit.> used only two[For a review on some of the various approaches to designing entropy functionals see Csiszar <cit.>.].  In the following sections we will mostly follow the procedures of Caticha and Vanslette, while also pointing out the more historical versions.

A tool for ranking correlations —
In order to better motivate the construction of correlation functionals, it will be instructive to consider what happens when we change coordinates through some kind of generic coordinate transformation.  A coordinate transformation $f:\mathbf{X}\rightarrow \mathbf{X}'$, is a special type of transformation of the proposition space $\mathbf{X}$ that respects certain properties.  It is essentially a continuous version of a reparameterization[A reparameterization is an isomorphism between discrete proposition spaces, $g:\mathcal{X}\rightarrow\mathcal{Y}$ which identifies for each proposition $x_i \in \mathcal{X}$, a unique proposition $y_i \in \mathcal{Y}$ so that the map $g$ is a bijection.].  For one, each proposition $x \in \mathbf{X}$ must be identified with one and only one proposition $x'\in \mathbf{X}'$ and vice versa.  This means that coordinate transformations must be bijections on proposition space – i.e. an isomorphism between sets.  The reason for this is simply by design, we would like to study the transformations that leave the proposition space invariant.  A general transformation of this type on $\sm$ which takes $\mathbf{X}$ to $\mathbf{X}' = f(\mathbf{X})$, is met with the following transformation of the densities,

\begin{equation}
p(x) \rightarrow p'(x') \quad \mathrm{where} \quad p(x)dx= p'(x')dx'.\label{i}
\end{equation}

Like we already mentioned, the coordinate transforming function $f:\mathbf{X}\rightarrow\mathbf{X}'$ must be a bijection in order for (<ref>) to hold, i.e. the map $f^{-1}:\mathbf{X}'\rightarrow\mathbf{X}$ is such that $f\circ f^{-1} = \mathrm{id}_{\mathbf{X}'}$ and $f^{-1}\circ f = \mathrm{id}_{\mathbf{X}}$.  While the densities $p(x)$ and $p'(x')$ are not necessarily equal, the probabilities defined in (<ref>) must be (according to the rules of probability theory, see the Appendix B of <cit.>). This indicates that $\rho\rightarrow\rho'=\rho$ is in the same location in the statistical manifold. That is, the global state of knowledge has not changed – what has changed is the way in which the local information in $\rho$ has been expressed, which must be invertible in general.

For a coordinate transformation (<ref>) involving two variables, $x \in \mathbf{X}$ and $y \in \mathbf{Y}$, we also have,

\begin{equation}
p(x,y) \rightarrow p'(x',y') \quad \mathrm{where}\quad p(x,y)dxdy = p'(x',y')dx'dy'.
\end{equation}

A few general properties of these transformations are as follows:  First,
the density $p(x,y)$ can be expressed in terms of the density $p'(x',y')$,

\begin{equation}
p(x,y) = p'(x',y')\gamma(x',y'),
\end{equation}

where $\gamma(x',y')$ is the Jacobian <cit.> that defines the transformation,

\begin{equation}
\gamma(x',y') = |\det\left(J(x',y')\right)|, \quad \mathrm{where}\quad J(x',y') = \begin{bmatrix}
\frac{\partial x'}{\partial x} & \frac{\partial x'}{\partial y} \\ \frac{\partial y'}{\partial x} & \frac{\partial y'}{\partial y}
\end{bmatrix}.
\end{equation}

For a finite number of variables $x = (x_1,\dots,x_N)$, general coordinate transformations $p(x_1,\dots,x_N) \rightarrow p'(x_1',\dots,x_N')$ are written,

\begin{equation}
p(x_1,\dots,x_N)\prod_{i=1}^Ndx_i = p'(x_1',\dots,x_N')\prod_{i=1}^Ndx_i',
\end{equation}

and the Jacobian becomes,

\begin{equation}
J(x'_1,\dots,x'_N) = \begin{bmatrix}
\frac{\partial x_1'}{\partial x_1} &\cdots &\frac{\partial x_1'}{\partial x_N} \\ \vdots & \ddots & \vdots \\ \frac{\partial x_N'}{\partial x_1} & \cdots & \frac{\partial x_N'}{\partial x_N}
\end{bmatrix}.
\end{equation}

One can also express the density $p'(x')$ in terms of the original density $p(x)$ by using the inverse transform,

\begin{equation}
p'(x') = p(f^{-1}(x'))\gamma(x) = p(x)\gamma(x).
\end{equation}

Split Invariant Coordinate Transformations —
Consider a class of coordinate transformations that result in a diagonal Jacobian matrix, i.e.,

\begin{equation}
\gamma(x'_1,\dots,x'_N) = \prod_{i=1}^N\frac{\partial x'_i}{\partial x_i} = \prod_{i=1}^N\gamma(x_i').\label{diag}
\end{equation}

These transformations act within each of the variable spaces independently, and hence they are guaranteed to preserve the definition of the split between any $n$-partitions of the propositions – and because they are coordinate transformations, they are invertible and do not change our state of knowledge, $(\rho,\mathbf{X})\rightarrow(\rho',\mathbf{X}') = (\rho,\mathbf{X})$.  We call such special types of transformations (<ref>) split invariant coordinate transformations.  The marginal distributions of $p(x)$ are preserved under split invariant coordinate transformations,

\begin{equation}
\forall\, \mathbf{X}_i \subset \mathbf{X} : p(x_i)dx_i = p'(x_i')dx_i'.\label{marginal}
\end{equation}

If one allows generic coordinate transformations of the joint space, then the marginal distributions may depend on variables outside of their original split. Thus, if one redefines the split after a coordinate transformation to new variables $\mathbf{X}\rightarrow\mathbf{X}'$, the original problem statement changes as to what variables we are considering correlations between and thus eq. (<ref>) no longer holds. This is apparent in the case of two variables $(x,y)$.  Let $x' = f_{x'}(x,y)$ so that

\begin{equation}
dx' = df_{x'} = \frac{\partial f_{x'}}{\partial x}dx + \frac{\partial f_{x'}}{\partial y}dy,
\end{equation}

which depends on $y$.  Redefining the split after this coordinate transformation breaks the original independence since a distribution which originally factors, $p(x,y) = p(x)p(y)$, would be made to have conditional dependence in the new coordinates, i.e. if $x' = f_{x'}(x,y)$ and $y' = f_{y'}(x,y)$, then,

\begin{equation}
p(x,y) = p(x)p(y) \rightarrow p'(x',y') = p'(x')p'(y'|x').
\end{equation}

So, even though the above transformation satisfies (<ref>), this type of transformation may change the correlations in $p(x)$ by allowing for the potential redefinition of the split $\mathbf{X}\rightarrow\mathbf{X}'$.  Hence, when designing our functional, we identify split invariant coordinate transformations as those which preserve correlations. These restricted coordinate transformations help isolate a single functional form for our global correlation quantifier.

A split in the proposition space can be defined by an index set $K \in \mathbf{K}$ which is a set of sets of indicies corresponding to the groupings of subspaces in proposition space.  Each value in $K$ corresponds to an indexing of the powerset of subspaces of $\mathbf{X}$[For example, the space of three variables, $\mathbf{X} = \mathbf{X}_1\times\mathbf{X}_2\times\mathbf{X}_3$, has four possible subspace splits,

\begin{align}
\{\{\mathbf{X}_1\},\{\mathbf{X}_2\},\{\mathbf{X}_3\}\}, \quad \{\{\mathbf{X}_1\},\{\mathbf{X}_2,\mathbf{X}_3\}\}, \quad \{\{\mathbf{X}_1,\mathbf{X}_2\},\{\mathbf{X}_2\}\}, \quad \{\{\mathbf{X}_1,\mathbf{X}_3\},\{\mathbf{X}_2\}\},\nonumber
\end{align}

so that their is a corresponding space $\mathbf{K}$ with the following elements,

\begin{equation}
\{\{1\},\{2\},\{3\}\}, \quad \{\{1\},\{2,3\}\}, \quad \{\{1,2\},\{3\}\}, \quad \{\{1,3\},\{2\}\}.
\end{equation}

]. A splitting of $\mathbf{X}$ is then given by the collection of subsets, $\{\mathbf{X}^{(i)}\}_{(k)} = \left\{\mathbf{X}^{(1)},\mathbf{X}^{(2)},\dots,\mathbf{X}^{(k)}\right\}$.  Given a particular split in the proposition space, a correlation quantifer can be defined as,

Let $\mathbf{X}$ be a joint proposition space and let $\mathbf{K}$ denote the set of possible splittings of $\mathbf{X}$.  The correlations present in $\rho$ with respect to a splitting $K \in \mathbf{K}$ is quantified by the real number,

\begin{align}
I:\sm\times \mathbf{K}&\rightarrow \mathbb{R}\nonumber\\
(\rho,\{\mathbf{X}^{(i)}\}_{(k)}) &\mapsto I[\rho,\mathbf{X}^{(1)};\dots;\mathbf{X}^{(k)}].
\end{align}

Whatever the value of our desired quantifier $I[\rho]$ gives for a particular distribution $\rho$, we expect that if we change $\rho \xrightarrow{*} \rho' = \rho + \delta \rho$, that our quantifier also changes $I[\rho] \rightarrow I[\rho'] = I[\rho] + \delta I$, and that this change of $I[\rho]$ reflects the change in the correlations, i.e. if $\rho$ changes in a way that increases the correlations, then $I[\rho]$ should also increase.  Thus, our quantifier should be an increasing functional of the correlations, i.e. it should provide a ranking of $\rho$'s.

§.§ The relative entropy

Before we begin, it is perhaps instructive to specify explicitly the design goal.
[The ME method]
Given a prior distribution $\varphi$, and a set of possible posterior distributions $\rho_1,\rho_2,\dots$, we seek to design a functional which ranks each posterior with respect to some order of preference.

The design goal imposes that the functional $S[\rho,\varphi]$ be scalar valued so that one can rank various posteriors with respect to the prior.  The order of preference in the ranking is determined by setting the various design criteria.  A general functional form for $S[\rho,\varphi]$ can be written as

\begin{equation}
S[\rho,\varphi] = S[p(x),q(x);p(x'),q(x');\dots],\label{Sbare}
\end{equation}

where $S[\rho,\varphi]$ depends on each value of the probabilities $p(x)$ and $q(x)$ for every $x \in \mathbf{X}$.  To constrain the functional form of (<ref>) further, we impose various design criteria (DC).  There are several ways to proceed by choosing different DC.  Whichever approach one takes <cit.>, there are some defining characteristics of the entropy functional that are common among all of them.  These criteria have to do with the behavior of $S[\rho,\varphi]$ when certain sectors of the underlying space $\mathbf{X}$ are independent.  Independence in $\mathbf{X}$ comes in two flavors, subdomain and subsystem independence.  The former refers to subsets of $\mathbf{X}$ while the latter to subspaces.  In Vanslette <cit.>, these two criteria on independence are enough to specify $S[\rho,\varphi]$ for the purposes of ranking.  The DC are imposed on the basis of the PMU.  While simple, the PMU is incredibly constraining.  By stating when one is not supposed to change their beliefs it is operationally unique, since there are infinitely many choices for how one ought to change them.  Consequently, the PMU imposes a level of objectivity into $S[p,q]$.

The DC we adopt in this thesis are the same as those given by Caticha in <cit.>.  The first DC concerns local, subdomain, updates to the probability distribution $p(x)$.
[Subdomain Independence]
Local information has local effects.

Assume that new information tells us we should update our beliefs about a particular subdomain $\mathcal{D} \subset \mathbf{X}$, but does not give us any information about the complement, $\mathcal{D}^c = \mathbf{X} \backslash \mathcal{D}$.  According to the PMU, we should not update our beliefs about probabilities which concern the complement domain, hence we design the inference procedure so that $q(x|\mathcal{D}^c)$ is not updated, i.e.

\begin{equation}
p(x|\mathcal{D}^c) = q(x|\mathcal{D}^c).\label{dc11}
\end{equation}

Eq. (<ref>) is an example of the objectivity that is demanded by the PMU.  Since no new information about $\mathcal{D}^c$ is given, the prior distribution takes presidence.  This point is summed up nicely in <cit.>,

“We emphasize: the point is not that we make the unwarranted assumption
that keeping $q(x|\mathcal{D}^c)$ unchanged is guaranteed to lead to correct inferences. It
need not; induction is risky. The point is, rather, that in the absence of any
evidence to the contrary there is no reason to change our minds and the prior
information takes priority.” (Caticha, 2020, Section 6.2.3)

This criteria appears in all four of the design derivations discussed throughout[In Shore and Johnson's approach to relative entropy <cit.>, axiom four is analogous to DC 1, which states on page 27 “IV. Subset Independence: It should not matter whether one treats an independent subset of system states in terms of a separate conditional density or in terms of the full system density.”  In Skilling's approach <cit.> DC 1 appears as axiom one which, like Shore and Johnson's axioms, is called Subset Independence and is justified with the following statement on page 175, “Information about one domain should not affect the reconstruction in a different domain, provided there is no constraint directly linking the domains.”  In Caticha <cit.> the axiom called Locality and is written on page four as “Criterion 1: Locality. Local information has local effects.”  Finally, in Vanslette's work <cit.>, the subset independence criteria is stated on page three as follows, “Subdomain Independence:  When information is received about one set of propositions, it should not effect or change the state of knowledge (probability distribution) of the other propositions (else information was also received about them too).”]. DC1 constrains the functional form of the relative entropy to be additive in non-overlapping subdomains,

\begin{equation}
S[\rho,\varphi] \xrightarrow{\mathbf{DC1}} \int dx\, F\left(p(x),q(x),x\right),\label{SDC1}
\end{equation}

where $F(p(x),q(x),x)$ is some undetermined function of the probabilities and possibly the coordinates.

[Coordinate invariance]
The system of coordinates carries no information.

Because we can choose whatever labels we want for the points $x \in \mathbf{X}$, any particular choice cannot have an impact on the ranking of the probability distributions.  While in some situations certain coordinates may be preferred over others, it is not an assumption that we should make unless explicitly stated.  As a result of DC2 the functional form of the entropy becomes,

\begin{equation}
S[\rho,\varphi] \xrightarrow{\mathbf{DC2}} \int dx\, q(x)\Phi\left(\frac{p(x)}{q(x)}\right).\label{dc2a}
\end{equation}

DC2 has replaced the function $F$, which had three arguments, by a function $\Phi$ which only has one.  To see the effects of DC2, consider a generic coordinate transformation in which $\mathbf{X} \rightarrow \mathbf{X}' = f(\mathbf{X})$ so that,

\begin{equation}
p(x) \rightarrow p'(x') = p(x)\gamma(x') \quad \mathrm{and} \quad q(x) \rightarrow q'(x') = q(x)\gamma(x').
\end{equation}

This particular transformation of the densities $p(x)$ and $q(x)$, and the coordinates $\mathbf{X}$, will always leave integrals of the form (<ref>) invariant,

\begin{align}
S[\rho,\varphi] &= \int dx\, F(p(x),q(x),x)\nonumber\\
&= \int dx'\, \gamma(x')F\left(p'(x')\gamma(x')^{-1},q'(x')\gamma(x')^{-1},f(x)\right).\label{dc1b}
\end{align}

The above expression is valid for any change of variables, and hence it imposes no constraints on the functional form of $F$.  Imposing DC2 amounts to saying that we could have started with the coordinates $x' = f(x)$ and it would have made no difference, i.e. the ranking provided by

\begin{equation}
S[\rho',\varphi'] = \int dx'\, F(p'(x'),q'(x'),x'),
\end{equation}

is the same as the one given in (<ref>).

[Subsystem Independence]
When two systems are a priori believed to be independent and we receive
independent information about each then it should not matter if one is
included in the analysis of the other or not (and vice versa).

Like DC1, DC3 imposes the PMU by saying what not to do, rather than what one should do.  In particular, one should not introduce correlations between independent subsystems unless the constraints suggest otherwise[In Shore and Johnson's approach <cit.>, axiom three concerns subsystem independence and is stated on page 27 as “III. System Independence: It should not matter whether one accounts for independent information about independent systems separately in terms of different densities or together in terms of a joint density.”  In Skillings approach <cit.>, the axiom concerning subsystem independence is given by axiom three on page 179 and provides the following comment on page 180 about its consequences “This is the crucial axiom, which reduces S to the entropic form. The basic point is that when we seek an uncorrelated image from marginal data in two (or more) dimensions, we need to multiply the marginal distributions. On the other hand, the variational equation tells us to add constraints through their Lagrange multipliers. Hence the gradient $\delta S/\delta f$ must be the logarithm.”  In Caticha's design derivation <cit.>, axiom three concerns subsystem independence and is written on page 5 as “Criterion 3: Independence.  When systems are known to be independent it should not matter whether they are treated separately or jointly.” Finally, in Vanslette <cit.> on page 3 we have “Subsystem Independence:  When two systems are a priori believed to be independent and we only receive information about one, then the state of knowledge of the other system remains unchanged.”].  Consider that $\mathbf{X}$ forms a composite system, $\mathbf{X} = \mathbf{X}_1\times\mathbf{X}_2$, and that our initial information suggests that the two systems be independent.  This amounts to the prior being the product $q(x_1,x_2) = q(x_1)q(x_2)$.  Now suppose we obtain information about $\mathbf{X}_1$ such that the prior $q(x_1)$ is by itself updated to the posterior $p(x_1)$.  Suppose that we also obtain information that the prior $q(x_2)$ is by itself updated to $p(x_2)$.  DC3 imposes that the joint prior $q(x_1)q(x_2)$ be updated to the joint posterior $p(x_1)p(x_2)$, since in the absence of information to the contrary, inferences about one subsystem do not affect inferences about the other.

Imposing DC3 leads to the final functional form of the relative entropy,

\begin{equation}
S[\rho,\varphi] \xrightarrow{\mathbf{DC3}} -\int dx\, p(x)\log\frac{p(x)}{q(x)},\label{dc3a}
\end{equation}

up to terms which do not affect the updating scheme.  As we have expressed before, the point of DC3 is not to impose some type of consistency requirement between the prior and some new information which perhaps tells us that the variables remain independent.  On the contrary, the role of DC3 is to impose that in the absence of new information about correlations, the prior must necessarily take presidence.  This point is summed up nicely in <cit.>,

“We emphasize that the point is not that when we have no evidence for correlations we draw the conclusion that the systems must necessarily be independent. Induction involves risk; the systems might in actual fact be correlated. The point is rather that if the joint prior reflected independence and the new evidence is silent on the matter of correlations, then the only evidence we actually have – namely, the prior – takes precedence and there is no reason to change our minds. As before, a feature of the probability distribution – in this case, independence – will not be updated unless the evidence requires it.” (Caticha, 2020)

Summary of the results —
The functional in (<ref>) is constructed to serve the purposes of the design goal, namely, to allow one to update a prior distribution $q(x)$ to a posterior distribution $p(x)$ when new information comes in the form of constraints $\mathcal{C}$.  In this sense, we have extended the method of MaxEnt to include arbitrary priors.  It is also the case that the functional in (<ref>) requires no further interpretation – it does not represent heat, disorder or an amount of information – it is simply a tool for updating.  Again, to quote Caticha <cit.>, “we do not need to know what entropy means; we only need to know how to use it.”

If one wishes to compare with other approaches to entropic inference, the work by Csiszár <cit.> provides a nice summary of the various axioms used by many authors (including Azcél <cit.>, Shore and Johnson <cit.> and Jaynes <cit.>) in their definitions of information theoretic measures[A list is given on page 3 of <cit.> which includes the following for conditions on an entropy function $H(P)$; (1) Positivity ($H(P) \geq 0$), (2) Expansibility (“expansion” of $P$ by a new component equal to $0$ does not change $H(P)$, i.e. embedding in a space in which the probabilities of the new propositions are zero), (3) Symmetry ($H(P)$ is invariant under permutation of the probabilities), (4) Continuity ($H(P)$ is a continuous function of $P$), (5) Additivity ($H(P\times Q) = H(P) + H(Q)$), (6) Subadditivity ($H(X,Y) \leq H(X) + H(Y)$), (7) Strong additivity ($H(X,Y) = H(X) + H(Y|X)$), (8) Recursivity ($H(p_1,\dots,p_n) = H(p_1+p_2,p_3,\dots,p_n) + (p_1+p_2)H(\frac{p_1}{p_1+p_2},\frac{p_2}{p_1+p_2})$) and (9) Sum property ($H(P) = \sum_{i=1}^ng(p_i)$ for some function $g$).].  One could associate the design criteria in this work to some of the common axioms enumerated in <cit.>, although some of them will appear as consequences of imposing a specific design criterion, rather than as an ansatz.  For example, the strong additivity condition is the result of imposing DC1 and DC3.  Continuity of $S[\rho,\varphi]$ with respect to $\rho$ is imposed through the design goal, and symmetry is a consequence of DC1.  In summary, Design Goal$\rightarrow$continuity, DC1$\rightarrow$symmetry, (DC1 + DC2)$\rightarrow$strong additivity.  As was shown by Shannon <cit.> and others <cit.>, various combinations of these axioms, as well as the ones mentioned in footnote 4, are enough to characterize entropic measures.

§.§ Proof of the main result

We begin by considering first a discrete universe of discourse $\mathcal{X} = \{x_1,\dots,x_N\}$ consisting of $N$ variables and $|\mathcal{X}|$ possible values $x_i \in \mathcal{X}$ where $i = \{1,\dots,|\mathcal{X}|\}$.  We wish to update a prior distribution $Q(x_i)$ to a posterior $P(x_i)$ when new information becomes available.  At this point, in accordance with (<ref>), the entropy functional $S[P,Q]$ takes the general form,

\begin{equation}
S[P,Q] = S[P(x_1),Q(x_1);P(x_2),Q(x_2);\dots;P(x_N),Q(x_N)].\label{disc}
\end{equation}

DC1 Implementation —
Consider that the space $\mathcal{X}$ can be broken up into two disjoint domains of interest, $\mathcal{D} \subseteq \mathcal{X}$ and its complement $\mathcal{D}^c = \mathcal{X}\backslash \mathcal{D}$, so that $\mathcal{D} \cap \mathcal{D}^c = \emptyset$ and $\mathcal{D} \cup \mathcal{D}^c = \mathcal{X}$.  Consider also that we obtain some information that refers strictly to the subdomain $\mathcal{D}$, but not to its complement.  By DC1 we impose that we will not update our beliefs relative to the complement domain so that the posterior probabilities $P(x_i|\mathcal{D}^c)$ must follow,

\begin{equation}
P(x_i|\mathcal{D}^c) = Q(x_i|\mathcal{D}^c).
\end{equation}

To determine the consequences of this situation, we impose a set of local constraints.  Consider the constraint on $\mathcal{D}$,

\begin{equation}
\sum_{x_i \in \mathcal{D}} P(x_i)A(x_i) = A_{\mathcal{D}},\label{proof1}
\end{equation}

where $A_{\mathcal{D}}$ is a constant.  DC1 imposes that the constraint (<ref>) shall have no influence on the conditional probabilities $p(x_i|\mathcal{D}^c)$, however it could introduce a scaling of the probabilities $P(x_i)$ over $\mathcal{D}^c$.  To deal with this, consider that we also impose constraints on the normalization of each subdomain

\begin{equation}
P(\mathcal{D}) = \sum_{x_i \in \mathcal{D}} P(x_i) \qquad \mathrm{and} \qquad P(\mathcal{D}^c) = \sum_{x_i \in \mathcal{D}^c}P(x_i),\label{dc1norm}
\end{equation}

such that $P(\mathcal{D}) + P(\mathcal{D}^c) = 1$.  To search the space of possible posterior distributions $P(x_i)$ which satisfy (<ref>) and (<ref>), we maximize the entropy functional subject to the constraints in (<ref>) and (<ref>) using the Lagrange multiplier method,

\begin{align}
\delta\left[\vphantom{\sum_{x_i \in \mathcal{D}}}S[P,Q]\right. &- \lambda_{\mathcal{D}}\left(\sum_{x_i \in \mathcal{D}}P(x_i) - P(\mathcal{D})\right) - \lambda_{\mathcal{D}^c}\left(\sum_{x_i \in \mathcal{D}^c}P(x_i) - P(\mathcal{D}^c)\right)\nonumber\\
&\left.- \mu_{\mathcal{D}}\left(\sum_{x_i \in \mathcal{D}}P(x_i)A(x_i) - \langle A\rangle_{\mathcal{D}}\right)\right] = 0,
\end{align}

which gives the set of $N+3$ differential equations – one for each $P(x_i)$ and the Lagrange multipliers,

\begin{align}
\frac{\delta S[P,Q]}{\delta P(x_i)} &= \lambda_{\mathcal{D}^c}, \quad \forall\, x_i \in \mathcal{D}^c\\
\frac{\delta S[P,Q]}{\delta P(x_i)} &= \lambda_{\mathcal{D}} + \mu_{\mathcal{D}}A(x_i), \quad \forall\, x_i \in \mathcal{D}.
\end{align}

Given the general form of $S$ in (<ref>), the derivative $\delta S/\delta P(x_i)$ could in principle depend on all $2N$ variables,

\begin{equation}
\frac{\delta S[P,Q]}{\delta P(x_i)} = f_i\left(P(x_1),Q(x_1);P(x_2),Q(x_2);\dots;P(x_N),Q(x_N)\right).
\end{equation}

However, this would violate DC1 since any arbitrary change to the constraint over $\mathcal{D}$ would necessarily influence the probabilities in $\mathcal{D}^c$.  In order to prevent the influence of the constraints on the probabilities in the subdomain $\mathcal{D}^c$, we must impose that the variations in a particular subdomain must only be influenced by the probabilies in that subdomain,

\begin{equation}
\frac{\delta S[P,Q]}{\delta P(x_i|\mathcal{D})} = f_i\left(P(x_1|\mathcal{D}),\dots,P(x_N|\mathcal{D});Q(x_1),\dots,Q(x_N)\right).\label{proof2}
\end{equation}

Since the above (<ref>) must hold for arbitrary partitions of $\mathcal{X}$, then in the most restrictive case, $\delta S/\delta P(x_i)$ can only depend on the individual $P(x_i)$ and possibly all the $Q(x_i)$,

\begin{equation}
\frac{\delta S[P,Q]}{\delta P(x_i)} = f_i\left(P(x_i);Q(x_1),Q(x_2),\dots,Q(x_N)\right).\label{proof3}
\end{equation}

We can restrict the form of (<ref>) even further by examining what happens when we make an arbitrary change to the prior outside of $\mathcal{D}$.  Consider that we change $Q(x_j)$ to $Q(x_j) + \delta Q(x_j)$ with $x_j \in \mathcal{D}^c$.  This would suggest that (<ref>) becomes,

\begin{equation}
\frac{\delta S[P,Q]}{\delta P(x_i)} = f_i\left(P(x_i);Q(x_1),\dots,Q(x_j) + \delta Q(x_j),\dots,Q(x_N)\right).\label{proof4}
\end{equation}

This demonstrates that $P(x_i)$ with $x_i \in \mathcal{D}$ could be influenced by information in $\mathcal{D}^c$, unless $\delta S/\delta P(x_i)$ is independent of all prior information in $\mathcal{D}^c$

\begin{equation}
\frac{\delta S[P,Q]}{\delta P(x_i)} = f_i\left(P(x_i);Q(x_1|\mathcal{D}),Q(x_2|\mathcal{D}),\dots,Q(x_N|\mathcal{D})\right).\label{proof5}
\end{equation}

Since again (<ref>) must hold for any arbitrary partition of $\mathcal{X}$, we must have

\begin{equation}
\frac{\delta S[P,Q]}{\delta P(x_i)} = f_i(P(x_i),Q(x_i)), \quad \forall\, x_i \in \mathcal{X}.\label{proof6}
\end{equation}

Integrating over each $x_i \in \mathcal{X}$ we find,

\begin{equation}
S[P,Q] \xrightarrow{\mathbf{DC1}} \sum_{i=1}^{|\mathcal{X}|}F_i\left(P(x_i),Q(x_i)\right),
\end{equation}

for some undetermined function $F$ which could in principle depend on the point $x_i \in \mathcal{X}$.  Taking the continuum limit, $\mathcal{X} \rightarrow \mathbf{X}$, amounts to sending $F_i(P(x_i),Q(x_i))$ to $F(p(x),q(x),x)$ and replacing the sum by an integral,

\begin{equation}
S[\rho,\varphi] = \int dx\, F(p(x),q(x),x),
\end{equation}

which is eq. (<ref>).

DC2 Implementation —  In order to impose coordinate invariance, we can write (<ref>) in a form which explicitly incorporates the density $q(x)$ as a volume measure,

\begin{align}
S[\rho,\varphi] &= \int dx\, q(x)\frac{1}{q(x)}F\left(\frac{p(x)}{q(x)}q(x),q(x),x\right)\nonumber\\
&= \int dx\, q(x)\Phi\left(\frac{p(x)}{q(x)},q(x),x\right),
\end{align}

where we introduced the function $\Phi$,

\begin{equation}
\Phi\left(\frac{p(x)}{q(x)},q(x),x\right) \stackrel{\mathrm{def}}{=} \frac{1}{q(x)}F\left(\frac{p(x)}{q(x)}q(x),q(x),x\right).
\end{equation}

We can restrict the functional form of $\Phi$, and equivalently $F$, by appealing to special cases.  Consider the coordinate transformation,

\begin{equation}
x \rightarrow x' = f(x) \quad \Rightarrow \quad p(x)dx = p'(x')dx' \quad \mathrm{and}\quad q(x)dx = q'(dx').
\end{equation}

This amounts to sending $\Phi$ to

\begin{equation}
\Phi\left(\frac{p'(x')}{q'(x')},q'(x'),x'\right) = \Phi\left(\frac{p(x)}{q(x)},q(x)\gamma(x'),x'\right).
\end{equation}

Consider the special case in which the Jacobian $\gamma(x') = 1$, which gives,

\begin{equation}
\Phi\left(\frac{p'(x')}{q'(x')},q'(x'),x'\right) = \Phi\left(\frac{p(x)}{q(x)},q(x),x'\right).
\end{equation}

To impose DC2 is to require that the right hand side be equivalent to,

\begin{equation}
\Phi\left(\frac{p(x)}{q(x)},q(x),x'\right) \stackrel{\mathbf{DC2}}{=}\Phi\left(\frac{p(x)}{q(x)},q(x),x\right)
\end{equation}

with the original coordinate.  Thus, it must be the case that $\Phi$, and also $F$, be independent of the coordinates $\mathbf{X}$,

\begin{equation}
\Phi \xrightarrow{\mathbf{DC2}} \Phi\left(\frac{p(x)}{q(x)},q(x)\right).
\end{equation}

We can restrict even further by considering a coordinate transformation with an arbitrary Jacobian,

\begin{equation}
\Phi\left(\frac{p'(x')}{q'(x')},q'(x')\right) = \Phi\left(\frac{p(x)}{q(x)},q(x)\gamma(x')\right).
\end{equation}

Again, to impose DC2 amounts to forcing $\Phi\left(\frac{p(x)}{q(x)},q(x)\gamma(x')\right)$ to be equivalent to $\Phi\left(\frac{p(x)}{q(x)},q(x)\right)$ and hence $\Phi$ must also be independent of its second argument,

\begin{equation}
\Phi \xrightarrow{\mathbf{DC2}} \Phi\left(\frac{p(x)}{q(x)}\right).
\end{equation}

As a result of DC2 we have arrived at (<ref>),

\begin{equation}
S[\rho,\varphi] \xrightarrow{\mathbf{DC2}} \int dx\, q(x)\Phi\left(\frac{p(x)}{q(x)}\right).\label{dc2result}
\end{equation}

DC3 Implementation —
Consider the situation in which the universe of discource consists of two subspaces, $\mathbf{X} = \mathbf{X}_1\times\mathbf{X}_2$, which are constrained to be independent, $q(x_1,x_2) \xrightarrow{I_q} q(x_1)q(x_2)$.  Next, consider that new information is given for each subspace individually such that $q(x_1)\rightarrow p(x_1)$ and $q(x_2)\rightarrow p(x_2)$.  Naturally, if the systems were treated individually we would find the posteriors $p(x_1)$ and $p(x_2)$ from the individual priors.  According to DC3 on the otherhand, if we were to treat the systems jointly then we should find the joint posterior $q(x_1)q(x_2)\rightarrow p(x_1)p(x_2)$, since no new information has introducted correlations.

At this point we take some inspiration from the derivation by Vanslette <cit.> and examine several special cases.

Case 1: Updating one subsystem —
Consider the special case in which we constrain the distribution over $\mathbf{X}_1$ to take the known value,

𝐂1      ∫dx_2  p(x_1,x_2) = p(x_1).

The eq. (<ref>) refers to an infinite number of constraints, one for each value of $x_1 \in \mathbf{X}_1$.  Hence, it requires the introduction of a Lagrange multiplier that is a function of $\mathbf{X}_1$ such that by maximizing the entropy,

\begin{equation}
\delta\left[S[\rho,\varphi] - \int dx_1\, \lambda_1(x_1)\left(\int dx_2\, p(x_1,x_2) - p(x_1)\right)\right] = 0,
\end{equation}

we are left with the following differential equation,

\begin{equation}
\frac{\partial \Phi}{\partial p(x_1,x_2)} = \lambda_1(x_1),\label{proof8}
\end{equation}

where we used the fact that for fixed $q(x)$, $\delta S = \int dx\, \partial\Phi/\partial p(x) \delta p(x)$.  The left hand side can be recast in a form similar to (<ref>),

\begin{equation}
\frac{\delta S[\rho,\varphi]}{\delta p(x_1,x_2)} = \frac{\partial \Phi}{\partial p(x_1,x_2)} = \phi(p(x_1,x_2),q(x_1,x_2)),
\end{equation}

where $\phi$ is a function of the posterior and the prior.  To examine the functional form of $\phi$ with respect to (<ref>) we first make use of the information in the prior, $I_q$, – namely, that the prior is independent,

\begin{equation}
\phi \xrightarrow{I_q} \phi(p(x_1,x_2),q(x_1)q(x_2)).
\end{equation}

Now, since the constraint C1 in (<ref>) does not introduce any correlations between $\mathbf{X}_1$ and $\mathbf{X}_2$, then by DC3 we impose that the posterior must be a product also,

\begin{equation}
\phi \xrightarrow{\mathbf{DC3}} \phi(p(x_1)p(x_2),q(x_1)q(x_2)) = \lambda_1(x_1).\label{proof9}
\end{equation}

Besides being silent about any correlations, the constraint C1 in (<ref>) is also silent about any new information concerning subsystem two, and hence by the PMU the distribution over $\mathbf{X}_2$ is not updated,

\begin{equation}
q(x_2) \xrightarrow{\mathbf{PMU}} p(x_2) = q(x_2).
\end{equation}

This gives for the function $\phi$,

\begin{equation}
\phi \xrightarrow{\mathbf{PMU}} \phi(p(x_1)q(x_2),q(x_1)q(x_2)) = \lambda_1(x_1).\label{proof10}
\end{equation}

We can simplify this expression some more by recognizing that the right hand side is independent of $\mathbf{X}_2$.  This should suggest that the left hand side is also independent of $\mathbf{X}_2$, which we've already determined from DC2.  Going back to DC1 however we can argue for something more – that the left hand side of (<ref>) is independent of $q(x_2)$ entirely.  To see this, we are reminded that (<ref>) represents a single equation for each value of $x_1$ and $x_2$.  This means that for a given $x_1 \in \mathbf{X}_1$, both sides of (<ref>) must be independent of arbitrary variations with respect to $x_2$.  These include variations with respect to the prior $q(x_2)$, which, with all other quantities held fixed leads to,

\begin{equation}
\delta \phi = \frac{\partial \phi}{\partial q(x_2)}\frac{\partial q(x_2)}{\partial x_2}\delta x_2 = 0,
\end{equation}

which means that for arbitrary $\delta x_2$ we must have $\partial \phi/\partial q(x_2) = 0$.  Thus, (<ref>) must be independent of $q(x_2)$,

\begin{equation}
\phi \xrightarrow{\mathbf{DC1}}\phi(p(x_1),q(x_1)) = \lambda_1(x_1).\label{proof11}
\end{equation}

In the special case where no constraints are implemented, eq. (<ref>) reduces to,

\begin{equation}
\phi(p(x_1),q(x_1)) \xrightarrow{\lambda=0} \phi(q(x_1),q(x_1)) = 0,\label{noconstraints}
\end{equation}

where the right hand side is zero since there are no constraints and hence no Lagrange multipliers.  Through a similar set of arguments involving a constraint of the form,

𝐂2      ∫dx_1  p(x_1,x_2) = p(x_2).      ,

one can arrive at an equation analagous to (<ref>),

\begin{equation}
\phi \xrightarrow{\mathbf{C2}} \phi(p(x_2),q(x_2)) = \lambda_2(x_2).\label{proof12}
\end{equation}

Since the right hand side of (<ref>) is independent of $x_2$, then

Case 2: Updating both independent subsystems —  Let us now consider the special case in which we update both subsystems according to constraints C1 and C2,

𝐂1,𝐂2      ∫dx_1  p(x_1,x_2) = p(x_2)   and   ∫dx_2 p(x_1,x_2) = p(x_1).

This again constitutes an infinite number of constraints, one for each $x_1$ and $x_2$, and so requires an infinite number of Lagrange multipliers, which we label as $\lambda_1(x_1)$ and $\lambda_2(x_2)$.  Maximizing the relative entropy with respect to the constraints (<ref>),

\begin{align}
\delta\left[\vphantom{\int}S[\rho,\varphi] \right.&- \int dx_1\, \lambda_1(x_1)\left(\int dx_2\, p(x_1,x_2) - p(x_1)\right)\nonumber\\
&\left. - \int dx_2\, \lambda_2(x_2)\left(\int dx_1\, p(x_1,x_2) - p(x_2)\right)\right] = 0,
\end{align}

leads to the differential equation,

\begin{equation}
\frac{\partial \Phi}{\partial p(x_1,x_2)} = \lambda_1(x_1) + \lambda_2(x_2).\label{diffeq}
\end{equation}

Recasting the derivative of $\Phi$ as the function $\phi$ we get the functional equation,

\begin{equation}
\phi(p(x_1,x_2),q(x_1,x_2)) = \lambda_1(x_1) + \lambda_2(x_2).
\end{equation}

We will examine this equation in the same way as with case one and two.  First, using the fact that the prior contains no correlations between $\mathbf{X}_1$ and $\mathbf{X}_2$ we have,

\begin{equation}
\phi \xrightarrow{I_q} \phi(p(x_1,x_2),q(x_1)q(x_2)).
\end{equation}

The constraints C1 and C2 are also silent about and new information concerning correlations between $\mathbf{X}_1$ and $\mathbf{X}_2$ and hence, by DC3 we impose that the posterior must also be independent,

\begin{equation}
\phi\xrightarrow{\mathbf{DC3}} \phi(p(x_1)p(x_2),q(x_1)q(x_2)) = \lambda_1(x_1) + \lambda_2(x_2).\label{proof13}
\end{equation}

Unlike with (<ref>) and (<ref>), both sides of this equation are dependent on variations with respect to the distributions over both $\mathbf{X}_1$ and $\mathbf{X}_2$ so that the left hand side is general.  We can however identify some limiting cases to constrain the right hand side to a more manageable form.  First, consider rewriting (<ref>) in terms of $\lambda_1(x_1)$,

\begin{equation}
\lambda_1(x_1) = \phi(p(x_1)p(x_2),q(x_1)q(x_2)) - \lambda_2(x_2).\label{proof14}
\end{equation}

Since the left hand side of (<ref>) is independent of $x_2$, then the right hand side must be as well.  If we again consider arbitrary variations with respect to $x_2$ in (<ref>) we find,

\begin{equation}
\frac{\partial\phi}{\partial p(x_2)}\frac{\partial p(x_2)}{\partial x_2} + \frac{\partial \phi}{\partial q(x_2)}\frac{\partial q(x_2)}{\partial x_2} = \frac{\partial \lambda_2}{\partial x_2},
\end{equation}

which means that for arbitrary variations of $x_2$, the individual variations of $p(x_2)$, $q(x_2)$ and the Lagrange multiplier $\lambda_2(x_2)$ must cancel out.  Whatever the relationship between $\phi(p(x_1)p(x_2),q(x_1)q(x_2))$ and $\lambda_2(x_2)$ is, it must have the effect of removing dependence not only of $x_2$ but of the distributions $p(x_2)$ and $q(x_2)$ as well.  To see this, consider as in (<ref>) that we were to pick a different prior for $x_2$ which differed by some amount $q(x_2) \rightarrow q'(x_2) = q(x_2) + \delta q_2$.  We would then have that (<ref>) becomes,

\begin{align}
\lambda_1(x_1) \xrightarrow{\delta q_2}\lambda_1'(x_1) &= \phi(p(x_1)p(x_2),q'(x_1)q(x_2)) - \lambda_2(x_2)\nonumber\\
&= \lambda_1(x_1) + \sum_{n=0}^{\infty}\frac{\partial^n\phi}{\partial q_2^n}(\delta q_2)^n \neq \lambda_1(x_1).
\end{align}

Thus the right hand side of (<ref>) must be independent of $q(x_2)$.  For similar reasons it must also be independent of $p(x_2)$.  Thus, $\lambda_1(x_1)$ must be equal to some unknown function of $p(x_1)$ and $q(x_1)$ only,

\begin{equation}
\lambda_1(x_1) \rightarrow \psi_1(p(x_1),q(x_1)).
\end{equation}

A similar argument can be made for the subsystem $\mathbf{X}_2$ such that,

\begin{equation}
\lambda_2(x_2) \rightarrow \psi_2(p(x_2),q(x_2)).
\end{equation}

This gives (<ref>) in the form of a functional equation for $p(x_1)$, $p(x_2)$, $q(x_1)$ and $q(x_2)$,

\begin{equation}
\phi(p(x_1)p(x_2),q(x_1)q(x_2)) = \psi_1(p(x_1),q(x_1)) + \psi_2(p(x_2),q(x_2)).
\end{equation}

Which is a form of Pexider's equation <cit.> which is itself a special case of Cauchy's equation.  To construct a solution, first consider the rescaling of the probabilities via the logarithm,

\begin{equation}
p(x) \rightarrow \ell_p(x) = \log p(x) \quad \mathrm{and} \quad q(x) \rightarrow \ell_q(x) = \log q(x).
\end{equation}

Then, by scaling $\phi$, $\psi_1$ and $\psi_2$ via the exponential function we find,

\begin{align}
\phi'(\ell_p(x_1) + \ell_p(x_2),\ell_q(x_1) + \ell_q(x_2)) &= \psi_1'(\ell_p(x_1),\ell_q(x_1))\nonumber\\
&+ \psi_2'(\ell_p(x_2),\ell_q(x_2)),\label{proof15}
\end{align}

where $\phi' = \phi \circ \exp$, $\psi_1' = \psi_1\circ \exp$ and $\psi_2' = \psi_2\circ \exp$.  This brings us to the following theorem from Pexider <cit.>,

The most general solution to the many-variable Pexider's equation,

\begin{equation}
f(x_1 + y_1,x_2 + y_2,\dots,x_n + y_n) = g(x_1,x_2,\dots,x_n) + h(y_1,y_2,\dots,y_n),\label{pexider1}
\end{equation}

\begin{equation}
f(t) = \xi(t) + a + b, \quad g(t) = \xi(t) + a \quad \mathrm{and} \quad h(t) = \xi(t) + b,\label{pexider2}
\end{equation}

where $\xi(t)$ is an arbitrary solution of Cauchy's equation and $a$ and $b$ are constants.

A proof of [pexider]theorem 3.1 is given in the [proofa1]appendix.  From (<ref>) and (<ref>) it follows that (<ref>) can be written as,

\begin{align}
\xi(\ell_p(x_1) + \ell_p(x_2),\ell_q(x_1) + \ell_q(x_2)) &= \xi(\ell_p(x_1),\ell_q(x_1))\nonumber\\
&+ \xi(\ell_p(x_2),\ell_q(x_2)),\label{proof16}
\end{align}

which is Cauchy's basic equation in two variables <cit.>.  The general solution for a many-variable version of the basic Cauchy equation is given from the following theorem,

The general solution of the functional equation,

\begin{equation}
f(x_1 + y_1,x_2 + y_2,\dots,x_n + y_n) = f(x_1,x_2,\dots,x_n) + f(y_1,y_2,\dots,y_n),
\end{equation}

with $f(0,\dots,0,x_k,0,\dots,0)$ continuous for $k = \{1,2,\dots,n\}$ is

\begin{equation}
f(x_1,x_2,\dots,x_n) = c_1x_1 + c_2x_2 + \dots + c_nx_n,
\end{equation}

where $c_k \in \mathbb{R}$.

A proof of [azcel2]theorem 3.2 is also given in the [proofa2]appendix.  From here, the solution for (<ref>) is given by,

\begin{align}
\xi(\ell_p(x_1) + \ell_p(x_2),\ell_q(x_1) + \ell_q(x_2)) &= c_p(\ell_p(x_1) + \ell_p(x_2))\nonumber\\
&+ c_q(\ell_q(x_1) + \ell_q(x_2)).
\end{align}

Switching back to the original variables and substituting $\xi$ for the original $\phi$ we have,

\begin{align}
\phi(p(x_1)p(x_2),q(x_1)q(x_2)) &= c_p(\ell_p(x_1) + \ell_p(x_2))\nonumber\\
&+ c_q(\ell_q(x_1) + \ell_q(x_2)) + c_{\xi}\nonumber\\
&= c_p\log(p(x_1)p(x_2))\nonumber\\
&+ c_q\log(q(x_1)q(x_2)) + c_{\xi},
\end{align}

where $c_{\xi}$ is an arbitrary constant.  To fix $c_{\xi}$ and the other two constants, $c_p$ and $c_q$, consider the special case for when there are no constraints so that $p(x_1)p(x_2) = q(x_1)q(x_2)$.  Then we must have according to (<ref>),

\begin{equation}
\phi(q(x_1)q(x_2),q(x_1)q(x_2)) = (c_p + c_q)\log(q(x_1)q(x_2)) + c_{\xi} = 0,
\end{equation}

which means that in general,

\begin{equation}
\frac{c_{\xi}}{(c_p + c_q)} = -\log(q(x_1)q(x_2)).
\end{equation}

The left hand side however is a constant, independent of the argument of the logarithm on the right hand side.  The only reasonable solution is to set $c_{\xi} = 0$, which leads to,

\begin{equation}
\phi(q(x_1)q(x_2),q(x_1)q(x_2)) = (c_p + c_q)\log(q(x_1)q(x_2)) = 0,
\end{equation}

so that $c_q = -c_p$.  Thus, for general $p(x_1,x_2)$ and $q(x_1,x_2)$ we have

\begin{equation}
\phi(p(x_1,x_2),q(x_1,x_2)) = |C|\log\left(\frac{p(x_1,x_2)}{q(x_1,x_2)}\right),
\end{equation}

where $|C|$ is an overall constant.

The general solution —
We can rewrite the expression for the differential equation (<ref>) using the solution to the functional equation for an arbitrary prior $\varphi$ and posterior distribution $\rho$,

\begin{equation}
\frac{\delta S[\rho,\varphi]}{\delta \rho} = |C|\log\left(\frac{p(x)}{q(x)}\right).
\end{equation}

This equation can be integrated with respect to $p(x)$ to give,

\begin{equation}
S[\rho,\varphi] = |C|\int dx\, p(x)\log\left(\frac{p(x)}{q(x)}\right) + |B|\int dx\, p(x) + |A|, \label{gensol}
\end{equation}

where $|A|$ and $|B|$ are constants.  Since $S[p,q]$ is designed for ranking distributions $p(x)$ with respect to the prior $q(x)$, the overall constant $|A|$ can be set to zero, since it only introduces an overall scale which is independent of the probabilities.  Since a normalization constraint will always be imposed, the constant $|B|$ can simply be absorbed into whatever the normalization is set to.  Thus the term $|B|\int dx\, p(x)$ can also be dropped.  This leaves the relative entropy in the form,

\begin{equation}
S[\rho,\varphi] = |C|\int dx\, p(x)\log\left(\frac{p(x)}{q(x)}\right).
\end{equation}

Comparison with eq. (<ref>) shows that the function $\Phi$ is equivalent to[One can find this solution by simply differentiating (<ref>) with respect to $x_1$ and $x_2$ which leads to the equation $y\Phi'''(y) + \Phi''(y) = 0$ where the primes denote the derivative with respect to $y$.  The solution to this differential equation is $\Phi(y) = ay\log y + by + c$, which is equivalent to (<ref>) <cit.>.],

\begin{equation}
\Phi\left(\frac{p(x)}{q(x)}\right) = \frac{p(x)}{q(x)}\log\left(\frac{p(x)}{q(x)}\right).
\end{equation}

The arbitrary constant $|C|$ can be set to $\pm 1$ for convenience depending on whether one wishes to maximize or minimize the relative entropy.  Since $x \log (ax)$ for $a > 0$ is a strictly convex function, it has a unique minimum.  This means that $-x\log(ax)$ is a strictly concave function with a unique maximum.  Thus, picking the convention to be that of maximizing the relative entropy, we choose $|C| = -1$ to get the final functional form,

\begin{equation}
S[\rho,\varphi] = -\int dx\, p(x)\log\left(\frac{p(x)}{q(x)}\right).
\end{equation}

Bayes' rule as a special case —
It was first shown Williams <cit.> that Bayes' rule can be derived as a special case of MaxEnt, however the result went unappreciated.  Caticha and Giffin <cit.> demonstrated the same result from the perspective of the ME method by associating data with constraint information, which ultimately allows for many generalizations.  The problem is as follows: consider that you wish to conduct inference on a set of parameters $\theta \in \Theta$ so that you can update a prior distribution $q(\theta)$ to the appropriate posterior $p(\theta)$ when information comes in the form of data $x \in \mathbf{X}$.  The relation between $\Theta$ and $\mathbf{X}$ is determined through a likelihood function $q(x|\theta)$ which is also given as prior information.

The key insight from <cit.> is to recognize that before the data $x \in \mathbf{X}$ is specified, its value is unknown and hence a generic inference requires that the universe of discourse be the joint space $\Theta \times \mathbf{X}$ and not just the parameter space $\Theta$.  It is therefore the joint distribution $q(x,\theta)$ which needs to be updated and not simply $q(\theta)$.  With this in mind, consider that we now collect data which takes the value $x' \in \mathbf{X}$.  Then, the posterior $p(x)$ is constrained to reflect the known value,

\begin{equation}
p(x) = \int d\theta\, p(x,\theta) = \delta(x - x').\label{bayesconstraint}
\end{equation}

While this data constrains the form of $p(x,\theta)$, it is not sufficient to determine $p(\theta|x)$ since any choice will satisfy,

\begin{equation}
p(x,\theta) = p(x)p(\theta|x) = \delta(x - x')p(\theta|x).
\end{equation}

To determine the full posterior, we maximize the relative entropy with respect to the constraint (<ref>)

\begin{align}
0 = \delta \left[\vphantom{\int}S[\rho,\varphi] \right.&- \alpha\left(\int dxd\theta\,p(x,\theta) - 1\right)\nonumber\\
&\left.- \int dx\, \lambda(x)\left(\int d\theta\, p(x,\theta) - \delta(x - x')\right)\right],
\end{align}

which leads to the joint posterior

\begin{equation}
p(x,\theta) = q(x,\theta)\frac{e^{\lambda(x)}}{Z},
\end{equation}

where $Z = \int dxd\theta\, q(x,\theta)\exp(\lambda(x))$ is a normalization factor.  The Lagrange multiplier $\lambda(x)$ can be determined from (<ref>),

\begin{equation}
\int d\theta\, p(x,\theta) = \int d\theta\,q(x)q(\theta|x)\frac{e^{\lambda(x)}}{Z} = q(x)\frac{e^{\lambda(x)}}{Z} = \delta(x - x'),
\end{equation}

so that the joint posterior can be written

\begin{equation}
p(x,\theta) = q(x,\theta)\frac{\delta(x - x')}{q(x)} = q(\theta|x)\delta(x - x').
\end{equation}

Thus, the posterior $p(\theta)$ is,

\begin{equation}
p(\theta) = \int dx\, p(x,\theta) = q(\theta|x') = q(\theta)\frac{q(x'|\theta)}{q(x')},
\end{equation}

which is Bayes' rule.  Further generalizations to Bayes' rule are also discussed in <cit.>.

§.§ Summary

In this chapter we designed an entropic inference for the purposes of updating our beliefs in the presence of new information that comes in the form of constraints.  By imposing certain design criteria which adhere to the principle of minimal updating, we were able to use eliminative induction to find a general functional form of the relative entropy ([section42]Section 4.2).  We also designed a tool for ranking the correlations present within a probability distribution called the total correlation.  This was achieved by a similar set of design criteria used for designing the relative entropy with the exception of split coordinate invariance ([corollary431]Corollary 4.3.1).  The results of [section43]Section 4.3 produce an entire family of correlation quantifiers including the mutual information ([section433]Section 4.3.3).  In the next section we will employ a practical use of the mutual information in machine learning.

§ RULES FOR DEDUCTIVE INFERENCE

In this Appendix we will construct the simple system $\mathcal{D} = \left\{\mathcal{A},\Omega,\mathcal{I},\mathcal{Z},\mathcal{T}\right\}$, where $\mathcal{A}$ is a set of well-formed formulas called [propositions]propositions, $\Omega$ is a collection of disjoint subsets of [connectives]logical connectives (relations), $\mathcal{I}$ is a set of distinguished propositions called [axioms]axioms, which include the three laws of thought, and $\mathcal{Z}$ is a set of [inferencerules]inference rules concerning the logical connectives.  The object $\mathcal{T}$ contains any additional structures we choose to add to our framework.  To build the system $\mathcal{D}$, we will first discuss the construction of $\mathcal{A}$.

§.§ The space of propositions $\mathcal{A}$

The universe of discourse, or well formed formulas of $\Sigma^*$, of interest will be the space of propositions, which we label as $\mathcal{A}$.  An element of $\mathcal{A}$ is a single proposition[We recognize that often in foundational mathematics, the calculus of logic is developed before any notion of sets is constructed.  Here we take some liberties with this and assume that the definition of the epsilon relation $\in$ and the axioms of ZF set theory are understood in advance.], $a \in \mathcal{A}$, which is defined as,

A proposition, $a \in \mathcal{A}$, is a statement, or variable, which is either true or false (labeled $\mathbf{T}$ and $\mathbf{F}$ respectively).  The set $\mathcal{A}$ is defined as the countable collection,

\begin{equation}
\mathcal{A} \stackrel{\mathrm{def}}{=} \left\{\frac{}{}a\,\middle|\, (\nu(a)\Leftrightarrow \mathbf{T})\vee (\nu(a) \Leftrightarrow \mathbf{F})\right\}.
\end{equation}

As pointed out by Quine <cit.>, the “meaning” of propositions can be vague even if their “truth” value as it relates to the real world is well defined.  At this level of discourse, we only assume that whatever propositions are, they have a well defined truth value.  In this sense, all theorems are propositions in themselves, however statements which fail to be valid for reasons of incompleteness are not propositions by definition.

The definition [proposition]above establishes the semantics as consisting of a bivalent, two-valued or Boolean logic <cit.>.  This fact is often imposed as a principle of bivalence <cit.>.  While there are only two options, true or false, this principle is not necessarily the same as the [lawsoflogic]law of excluded middle. One can choose to impose the law of excluded middle while excluding the principle of bivalence[For example, in a three valued logic where one has three possible truth values, $\mathbf{A},\mathbf{B}$ and $\mathbf{C}$, one no longer has a principle of bivalence however one can always still impose the law of excluded middle by demanding that all propositions must take one of the three possible truth values, i.e. if a proposition is not $\mathbf{A}$ or $\mathbf{B}$ then it must be $\mathbf{C}$.] or vice versa <cit.>.

The function $\nu$ is called a valuation, which is a map from the set $\mathcal{A}$ to its truth value.

A valuation $\nu:\mathcal{A} \rightarrow \left\{\mathbf{T},\mathbf{F}\right\}$ is a function which returns the truth value of a proposition.

As was discussed in the preface, the meaning of “truth” is still a subject of debate among philosophers.  Here we simply identify that true and false are two mutually exclusive properties of propositions, but we do not offer any explanation as to how any particular proposition obtains its “truth” value, which would require a metatheory.  Instead we take the truth values of propositions as given and only worry about developing the calculus for how to manipulate them.

For any particular universe of discourse $\mathcal{A}$, we can collect all of the associated truth values for each $a \in \mathcal{A}$ into a set which is called a world.  A world is simply an identification of which propositions $a \in \mathcal{A}$ are true, and which are false[There are several ways in which to define a world $w$ <cit.>.  One is the specification described here in which $w$ is a collection of propositions and their associated truth values.  Another possibility is to simply have $w$ contain only propositions which are true, i.e. if $a$ is true in $w$ then $a \in w$, and if $b$ is false in $w$ then $\neg b \in w$.].  We require in the very least that any such world be internally consistent, i.e. that it not contain contradictions.  As we will see in conditional logic, in order to address propositions about the future one must extend classical logic to include what are typically called possible worlds.

A world $w$, is a collection of truth values <cit.> associated to an underlying set of propositions $\mathcal{A}$.  The collection of all possible worlds $w$ over the set of propositions $\mathcal{A}$ is called $W$.

For each $a \in \mathcal{A}$, a world $w \in W$ records a truth value, so that $|w| = |\mathcal{A}|$, i.e. the number of elements in $w$ is the same as in $\mathcal{A}$.  Instead of defining $w$ in terms of truth values, one could instead define it as the collection of propositions from $\mathcal{A}$ that are true in $w$.  Due to the law of excluded middle, this representation gives $|w| = |\mathcal{A}|/2$.  Some uses of possible worlds allow for worlds $w \in W$ to only have truth values for a subset of $\mathcal{A}$, however we will not consider that direction here.

Possible worlds introduce an element of subjectivity into logical inference, since different agents can each assign a possible world as a candidate for the actual world.  The actual world, $\tilde{w} \in W$, is the one that corresponds to ultimate reality and hence contains in it solely objective facts.  Some philosophers take an extreme position on the existence of possible worlds.  David Lewis <cit.> in particular pushed for an agenda of modal realism, in which each possible world is just as real as the actual world.  This is similar to positions held by many of the many worlds approach to quantum mechanics crowd[Theories such as quantum mechanics present a significant challenge to traditional logic since they describe a world which is necessarily indeterministic, i.e. the truth value of propositions about the future are not only unknown but also unknowable according to the theory.  Theories which attempt to address these issues fall under the category of quantum logic, a subject first initiated by von Neumann <cit.>.] <cit.>.  We do not adopt either of these positions here, since there must exist some ultimate reality, and that ultimate reality is what should be associated to the actual world.

The definition of a world allows us to generalize the valuation function (<ref>) to one which takes a world as an argument,

Let $w$ be a possible world over the propositions $\mathcal{A}$.  The valuation of a statement $a \in \mathcal{A}$ with respect to the world $w$ is

\begin{equation}
v:\mathcal{A}\times W \rightarrow \left\{\mathbf{T},\mathbf{F}\right\}.\label{truthworld}
\end{equation}

Semantically, if $a$ is true in $w$, then

\begin{equation}
w \vDash a,
\end{equation}

or, “$w$ proves $a$.”  Another notation is to write the semantic consequence with the subscript of the world in which it is true, $\vDash_w a$.  The use of possible worlds will become important when we discuss modal logic in [modallogic]Section 8

A world forms an interconnected web of beliefs of complete knowledge <cit.>, which, as a rule of thumb, must necessarily be internally consistent.  Thus, any consistent world must adhere to the laws of logic.  If a possible world $w \in W$ contains a contradiction, then by the [explosion]principle of explosion, every proposition in it is necessarily true.  This world is sometimes labeled $\lambda \in W$.

We can form an equivalence between subsets of $W$ and the space of propositions $\mathcal{A}$.  Consider labeling a subset of $W$ in which $a$ is true as[We say that the subset of worlds $W_a \subseteq W$ in which $a$ is true is generated by the proposition $a$.],

\begin{equation}
W_a \stackrel{\mathrm{def}}{=} \left\{\frac{}{}w\,\middle|\, \nu(a,w)\right\}\subseteq W.\label{setofsubsetsworlds}
\end{equation}

We then identify the set of subsets of (<ref>) as $\mathcal{W} = \left\{W_a \in 2^{W}\right\}$, which is isomorphic[One can show that each set $W_a \in \mathcal{W}$ uniquely corresponds to a proposition $a \in \mathcal{A}$ so that there exists some bijection $f:\mathcal{A}\rightarrow\mathcal{W}$.] to the space of propositions $\mathcal{A}$,

\begin{equation}
\mathcal{W} \cong_{\mathrm{set}} \mathcal{A}.
\end{equation}

§.§ Logical connectives $\Omega$

One can define maps from one or many propositions to another.  These are called $n$-ary connectives, of which the unary and binary ones will be of importance.  The set of $n$-ary connectives can be written as the union of $n$ disjoint subsets,

\begin{equation}
\Omega = \bigcup_{i\geq 0}\Omega_i = \Omega_0\cup\Omega_1\cup\Omega_2\cup\dots\cup\Omega_k\cup\dots\cup\Omega_n.
\end{equation}

The sets $\Omega_1$ and $\Omega_2$ are the unary and binary connectives respectively.  Unary connectives are defined as,

A unary connective, $u\in\Omega_1:\mathcal{A}\rightarrow\mathcal{A}$, is a map from a single proposition to another, i.e. $a\mapsto u(a) \in \mathcal{A}$.

Since the number of possible truth values is two, there are $2^2 = 4$ unique unary connectives on proposition space.  They are called, tautology, contradiction, identity and negation.  The tautology ($\top$) and contradiction ($\bot \Leftrightarrow \neg \top$)[Often times the tautology and contradiction connectives are defined as elements of $\Omega_0$.] connectives map every proposition to either true or false[In other words, no matter what the propositions $a$ is, $\forall\, a : \top a \Leftrightarrow \mathbf{T}$ and $\forall\, a : \bot a \Leftrightarrow \mathbf{F}$.], regardless of their actual truth value.  The identity connective ($\mathrm{Id}$) simply returns the original proposition (i.e. $\forall\, a : \mathrm{Id}(a) \Leftrightarrow a$), and the negation map ($\neg$) returns the complement of the proposition (i.e. $\forall\, a : \neg(a) \Leftrightarrow \neg a$) so that $\nu(\neg a) \Leftrightarrow \neg\nu(a)$.  A convenient construction for determining the truth values of $n$-ary connectives are truth tables.  The following is the truth table for the unary connectives,

$a$     $\top$     $\bot$     $\mathrm{Id}$     $\neg$

$\tru$     $\tru$     $\fals$     $\tru$     $\fals$

$\fals$     $\tru$     $\fals$     $\fals$     $\tru$

The four unary operators acting on a proposition $a$.

The binary connectives are defined similarly,

A binary connective, $w\in\Omega_2:\mathcal{A}\times\mathcal{A}\rightarrow \mathcal{A}$, is a map from two propositions to another, i.e. $(a,b) \mapsto w(a,b) \in \mathcal{A}$.