system, so that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$
satisfies the geometric locus of the decision boundary
$k_{\mathbf{s}}\boldsymbol{\kappa}+$ $\boldsymbol{\kappa}_{0}=0$ of the system
in terms of a critical minimum eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and a minimum expected
risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$,
such that the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
exhibited by the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ are both
regulated by the _total_ _value_ of the scale factors $\psi_{1i\ast}$ and
$\psi_{2i\ast}$—for the components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
of the principal eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$ and the inverted joint covariance matrix
$\mathbf{Q}^{-1}$ associated with associated with the pair of random quadratic
forms $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$, at which point values
of the scale factors $\psi_{1i\ast}$ and $\psi_{2i\ast}$ are functions of
covariance and distribution information for all of the extreme vectors
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$—in a given
collection of extreme vectors—relative to the covariance and distribution
information represented by the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$ for a given collection
$\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}$ of feature vectors
$\mathbf{x}_{i}$.

We use the KKT condition in (12.7) and the theorem of Karush, Kuhn, and Tucker
to demonstrate that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}$ $=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ of
any given minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
satisfies the geometric locus of the decision boundary
$k_{\mathbf{s}}\boldsymbol{\kappa}+$ $\boldsymbol{\kappa}_{0}=0$ of the system
in terms of its critical eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and its minimum expected
risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$,
such that the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
are regulated by the _total value_ of the scale factors $\psi_{1i\ast}$ and
$\psi_{2i\ast}$ for the components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
of the principal eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$ and the inverted joint covariance matrix
$\mathbf{Q}^{-1}$ associated with the pair of random quadratic forms
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$.

Correspondingly, we demonstrate that the elements
$\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
and the eigenvalues $\lambda_{N}\leq\mathbf{\ldots}\leq\lambda_{1}$ of the
joint covariance matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ in the Wolfe-dual
eigenenergy functional
$\max\Xi_{\boldsymbol{\psi}}\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\frac{\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}}{2}$
of (12.9), wherein $\boldsymbol{\psi}^{T}\mathbf{y}=0$ and $\psi_{i\ast}>0$,
along with the scale factors $\psi_{1i\ast}$ and $\psi_{2i\ast}$ for the
components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
of the principal eigenvector $\boldsymbol{\psi}_{\max}$ of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$

$\boldsymbol{\psi}_{\max}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}+\sum\nolimits_{i=1}^{l_{2}}\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$

are statistically interconnected with the components
$\psi_{1_{i_{\ast}}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i_{\ast}}}k_{\mathbf{x}_{2_{i\ast}}}$ of the geometric locus of the
novel principal eigenaxis

$\boldsymbol{\kappa}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i_{\ast}}}k_{\mathbf{x}_{1_{i\ast}}}-\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i_{\ast}}}k_{\mathbf{x}_{2_{i\ast}}}$

in such a manner that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}$ $=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ is
the principal part of an equivalent representation of the pair of random
quadratic forms $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$, so that the geometric
locus of the novel principal eigenaxis $\boldsymbol{\kappa}$
$=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ satisfies the geometric
locus of the decision boundary $k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}=0$ of a minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
in terms of a critical minimum eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and a minimum expected
risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
in the following manner

$\displaystyle\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$
$\displaystyle=\sum\nolimits_{i=1}^{l_{1}}\psi_{1i\ast}\left(1-\xi_{i}\right)+\sum\nolimits_{i=1}^{l_{2}}\psi_{2i\ast}\left(1-\xi_{i}\right)$
$\displaystyle=\boldsymbol{\psi}_{\max}-\left(\sum\nolimits_{i=1}^{l_{1}}\xi_{i}\psi_{1i\ast}+\sum\nolimits_{i=1}^{l_{2}}\xi_{i}\psi_{2i\ast}\right)\text{,}$

at which point the total value of the scale factors $\psi_{1i\ast}$ and
$\psi_{2i\ast}$ for the components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
of the principal eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and the inverted joint
covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$ regulates the total
allowed eigenenergy $\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and
the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
exhibited by the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}$, wherein the regularization parameters $\xi_{i}=\xi\ll
1$ determine negligible constraints.

###  12.10 A Vector-valued Cost Function

We demonstrate that the Wolfe-dual eigenenergy functional uses a _vector-
valued cost function_ to find the principal eigenvector
$\boldsymbol{\psi}_{\max}$ of the joint covariance matrix $\mathbf{Q}$ of the
random quadratic form $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ in
(12.9), so that the Wolfe-dual eigenenergy functional

$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2\text{,}$

wherein $\boldsymbol{\psi}^{T}\mathbf{y}=0$ and $\psi_{i\ast}>0$, is
_maximized_ by the largest eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$

$\mathbf{Q}\boldsymbol{\psi}_{\max}=\lambda_{1}\boldsymbol{\psi}_{\max}\text{,}$

at which point the random quadratic form
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}$, plus the
total allowed eigenenergy
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
and the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}\right)$
exhibited by the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}$ $=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$
_jointly_ reach their _minimum_ values.

###  12.11 A Note on Scalar-valued Cost Functions

Bayes’ decision rule and support vector learning machines both _use scalar-
valued cost functions_ to _find_ decision functions or indicator functions of
binary classification systems.

By the analysis presented in this treatise, it will be seen that any given
statistical method or machine learning algorithm that employs _scalar-valued
cost functions_ —for the fundamental problem of finding discriminant functions
of minimum risk binary classification systems—grossly _oversimplifies_ the
_complexity_ of the problem.

We now devise vector expressions for the primal and the Wolfe-dual novel
principal eigenaxis.

###  12.12 The Primal Novel Principal Eigenaxis

By the KKT conditions in (12.3) and (12.6), it follows that the geometrical
and statistical structure of the geometric locus of the primal novel principal
eigenaxis $\boldsymbol{\kappa}$ is determined by the vector expression

(12.10)
$\boldsymbol{\kappa}=\sum\nolimits_{i=1}^{N}y_{i}\psi_{i}k_{\mathbf{x}_{i}}\text{,}$

where $\psi_{i}\geq 0$, wherein $\psi_{i}>0$ by the KKT condition of
complementary slackness. Reproducing kernels $k_{\mathbf{x}_{i}}$ for feature
vectors $\mathbf{x}_{i}$ that are correlated with scaled unit reproducing
kernels for feature vectors
$\psi_{i}\frac{k_{\mathbf{x}_{i}}}{\left\|k_{\mathbf{x}_{i}}\right\|}$ that
have non-zero magnitudes $\psi_{i}>0$ are called extreme vectors.

Let the scaled extreme vectors that belong to class $\omega_{1}$ and class
$\omega_{2}$ be denoted by $\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ respectively, so that
$\psi_{1_{i\ast}}$ is the scale factor for the extreme vector
$k_{\mathbf{x}_{1_{i\ast}}}$ and $\psi_{2_{i\ast}}$ is the scale factor for
the extreme vector $k_{\mathbf{x}_{2_{i\ast}}}$. Let there be $l_{1}$ scaled
extreme vectors
$\left\\{\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}\right\\}_{i=1}^{l_{1}}$
that belong to class $\omega_{1}$ and $l_{2}$ scaled extreme vectors
$\left\\{\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}\right\\}_{i=1}^{l_{2}}$
that belong to class $\omega_{2}$.

Using (12.10), class membership statistics and the assumptions outlined above,
it follows that the structure of the geometric locus of the primal novel
principal eigenaxis $\boldsymbol{\kappa}$ is determined by the vector
difference of a pair of directed line segment estimates

(12.11) $\displaystyle\boldsymbol{\kappa}$
$\displaystyle=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}-\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$
$\displaystyle=\boldsymbol{\kappa}_{1}{\large-}\boldsymbol{\kappa}_{2}\text{,}$

at which point the primal novel principal eigenaxis $\boldsymbol{\kappa}$ is a
locus of principal eigenaxis components
$\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$, where $\boldsymbol{\kappa}_{1}$
and $\boldsymbol{\kappa}_{2}$ denote the sides of $\boldsymbol{\kappa}$, such
that the side $\boldsymbol{\kappa}_{1}$ is determined by the vector expression
$\boldsymbol{\kappa}_{1}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$,
the side $\boldsymbol{\kappa}_{2}$ is determined by the vector expression
$\boldsymbol{\kappa}_{2}=\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$,
and the geometric locus of the primal novel eigenaxis $\boldsymbol{\kappa}$ is
determined by the vector difference of side $\boldsymbol{\kappa}_{1}$ and side
$\boldsymbol{\kappa}_{2}$.

###  12.13 The Wolfe-dual Novel Principal Eigenaxis

Given the Wolfe-dual eigenenergy functional
$\max\Xi_{\boldsymbol{\psi}}\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\frac{\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}}{2}$
in (12.9), wherein $\boldsymbol{\psi}^{T}\mathbf{y}=0$ and $\psi_{i\ast}>0$,
and the guarantee provided by the strong duality theorem [24, 47, 48, 54], it
follows that the principal eigenaxis components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on the geometric locus of the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}$ are symmetrically and equivalently related to the
principal eigenaxis components $\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$
and $\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ on the geometric locus of the
primal novel principal eigenaxis $\boldsymbol{\kappa}$, so that the
geometrical and statistical structure of the geometric locus of the Wolfe-dual
novel principal eigenaxis $\boldsymbol{\psi}$ is determined by the vector sum
of a pair of directed line segment estimates

(12.12) $\displaystyle\boldsymbol{\psi}$
$\displaystyle=\sum\nolimits_{i=1}^{l_{1}}\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}+\sum\nolimits_{i=1}^{l_{2}}\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
$\displaystyle=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}\text{,}$

at which point the Wolfe-dual novel principal eigenaxis $\boldsymbol{\psi}$ is
a locus of principal eigenaxis components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$,
where $\boldsymbol{\psi}_{1}$ and $\boldsymbol{\psi}_{2}$ denote the sides of
$\boldsymbol{\psi}$, such that the side $\boldsymbol{\psi}_{1}$ is determined
by the vector expression
$\boldsymbol{\psi}_{1}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$,
the side $\boldsymbol{\psi}_{2}$ is determined by the vector expression
$\boldsymbol{\psi}_{2}=\sum\nolimits_{i=1}^{l_{2}}\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$,
and the geometric locus of $\boldsymbol{\psi}$ is determined by the vector sum
of side $\boldsymbol{\psi}_{1}$ and side $\boldsymbol{\psi}_{2}$.

We now turn our attention to the eigenstructure content of the joint
covariance matrix $\mathbf{Q}$—of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$—in the Wolfe-dual
eigenenergy functional
$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2$
in (12.9).

##  13 Eigenstructure Content of Data Matrices

The machine learning algorithm being investigated is also solving a system
identification problem, such that the overall statistical structure and
behavior and properties of a minimum risk binary classification system are
determined by transforming a collection of training data into a data-driven
mathematical model that represents fundamental aspects of the system.

Indeed, we will demonstrate that the machine learning algorithm being
investigated _statistically pre-wires the important generalizations_ for a
minimum risk binary classification system $k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
_within the geometric locus of the novel principal eigenaxis_
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ of the
system.

We now demonstrate that any given minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
based on eigenstructure deficiencies is _ill-posed_ —such that the geometric
locus of the novel principal eigenaxis $\boldsymbol{\kappa}$ of the system is
not characteristic of the system (is _not unique_) and is _unstable_ —at which
point the total allowed _eigenenergy_
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
exhibited by the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ is
_maximized_.

Recall that solving a system identification problem involves solving an
inverse problem, such that a collection of training data are used to infer the
values of the parameters characterizing a given system. Thus, the machine
learning algorithm being investigated is also _solving_ a _well-posed_ inverse
problem—so that solutions obtained by the algorithm are both _unique_ and
_stable_.

Given the novel principal eigen-coordinate transform method expressed by
Theorem 7.1 and Corollary 7.2, we know that the essential information content
of any given training data set—that is used to find a geometric locus of a
novel principal eigenaxis $\boldsymbol{\kappa}$—is contained within the
eigenstructures of the data set, such that all of the individual pattern
vectors ‘add up’ to a complete and sufficient eigenstructure.

We also recognize that Bellman’s “curse of dimensionality” is concerned with
the fundamental problem of parameter estimates that are based on insufficient
eigenstructures [3]. We show that solutions obtained by the machine learning
algorithm being examined—which are based on eigenstructure deficiencies—are
generally ill-posed and ill-conditioned, and must be constrained in some
manner.

###  13.1 Insufficient Learning Capacity

A training set of $N$ pattern vectors with dimension $d$ has at most $d$ non-
zero eigenvalues, given that $N>d$. Accordingly, the Wolfe-dual principal
eigenspace that is associated with the Wolfe-dual eigenenergy functional
$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2$
in (12.9) is spanned by $d$ or fewer eigenfunctions [67, 65]. Thus, the
machine learning algorithm being examined has insufficient “learning capacity”
whenever $N>d$.

Eigenstructures are an inherent part of machine learning algorithms. Moreover,
machine learning solutions with eigenstructure deficiencies are generally ill-
posed and ill-conditioned, and must be constrained in some manner [23, 29, 30,
31, 42, 43, 86].

###  13.2 Complete Eigenstructures

In order for the machine learning algorithm being examined to find
discriminant functions of minimum risk binary classification systems, it is
vital that all of the $N$ feature vectors in any given collection of training
data

$\left(\mathbf{x}_{1}\mathbf{,}y_{1}\right),\ldots,\left(\mathbf{x}_{N}\mathbf{,}y_{N}\right)\in\mathbb{R}^{d}\times
Y,Y=\left\\{\pm 1\right\\}$

_boil down_ to a _complete eigenstructure_ of the joint covariance matrix
$\mathbf{Q}$—of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$—in the Wolfe-dual
eigenenergy functional
$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2$
in (12.9).

To see this, consider the Cayley–Hamilton theorem [39, 52], which states that
the roots $p\left(\lambda\right)=0$ of the characteristic polynomial
$p\left(\lambda\right)$ of the joint covariance matrix $\mathbf{Q}$

$\det\left(\begin{bmatrix}\left\|k_{\mathbf{x}_{1}}\right\|\left\|k_{\mathbf{x}_{1}}\right\|\cos\theta_{k_{\mathbf{x_{1}}}k_{\mathbf{x}_{1}}}-\lambda_{1}&\cdots&-\left\|k_{\mathbf{x}_{1}}\right\|\left\|k_{\mathbf{x}_{N}}\right\|\cos\theta_{k_{\mathbf{x}_{1}}k_{\mathbf{x}_{N}}}\\\
\vdots&\ddots&\vdots\\\
-\left\|k_{\mathbf{x}_{N}}\right\|\left\|k_{\mathbf{x}_{1}}\right\|\cos\theta_{k_{\mathbf{x}_{N}}k_{\mathbf{x}_{1}}}&\cdots&\left\|k_{\mathbf{x}_{N}}\right\|\left\|k_{\mathbf{x}_{N}}\right\|\cos\theta_{k_{\mathbf{x}_{N}}k_{\mathbf{x}_{N}}}-\lambda_{N}\end{bmatrix}\right)=0$

are the eigenvalues $\lambda_{N}\leq\mathbf{\ldots}\leq\lambda_{1}$ of
$\mathbf{Q}$.

Since the roots $p\left(\lambda\right)=0$ of the characteristic polynomial
$p\left(\lambda\right)$ of the joint covariance matrix $\mathbf{Q}$ vary
continuously with its coefficients [52], it follows that the eigenvalues
$\lambda_{N}\leq\mathbf{\ldots}\leq\lambda_{1}$ of the joint covariance matrix
$\mathbf{Q}$ vary continuously with the elements
$\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x_{i}}}k_{\mathbf{x}_{j}}}$
of $\mathbf{Q}$.

Thereby, the eigenvalues $\lambda_{N}\leq\mathbf{\ldots}\leq\lambda_{1}$ of
the joint covariance matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ _account for_ joint
variabilities between coordinates
$\left\\{\left\|k_{\mathbf{x}_{i}}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}k_{\mathbf{x}_{i}}}\right\\}_{i=1}^{d}$
and
$\left\\{\left\|k_{\mathbf{x}_{j}}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}k_{\mathbf{x}_{j}}}\right\\}_{i=1}^{d}$
of feature vectors $k_{\mathbf{x}_{i}}$ and $k_{\mathbf{x}_{j}}$ used to
construct $\mathbf{Q}$, such that each element
$y_{i}\left\|k_{\mathbf{x}_{i}}\right\|y_{j}\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
of the joint covariance matrix $\mathbf{Q}$ is correlated with the distance
$\left\|k_{\mathbf{x}_{i}}-k_{\mathbf{x}_{j}}\right\|$ between the loci of
certain feature vectors $k_{\mathbf{x}_{i}}$ and $k_{\mathbf{x}_{j}}$.

We will demonstrate that it is crucial that all of the individual feature
vectors ‘add up’ to a complete eigenstructure of the joint covariance matrix
$\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ _and_ the inverted joint
covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$, so that all of the
individual feature vectors ‘speak for themselves’ in such a manner that joint
variabilities between all of the feature vectors are ‘accounted for.’

It will be seen that the machine learning algorithm being examined uses the
condensed eigenstructures of the joint covariance matrix $\mathbf{Q}$ of the
random quadratic form $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
the inverted joint covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic
form $\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$ to _locate_ a
point of equilibrium

$\boldsymbol{\kappa}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i_{\ast}}}k_{\mathbf{x}_{1_{i\ast}}}-\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i_{\ast}}}k_{\mathbf{x}_{2_{i\ast}}}\text{,}$

at which point all of the critical minimums eigenenergies
$\left\|\psi_{1_{i\ast}}k_{\mathbf{x}_{1i\ast}}\right\|_{\min_{c}}^{2}$ and
$\left\|\psi_{1_{i\ast}}k_{\mathbf{x}_{1i\ast}}\right\|_{\min_{c}}^{2}$
exhibited by a minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
are symmetrically concentrated in such a manner that opposing and
counteracting forces and influences of the system are symmetrically balanced
with each other—about the geometric center of the locus of the novel principal
eigenaxis $\boldsymbol{\kappa}$—whereon the statistical fulcrum of the system
is located.

Thereby, it will be seen that any given minimum risk binary classification
system $k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
satisfies a state of statistical equilibrium so that the total allowed
eigenenergy $\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and the
expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
exhibited by the system are jointly minimized within the decision space of the
system, at which point the system exhibits the minimum probability of
classification error.

###  13.3 Why Complete Eigenstructures Are Essential

Let the Wolfe-dual novel principal eigenaxis $\boldsymbol{\psi}$ be
symmetrically and equivalently related to the principal eigenvector
$\boldsymbol{\psi}_{\max}$ of the joint covariance matrix $\mathbf{Q}$ of the
random quadratic form $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ in
the Wolfe-dual eigenenergy functional
$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2$
in (12.9), so that the primal novel principal eigenaxis $\boldsymbol{\kappa}$
is symmetrically and equivalently related to the Wolfe-dual novel principal
eigenaxis $\boldsymbol{\psi}$ in such a manner that the primal novel principal
eigenaxis $\boldsymbol{\kappa}$ is the principal part of an equivalent
representation of the pair of random quadratic forms
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$.

By Theorem 7.1 and Corollary 7.2, we know that an equivalent representation of
the pair of random quadratic forms
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$ is based on complete
eigenstructures of $\mathbf{Q}$ and $\mathbf{Q}^{-1}$, at which point the
joint covariance matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and the inverted joint
covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$ both have full rank.

We will demonstrate that finding an equivalent representation of the random
quadratic form $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ requires
finding the values of the active scale factors $\psi_{i\ast}>0$—which requires
finding the inverse $\mathbf{Q}^{-1}$ of the joint covariance matrix
$\mathbf{Q}$—which is an ill-posed inverse problem if $\mathbf{Q}$ has low
rank [23, 29, 30, 31, 42, 43, 86].

We have previously demonstrated that the values of the active scale factors
$\psi_{i\ast}>0$ and the resulting locations of the principal eigenaxis
components on $\boldsymbol{\psi}$ and $\boldsymbol{\kappa}$ are considerably
affected by the rank and the eigenstructure of the joint covariance matrix
$\mathbf{Q}$ [69]. For example, a low rank joint covariance matrix
$\mathbf{Q}$ determines a binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
that exhibits poor generalization behavior, so that the decision space
$Z=Z_{1}\cup Z_{2}$ of the system is partitioned in an unbalanced manner, at
which point _all_ of the training _data_ are _extreme points_.

Thereby, the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}$ of the binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is not characteristic of the system (is not unique) and is unstable—such that
the geometric locus of the novel principal eigenaxis $\boldsymbol{\kappa}$ of
the system does not represent an eigenaxis of symmetry that spans the decision
space of the system, at which point the _total allowed eigenenergy_
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
exhibited by the novel principal eigenaxis $\boldsymbol{\kappa}$ is
_maximized_.

Any given joint covariance matrix $\mathbf{Q}$ has low rank whenever $d<N$ for
a collection of $N$ feature vectors of dimension $d$. We have resolved these
eigenstructure deficiencies by the regularization method that is described
below.

###  13.4 A Regularization Method for Complete Eigenstructures

The regularized structure of the joint covariance matrix $\mathbf{Q}$ in
(12.9), wherein
$\mathbf{Q}\triangleq\epsilon\mathbf{I}+\widetilde{\mathbf{X}}\widetilde{\mathbf{X}}^{T}$
and $\epsilon\ll 1$, ensures that $\mathbf{Q}$ has full rank and a complete
eigenvector set
$\boldsymbol{\psi=}{\textstyle\sum\nolimits_{i=1}^{N}}\psi_{i}\mathbf{v}_{i}$
that spans the parameter space of the unknown scale factors $\psi_{i}$, so
that $\mathbf{Q}$ has a complete eigenstructure. The regularization constant
$C$ in (12.1) is related to the regularization parameter $\epsilon$ in (12.1)
by $\frac{1}{C}$.

These findings have been published in [65] and [67], where the joint
covariance matrix in (12.9) is a Gram matrix, and are readily extended to
Kernel Gram matrices. Simulation studies published in [65] and [67]
demonstrate that the decision space of a binary classification system is
partitioned in an unbalanced manner if a Gram matrix has low rank, where the
decision boundary is a linear decision boundary.

Thus, it follows that any given geometric locus of a novel principal eigenaxis
that is based on an incomplete eigenstructure—of a joint covariance matrix—is
not characteristic of a minimum risk binary classification system (is not
unique) and is unstable, so that the geometric locus of the novel principal
eigenaxis $\boldsymbol{\kappa}$ of the system does not represent an eigenaxis
of symmetry that spans the decision space of the system, at which point the
the total allowed eigenenergy exhibited by the novel principal eigenaxis is
maximized.

###  13.5 Values of Regularization Parameters

Given $N$ feature vectors of dimension $d$, where $d<N$, all of the
regularization parameters $\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and
all of its derivatives are set equal to a very small value $\xi_{i}=\xi\ll 1$,
e.g. $\xi_{i}=\xi=0.02$, wherein $C=\frac{1}{\xi}$.

Otherwise, given $N$ feature vectors of dimension $d$, where $N<d$, all of the
regularization parameters $\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and
all of its derivatives are set equal to the value of zero $\xi_{i}=\xi=0$,
wherein $C=\infty$.

###  13.6 Ill-posed Binary Classification Systems

Given conditions expressed by Theorem 7.1, Corollaries 7.2 \- 7.3 and Theorem
11.1, we realize that the shape of the decision space of any given minimum
risk binary classification system $k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is completely determined by the geometric locus of the novel principal
eigenaxis $\boldsymbol{\kappa}$ of the system, such that the novel principal
eigenaxis $\boldsymbol{\kappa}$ is the _principal part_ of an _equivalent
representation_ of the pair of random quadratic forms
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$—that is based on the
_eigenstructures_ of the joint covariance matrix $\mathbf{Q}$ of the random
quadratic form $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and the
inverted joint covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic
form $\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$—so that the
geometric locus of the novel principal eigenaxis $\boldsymbol{\kappa}$
represents an eigenaxis of symmetry that spans the decision space of the
system, at which point the eigenenergy exhibited by the principal eigenvector
$\boldsymbol{\psi}_{\max}$ of $\mathbf{Q}$ and $\mathbf{Q}^{-1}$, along with
the eigenenergy exhibited by the novel principal eigenaxis
$\boldsymbol{\kappa}$, are jointly minimized.

Indeed, we now demonstrate that an insufficient eigenstructure of a joint
covariance matrix $\mathbf{Q}$ of a random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$—in the Wolfe-dual
eigenenergy functional in (12.9)—of any given minimum risk binary
classification system $k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
determines an irregularly shaped decision space of the binary classification
system, at which point the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}$ of the system does not represent an eigenaxis of
symmetry that spans the decision space of the system.

Thereby, we demonstrate that any given minimum risk binary classification
system $k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
based on eigenstructure deficiencies is _ill-posed_ —so that the geometric
locus of the novel principal eigenaxis $\boldsymbol{\kappa}$ of the system is
_not unique_ and is _unstable_ —such that the geometric locus of the novel
principal eigenaxis $\boldsymbol{\kappa}$ does not represent an eigenaxis of
symmetry that spans the decision space of the system, at which point the total
allowed _eigenenergy_
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
exhibited by the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ is
_maximized_.

By way of demonstration, we now present regularization examples for full rank
and low rank Gram matrices, as well as full rank and low rank Polynomial
Kernel Gram matrices—that illustrate the generalization performance of the
machine learning algorithm being examined—for two binary classification
systems.

###  13.7 Regularization Example One

Consider the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+$
$\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
for two classes of random vectors that have similar covariance matrices, such
that the covariance matrices for class $\omega_{1}$ and class $\omega_{2}$ are
both given by

$\Sigma_{1}=\Sigma_{2}=\left[\begin{array}[c]{cc}0.95&0.45\\\
0.45&0.35\end{array}\right]\text{,}$

the mean vector for class $\omega_{1}$ is given by
$M_{1}=\begin{pmatrix}3,&0.25\end{pmatrix}^{T}$ and the mean vector for class
$\omega_{2}$ is given by $M_{2}=\begin{pmatrix}3,&-0.25\end{pmatrix}^{T}$,
wherein the probability density functions of the two classes of random vectors
determine overlapping distributions of the random vectors.

We now consider the generalization performance of the machine learning
algorithm being examined—for the binary classification system outlined
above—for full rank and low rank Gram matrices, as well as full rank and low
rank Polynomial Kernel Gram matrices.

##### A Full Rank Gram Matrix

Since the covariance matrices are similar, we can use a linear kernel in the
constrained optimization algorithm that resolves the inverse problem of binary
classification, such that the $N\times d$ matrix $\widetilde{\mathbf{X}}$ in
(12.8) is a matrix of $N$ labeled feature vectors
$\widetilde{\mathbf{X}}=\begin{pmatrix}y_{1}\mathbf{x}_{1},&y_{2}\mathbf{x}_{2},&\ldots,&y_{N}\mathbf{x}_{N}\end{pmatrix}^{T}$.
Now suppose that we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0.02$, wherein $C=50$.

Figure $8$ illustrates that the constrained optimization algorithm being
examined finds a stable and characteristic geometric locus of a novel
principal eigenaxis $\boldsymbol{\tau}=\boldsymbol{\tau_{1}-\tau}_{2}$, such
that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau_{1}-\tau}_{2}$ represents an eigenaxis of
symmetry that spans the decision space $Z=Z_{1}\cup Z_{2}$ of the minimum risk
binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\tau_{1}-\tau}_{2}\right\|_{\min_{c}}^{2}$ exhibited by
the novel principal eigenaxis $\boldsymbol{\tau}$ is minimized, so that
$\boldsymbol{\tau}$ is the solution of vector algebra locus equations that
represent the geometric loci of a linear decision boundary
$d\left(\mathbf{x}\right)=0$ and a pair of symmetrically positioned linear
decision borders $d\left(\mathbf{x}\right)=+1$ and
$d\left(\mathbf{x}\right)=-1$ that jointly partition the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
into symmetrical decision regions $Z_{1}$ and $Z_{2}$, where $80\%$ of the
training data used to construct the joint covariance matrix $\mathbf{Q}$ are
extreme points.

The linear decision boundary $d\left(\mathbf{x}\right)=0$ is black, the linear
decision border $d\left(\mathbf{x}\right)=+1$ is red, the linear decision
border $d\left(\mathbf{x}\right)=-1$ is blue, and all of the extreme points
$\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{1}\right)$
and $\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{2}\right)$ are enclosed in black circles. The error
rate of the minimum risk binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $24\%$.

Figure 8: Given a stable and characteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}$ that is based
on complete eigenstructures of a joint covariance matrix $\mathbf{Q}$ and the
inverted joint covariance matrix $\mathbf{Q}^{-1}$, the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in a symmetrically balanced manner, at which point all of the
training data add up to sufficient eigenstructures of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$, such that $80\%$ of the training data are extreme points
$\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{1}\right)$
and $\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{2}\right)$.

##### A Low Rank Gram Matrix

Next, suppose that we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0$, wherein $C=\inf$.

Figure $9$ illustrates that the constrained optimization algorithm being
examined finds an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}$, such that the
geometric locus of the novel principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau_{1}-\tau}_{2}$ does not represent an
eigenaxis of symmetry that spans the decision space $Z=Z_{1}\cup Z_{2}$ of the
binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}\right\|^{2}$ exhibited by
the novel principal eigenaxis $\boldsymbol{\tau}$ is maximized, so that
$\boldsymbol{\tau}$ is the solution of vector algebra locus equations that
partition the decision space $Z=Z_{1}\cup Z_{2}$ of the binary classification
system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
in an irregular manner, such that the geometric locus of the linear decision
border $d\left(\mathbf{x}\right)=-1$ irregularly partitions a collection of
feature vectors, where $100\%$ of the training data used to construct the
joint covariance matrix $\mathbf{Q}$ are extreme points.

The linear decision border $d\left(\mathbf{x}\right)=-1$ is blue, and all of
the extreme points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$
are enclosed in black circles. The error rate of the binary classification
system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $34\%$.

Figure 9: Given an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}$ that is based
on an incomplete eigenstructure of a joint covariance matrix $\mathbf{Q}$ and
the inverted joint covariance matrix $\mathbf{Q}^{-1}$, the decision space
$Z=Z_{1}\cup Z_{2}$ of the binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in an irregular and unbalanced manner, at which point all of
the training data boil down to insufficient eigenstructures of $\mathbf{Q}$
and $\mathbf{Q}^{-1}$, such that $100\%$ of the training data are extreme
points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$.

##### A Full Rank Polynomial Kernel Gram Matrix

In this example, we use a second-order polynomial reproducing kernel
$k_{\mathbf{x}}\left(\mathbf{s}\right)=\left(\mathbf{s}^{T}\mathbf{x}+1\right)^{2}$,
and we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0.02$, wherein $C=50$.

Figure $10$ illustrates that the constrained optimization algorithm being
examined finds a stable and characteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$, such
that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$
represents an eigenaxis of symmetry that spans the decision space $Z=Z_{1}\cup
Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
exhibited by the novel principal eigenaxis $\boldsymbol{\kappa}$ is minimized,
so that $\boldsymbol{\kappa}$ is the solution of vector algebra locus
equations that represent the geometric loci of a nearly-linear decision
boundary $d\left(\mathbf{s}\right)=0$ and a pair of symmetrically positioned
nearly-linear decision borders $d\left(\mathbf{s}\right)=+1$ and
$d\left(\mathbf{s}\right)=-1$ that jointly partition the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
into symmetrical decision regions $Z_{1}$ and $Z_{2}$, where $78\%$ of the
training data used to construct the joint covariance matrix $\mathbf{Q}$ are
extreme points.

The nearly-linear decision boundary $d\left(\mathbf{s}\right)=0$ is black, the
nearly-linear decision border $d\left(\mathbf{s}\right)=+1$ is red, the
nearly-linear decision border $d\left(\mathbf{s}\right)=-1$ is blue, and all
of the extreme points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$
are enclosed in black circles. The error rate of the minimum risk binary
classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $24\%$.

Figure 10: Given a stable and characteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ that is
based on a complete eigenstructure of a joint covariance matrix $\mathbf{Q}$
and the inverted joint covariance matrix $\mathbf{Q}^{-1}$, the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in a symmetrically balanced manner, at ehich point all of the
training data add up to sufficient eigenstructures of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$, such that only $78\%$ of the training data are extreme
points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$.

##### A Low Rank Polynomial Kernel Gram Matrix

Finally, we use a second-order polynomial reproducing kernel
$k_{\mathbf{x}}\left(\mathbf{s}\right)=\left(\mathbf{s}^{T}\mathbf{x}+1\right)^{2}$,
and we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0$, wherein $C=\inf$.

Figure $11$ illustrates that the constrained optimization algorithm being
examined finds an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$, such
that the geometric locus of a novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ does not
represent an eigenaxis of symmetry that spans the decision space $Z=Z_{1}\cup
Z_{2}$ of the binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|^{2}$ exhibited
by the novel principal eigenaxis $\boldsymbol{\kappa}$ is maximized, so that
$\boldsymbol{\kappa}$ is the solution of vector algebra locus equations that
partition the decision space $Z=Z_{1}\cup Z_{2}$ of the binary classification
system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
in an irregular manner, such that the geometric locus of a hyperbolic decision
border $d\left(\mathbf{x}\right)=-1$ partitions a collection of feature
vectors in an unbalanced manner, where $100\%$ of the training data used to
construct the joint covariance matrix $\mathbf{Q}$ are extreme points.

The hyperbolic decision border $d\left(\mathbf{s}\right)=-1$ is blue, and all
of the extreme points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$
are enclosed in black circles. The error rate of the binary classification
system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $36\%$.

Figure 11: Given an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ that is
based on an incomplete eigenstructure of a joint covariance matrix
$\mathbf{Q}$ and the inverted joint covariance matrix $\mathbf{Q}^{-1}$, the
decision space $Z=Z_{1}\cup Z_{2}$ of the binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in an irregular manner, at which point all of the training data
boil down to insufficient eigenstructures of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$, such that $100\%$ of the training data are extreme points
$\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{1}\right)$
and $\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{2}\right)$.

###  13.8 Regularization Example Two

Consider the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
for two classes of random vectors that have similar covariance matrices, such
that the covariance matrices for class $\omega_{1}$ and class $\omega_{2}$ are
both given by

$\Sigma_{1}=\Sigma_{2}=\left[\begin{array}[c]{cc}0.65&0.25\\\
0.25&0.45\end{array}\right]\text{,}$

the mean vector for class $\omega_{1}$ is given by
$M_{1}=\begin{pmatrix}1,&13\end{pmatrix}^{T}$ and the mean vector for class
$\omega_{2}$ is given by $M_{2}=\begin{pmatrix}6,&22\end{pmatrix}^{T}$,
wherein the probability density functions of the two classes of random vectors
determine non-overlapping distributions of the random vectors.

We now consider the generalization performance of the machine learning
algorithm being examined—for the binary classification system outlined
above—for full rank and low rank Gram matrices, as well as full rank and low
rank Polynomial Kernel Gram matrices.

##### A Full Rank Gram Matrix

Once more, since the covariance matrices are similar, we can use a linear
kernel in the constrained optimization algorithm that resolves the inverse
problem of binary classification, such that the $N\times d$ matrix
$\widetilde{\mathbf{X}}$ in (12.8) is a matrix of $N$ labeled feature vectors
$\widetilde{\mathbf{X}}=\begin{pmatrix}y_{1}\mathbf{x}_{1},&y_{2}\mathbf{x}_{2},&\ldots,&y_{N}\mathbf{x}_{N}\end{pmatrix}^{T}$.
Now suppose that we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0.02$, wherein $C=50$.

Figure $12$ illustrates that the constrained optimization algorithm being
examined finds a stable and characteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}$, such that the
geometric locus of the novel principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau_{1}-\tau}_{2}$ represents an eigenaxis of
symmetry that spans the decision space $Z=Z_{1}\cup Z_{2}$ of the minimum risk
binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}\right\|_{\min_{c}}^{2}$
exhibited by the novel principal eigenaxis $\boldsymbol{\tau}$ is minimized,
so that $\boldsymbol{\tau}$ is the solution of vector algebra locus equations
that represent the geometric loci of a linear decision boundary
$d\left(\mathbf{x}\right)=0$ and a pair of symmetrically positioned linear
decision borders $d\left(\mathbf{x}\right)=+1$ and
$d\left(\mathbf{x}\right)=-1$ that jointly partition the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
into symmetrical decision regions $Z_{1}$ and $Z_{2}$, where only $0.3\%$ of
the training data used to construct the joint covariance matrix $\mathbf{Q}$
are extreme points.

The linear decision boundary $d\left(\mathbf{x}\right)=0$ is black, the linear
decision border $d\left(\mathbf{x}\right)=+1$ is red, the linear decision
border $d\left(\mathbf{x}\right)=-1$ is blue, and all of the extreme points
$\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{1}\right)$
and $\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{2}\right)$ are enclosed in black circles. The error
rate of the minimum risk binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $0\%$.

Figure 12: Given a stable and characteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}$ that is based
on a complete eigenstructure of a joint covariance matrix $\mathbf{Q}$ and the
inverted joint covariance matrix $\mathbf{Q}^{-1}$, the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in a symmetrically balanced manner, at which point all of the
training data add up to sufficient eigenstructures of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$, such that only $0.3\%$ of the training data are extreme
points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$.

##### A Low Rank Gram Matrix

Next, suppose that we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0$, wherein $C=\inf$.

Figure $13$ illustrates that the constrained optimization algorithm being
examined finds an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}$, such that the
geometric locus of the novel principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau_{1}-\tau}_{2}$ does not represent an
eigenaxis of symmetry that spans the decision space $Z=Z_{1}\cup Z_{2}$ of the
binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}\right\|^{2}$ exhibited by
the novel principal eigenaxis $\boldsymbol{\tau}$ is maximized, so that
$\boldsymbol{\tau}$ is the solution of vector algebra locus equations that
partition the decision space $Z=Z_{1}\cup Z_{2}$ of the binary classification
system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
in an irregular manner, such that the geometric locus of the linear decision
border $d\left(\mathbf{x}\right)=-1$ partitions a collection of feature
vectors where $100\%$ of the training data used to construct the joint
covariance matrix $\mathbf{Q}$ are extreme points.

The linear decision border $d\left(\mathbf{x}\right)=-1$ is blue, and all of
the extreme points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$
are enclosed in black circles. Moreover, even though _all_ of the training
data are extreme points, the error rate of the binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $0\%$.

Figure 13: Given an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\tau}=\boldsymbol{\tau}_{1}-\boldsymbol{\tau}_{2}$ that is based
on an incomplete eigenstructure of a joint covariance matrix $\mathbf{Q}$ and
the inverted joint covariance matrix $\mathbf{Q}^{-1}$, the decision space
$Z=Z_{1}\cup Z_{2}$ of the binary classification system
$\mathbf{x}^{T}\boldsymbol{\tau}+\boldsymbol{\tau}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in an irregular manner, at which point all of the training data
boil down to insufficient eigenstructures of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$, such that $100\%$ of the training data are extreme points
$\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{1}\right)$
and $\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{2}\right)$.

##### A Full Rank Polynomial Kernel Gram Matrix

In this example, we use a second-order polynomial reproducing kernel
$k_{\mathbf{x}}\left(\mathbf{s}\right)=\left(\mathbf{s}^{T}\mathbf{x}+1\right)^{2}$,
and we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0.02$, wherein $C=50$.

Figure $14$ illustrates that the constrained optimization algorithm being
examined finds a stable and characteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$, such
that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$
represents an eigenaxis of symmetry that spans the decision space $Z=Z_{1}\cup
Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
exhibited by the novel principal eigenaxis $\boldsymbol{\kappa}$ is minimized,
so that $\boldsymbol{\kappa}$ is the solution of vector algebra locus
equations that represent the geometric loci of a nearly-linear decision
boundary $d\left(\mathbf{s}\right)=0$ and a pair of symmetrically positioned
nearly-linear decision borders $d\left(\mathbf{s}\right)=+1$ and
$d\left(\mathbf{s}\right)=-1$ that jointly partition the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
into symmetrical decision regions $Z_{1}$ and $Z_{2}$, where $0.3\%$ of the
training data used to construct the joint covariance matrix $\mathbf{Q}$ are
extreme points.

The nearly-linear decision boundary $d\left(\mathbf{s}\right)=0$ is black, the
nearly-linear decision border $d\left(\mathbf{s}\right)=+1$ is red, the
nearly-linear decision border $d\left(\mathbf{s}\right)=-1$ is blue, and all
of the extreme points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$
are enclosed in black circles. The error rate of the minimum risk binary
classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $0\%$.

Figure 14: Given a stable and characteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ that is
based on a complete eigenstructure of a joint covariance matrix $\mathbf{Q}$
and the inverted joint covariance matrix $\mathbf{Q}^{-1}$, the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in a symmetrically balanced manner, at which point all of the
training data add up to sufficient eigenstructures of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$, such that only $0.3\%$ of the training data are extreme
points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$.

##### A Low Rank Polynomial Kernel Gram Matrix

Finally, we use a second-order polynomial reproducing kernel
$k_{\mathbf{x}}\left(\mathbf{s}\right)=\left(\mathbf{s}^{T}\mathbf{x}+1\right)^{2}$,
and we let all of the regularization parameters
$\left\\{\xi_{i}\right\\}_{i=1}^{N}$ in (12.1) and all of its derivatives be
equal to $\xi_{i}=\xi=0$, wherein $C=\inf$.

Figure $15$ illustrates that the constrained optimization algorithm being
examined finds an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$, such
that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ does not
represent an eigenaxis of symmetry that spans the decision space $Z=Z_{1}\cup
Z_{2}$ of the binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|^{2}$ exhibited
by the novel principal eigenaxis $\boldsymbol{\kappa}$ is maximized, so that
$\boldsymbol{\kappa}$ is the solution of vector algebra locus equations that
partition the decision space $Z=Z_{1}\cup Z_{2}$ of the binary classification
system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
in an irregular manner, such that the geometric locus of a hyperbolic decision
border $d\left(\mathbf{x}\right)=-1$ partitions a collection of feature
vectors, where $100\%$ of the training data used to construct the joint
covariance matrix $\mathbf{Q}$ are extreme points.

The hyperbolic decision border $d\left(\mathbf{s}\right)=-1$ is blue, and all
of the extreme points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$
are enclosed in black circles. Yet again, even though all of the training data
are extreme points, the error rate of the binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is $0\%$.

Figure 15: Given an unstable and uncharacteristic geometric locus of a novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ that is
based on an incomplete eigenstructure of a joint covariance matrix
$\mathbf{Q}$ and the inverted joint covariance matrix $\mathbf{Q}^{-1}$, the
decision space $Z=Z_{1}\cup Z_{2}$ of the binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is partitioned in an irregular manner, at which point all of the training data
boil down to insufficient eigenstructures of $\mathbf{Q}$ and
$\mathbf{Q}^{-1}$, such that $100\%$ of the training data are extreme points
$\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{1}\right)$
and $\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{2}\right)$.

We now turn our attention to statistical relations between feature
vectors—inside reproducing kernel Hilbert spaces—that determine pointwise
covariance statistics, joint covariance statistics and conditional
distributions for individual feature vectors.

##  14 Statistical Relations Inside Hilbert Spaces

We have discovered that the overall structure and behavior and properties of
any given minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$—which
is found by the constrained optimization algorithm being examined—is
determined by elegant statistical relations and complex statistical
interconnections between all of the principal eigenaxis components and
likelihood components that lie on both sides of the novel principal eigenaxes
$\boldsymbol{\psi}$ and $\boldsymbol{\kappa}$—in accordance with algebraic and
geometrical and statistical conditions expressed by certain KKT conditions in
(12.3) - (12.7) that $\boldsymbol{\psi}$ and $\boldsymbol{\kappa}$ are subject
to.

We identify certain statistical relations and interconnections between all of
the dual components that lie on both sides of $\boldsymbol{\psi}$ and
$\boldsymbol{\kappa}$ by using statistical relations that we have
devised—based on statistical relations and interconnections between random
vectors inside Hilbert spaces and reproducing kernel Hilbert spaces—that
determine pointwise covariance statistics, joint covariance statistics and
conditional distributions for individual random points. The original versions
of these statistical relations can be found in our working papers [68] and
[69].

Axiom 14.1 expresses statistical relations between random vectors in Hilbert
spaces and reproducing kernel Hilbert spaces that determine pointwise
covariance statistics, joint covariance statistics and conditional
distributions for individual random points.

###### Axiom 14.1

Take any given random vectors $\mathbf{x\in}$ $\mathbb{R}^{d}$ and
$\mathbf{y\in}$ $\mathbb{R}^{d}$ in Hilbert space that are generated by
certain probability density functions $\mathbf{x\sim}$
$p\left(\mathbf{x}\right)$ and $\mathbf{y\sim}$ $p\left(\mathbf{y}\right)$,
such that all of the $d$ random point coordinates
$\left\\{\left\|\mathbf{x}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}\mathbf{x}}\right\\}_{i=1}^{d}$
on the locus of $\mathbf{x}$ and all of the $d$ random point coordinates
$\left\\{\left\|\mathbf{y}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}\mathbf{y}}\right\\}_{i=1}^{d}$
on the locus of $\mathbf{y}$ are random variables that have expected values
and covariances.

The random vectors $\mathbf{x}$ and $\mathbf{y}$ satisfy the law of cosines

$\left\|\mathbf{x-y}\right\|^{2}=\left\|\mathbf{x}\right\|^{2}+\left\|\mathbf{y}\right\|^{2}-2\left\|\mathbf{x}\right\|\left\|\mathbf{y}\right\|\cos\theta$

which reduces to the following relation between the random vectors
$\mathbf{x}$ and $\mathbf{y}$

$\displaystyle\left\|\mathbf{x}\right\|\left\|\mathbf{y}\right\|\cos\theta$
$\displaystyle=x_{1}y_{1}+x_{2}y_{2}+\cdots+x_{d}y_{d}$
$\displaystyle=\mathbf{x}^{T}\mathbf{y=y}^{T}\mathbf{x}\text{,}$

at which point the relation
$\left\|\mathbf{x}\right\|\left\|\mathbf{y}\right\|\cos\theta$ between the
random vectors $\mathbf{x}$ and $\mathbf{y}$ is correlated with the distance
$\left\|\mathbf{x-y}\right\|$ between the loci of the random points
$\mathbf{x}$ and $\mathbf{y}$.

Thereby, the relation
$\left\|\mathbf{x}\right\|\left\|\mathbf{y}\right\|\cos\theta$ between the
random vectors $\mathbf{x}$ and $\mathbf{y}$ determines a pointwise covariance
statistic that represents joint variations between all of the $d$ random point
coordinates on the locus of $\mathbf{x}$

$\mathbf{x=}\begin{pmatrix}\left\|\mathbf{x}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{1}\mathbf{x}},&\left\|\mathbf{x}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{2}\mathbf{x}},&\cdots,&\left\|\mathbf{x}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{d}\mathbf{x}}\end{pmatrix}^{T}$

and all of the $d$ random point coordinates on the locus of $\mathbf{y}$

$\mathbf{y=}\begin{pmatrix}\left\|\mathbf{y}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{1}\mathbf{y}},&\left\|\mathbf{y}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{2}\mathbf{y}},&\cdots,&\left\|\mathbf{y}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{d}\mathbf{y}}\end{pmatrix}^{T}\text{,}$

so that the covariance of the random variables
$\left\|\mathbf{x}\right\|\cos\alpha_{\mathbf{e}_{i}\mathbf{x}}$ and
$\left\|\mathbf{y}\right\|\cos\alpha_{\mathbf{e}_{i}\mathbf{y}}$ on the loci
of $\mathbf{x}$ and $\mathbf{y}$ is a function of the distance

$\left|\left\|\mathbf{x}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}\mathbf{x}}-\left\|\mathbf{y}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}\mathbf{y}}\right|$

between the random point coordinates
$\left\|\mathbf{x}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}\mathbf{x}}$ and
$\left\|\mathbf{y}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}\mathbf{y}}$ on
the orthonormal coordinate axis $\mathbf{e}_{i}$.

Furthermore, the relation
$\left\|\mathbf{x}\right\|\left\|\mathbf{y}\right\|\cos\theta$ between the
random vectors $\mathbf{x}$ and $\mathbf{y}$ represents the length
$\left\|\mathbf{x}\right\|$ of the random vector $\mathbf{x}$ times the signed
magnitude $\left\|\mathbf{y}\right\|\cos\theta$ of the vector projection of
the random vector $\mathbf{y}$ onto the random vector $\mathbf{x}$

(14.1) $\displaystyle\mathbf{x}^{T}\mathbf{y}$
$\displaystyle\mathbf{=}\left\|\mathbf{x}\right\|\times\left[\left\|\mathbf{y}\right\|\cos\theta\right]$
$\displaystyle\triangleq\left\|\mathbf{x}\right\|\operatorname{comp}_{\overrightarrow{\mathbf{x}}}\left(\overrightarrow{\mathbf{y}}\right)\text{,}$

such that the signed magnitude $\left\|\mathbf{y}\right\|\cos\theta$—expressed
by
$\operatorname{comp}_{\overrightarrow{\mathbf{x}}}\left(\overrightarrow{\mathbf{y}}\right)$—is
a random variable that is negative if $\pi/2<\theta\leq\pi$, at which point
the algebraic and geometric relationship
$\operatorname{comp}_{\overrightarrow{\mathbf{x}}}\left(\overrightarrow{\mathbf{y}}\right)$
between the locus of $\mathbf{x}$ and the locus of $\mathbf{y}$ determines an
implicit random vector, such that the magnitude and the direction of the
implicit random vector is determined by the signed magnitude
$\left\|\mathbf{y}\right\|\cos\theta$ along the locus of the random vector
$\mathbf{x}$.

Moreover, the signed magnitude
$\operatorname{comp}_{\overrightarrow{\mathbf{x}}}\left(\overrightarrow{\mathbf{y}}\right)$
of the vector projection of the random vector $\mathbf{y}$ onto the random
vector $\mathbf{x}$ represents a distribution of the random vector
$\mathbf{y}$ that is conditional on the relationship
$\left\|\mathbf{x}\right\|\times\left[\left\|\mathbf{y}\right\|\cos\theta\right]$,
such that the signed magnitude
$\operatorname{comp}_{\overrightarrow{\mathbf{x}}}\left(\overrightarrow{\mathbf{y}}\right)$
represents a certain portion $\left\|\mathbf{y}\right\|\cos\theta$ of the
random vector $\mathbf{y}$ that is distributed along the locus of the random
vector $\mathbf{x}$, at which point the distribution of the random vector
$\mathbf{y}$ is either positive or negative.

Now let $k_{\mathbf{x}}\mathbf{\ }$and $k_{\mathbf{y}}$ be any given random
vectors in a reproducing kernel Hilbert space, such that a reproducing kernel
in Hilbert space is a certain reproducing kernel
$k_{\mathbf{x}}\left(\mathbf{s}\right)$ that is defined on random vectors
$\mathbf{x\in\mathbb{R}}^{d}$, so that $k_{\mathbf{x}}$ and $k_{\mathbf{y}}$
are reproducing kernels for the random points $\mathbf{x\in}$ $\mathbb{R}^{d}$
and $\mathbf{y\in}$ $\mathbb{R}^{d}$, at which point all of the $d$ random
point coordinates
$\left\\{\left\|k_{\mathbf{x}}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}k_{\mathbf{x}}}\right\\}_{i=1}^{d}$
on the locus of $k_{\mathbf{x}}$ and all of the $d$ random point coordinates
$\left\\{\left\|k_{\mathbf{y}}\right\|\cos\mathbb{\alpha}_{\mathbf{e}_{i}k_{\mathbf{y}}}\right\\}_{i=1}^{d}$
on the locus of $k_{\mathbf{y}}$ are random variables that have expected
values and covariances.

Since the random vectors $k_{\mathbf{x}}$ and $k_{\mathbf{y}}$ satisfy the law
of cosines

$\left\|k_{\mathbf{x}}\mathbf{-}k_{\mathbf{y}}\right\|^{2}=\left\|k_{\mathbf{x}}\right\|^{2}+\left\|k_{\mathbf{y}}\right\|^{2}-2\left\|k_{\mathbf{x}}\right\|\left\|k_{\mathbf{y}}\right\|\cos\theta\text{,}$

it follows that the relation
$\left\|k_{\mathbf{x}}\right\|\left\|k_{\mathbf{y}}\right\|\cos\theta_{k_{\mathbf{x}}k_{\mathbf{y}}}$
between the reproducing kernels for the random points $\mathbf{x\in}$
$\mathbb{R}^{d}$ and $\mathbf{y\in}$ $\mathbb{R}^{d}$ determines a pointwise
covariance statistic for the random points $k_{\mathbf{x}}$ and
$k_{\mathbf{y}}$, along with a distribution of the random vector
$k_{\mathbf{y}}$ that is conditional on the relationship
$\left\|k_{\mathbf{x}}\right\|\times\left[\left\|k_{\mathbf{y}}\right\|\cos\theta\right]$,
such that the signed magnitude
$\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}}}}\left(\overrightarrow{k_{\mathbf{y}}}\right)$
represents a certain portion $\left\|k_{\mathbf{y}}\right\|\cos\theta$ of the
random vector $k_{\mathbf{y}}$ that is distributed along the locus of the
random vector $k_{\mathbf{x}}$, at which point the distribution of the random
vector $k_{\mathbf{y}}$ is either positive or negative.

Therefore, take the $N\times N$ kernel matrix $\mathbf{Q}$ for any given
collection of random vectors $\left\\{k_{\mathbf{x}_{j}}\right\\}_{j=1}^{N}$

$\mathbf{Q}=\begin{pmatrix}\left\|k_{\mathbf{x}_{1}}\right\|\left\|k_{\mathbf{x}_{1}}\right\|\cos\theta_{k_{\mathbf{x_{1}}}k_{\mathbf{x}_{1}}}&\cdots&\left\|k_{\mathbf{x}_{1}}\right\|\left\|k_{\mathbf{x}_{N}}\right\|\cos\theta_{k_{\mathbf{x}_{1}}k_{\mathbf{x}_{N}}}\\\
\vdots&\ddots&\vdots\\\
\left\|k_{\mathbf{x}_{N}}\right\|\left\|k_{\mathbf{x}_{1}}\right\|\cos\theta_{k_{\mathbf{x}_{N}}k_{\mathbf{x}_{1}}}&\cdots&\left\|k_{\mathbf{x}_{N}}\right\|\left\|k_{\mathbf{x}_{N}}\right\|\cos\theta_{k_{\mathbf{x}_{N}}k_{\mathbf{x}_{N}}}\end{pmatrix}\text{,}$

such that each element $\mathbf{Q}\left(i,j\right)$ of $\mathbf{Q}$ represents
joint variations between all of the $d$ random point coordinates on the loci
of two random vectors $k_{\mathbf{x}_{i}}$ and $k_{\mathbf{x}_{j}}$ by the
relation
$\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$,
so that the signed magnitude
$\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}_{i}}}}\left(\overrightarrow{k_{\mathbf{x}_{j}}}\right)$
of the random vector $k_{\mathbf{x}_{j}}$ along the locus of the random vector
$k_{\mathbf{x}_{i}}$ determines a certain portion
$\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
of the random vector $k_{\mathbf{x}_{j}}$ that is distributed along the locus
of the random vector $k_{\mathbf{x}_{i}}$, at which point the distribution of
the random vector $k_{\mathbf{x}_{j}}$ is either positive or negative.

Thereby, the kernel matrix $\mathbf{Q}$ determines an estimate of a covariance
matrix, such that each row $i$ of $\mathbf{Q}$ represents joint variations
between all of the $d$ random point coordinates on the locus of a random
vector $k_{\mathbf{x}_{i}}$ and all of the $d$ random point coordinates of
each of the random vectors $k_{\mathbf{x}_{j}}$ in the collection
$\left\\{k_{\mathbf{x}_{j}}\right\\}_{j=1}^{N}$, along with a distribution of
the random vector $k_{\mathbf{x}_{i}}$ that is conditional on relations
$\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
between the random vector $k_{\mathbf{x}_{i}}$ and all of the random vectors
$k_{\mathbf{x}_{j}}$ in the collection
$\left\\{k_{\mathbf{x}_{j}}\right\\}_{j=1}^{N}$.

Next, take any given row $i$ of the $N\times N$ kernel matrix $\mathbf{Q}$. A
joint covariance statistic for the random vector $k_{\mathbf{x}_{i}}$ is given
by the expression

(14.2)
$\displaystyle\widehat{\operatorname*{cov}}\left(k_{\mathbf{x}_{i}}\right)$
$\displaystyle=\sum\nolimits_{j=1}^{N}\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
$\displaystyle=\left\|k_{\mathbf{x}_{i}}\right\|\sum\nolimits_{j=1}^{N}\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
$\displaystyle=\left\|k_{\mathbf{x}_{i}}\right\|\sum\nolimits_{j=1}^{N}\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}_{i}}}}\left(\overrightarrow{k_{\mathbf{x}_{j}}}\right)$
$\displaystyle=\left\|k_{\mathbf{x}_{i}}\right\|\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}_{i}}}}\left(\overrightarrow{\sum\nolimits_{j=1}^{N}k_{\mathbf{x}_{j}}}\right)\text{,}$

based on the $N$ relations
$\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
between the random vector $k_{\mathbf{x}_{i}}$ and the $N$ random vectors
$k_{\mathbf{x}_{j}}$ in the collection
$\left\\{k_{\mathbf{x}_{j}}\right\\}_{j=1}^{N}$, such that the relation
$\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
between the random vector $k_{\mathbf{x}_{i}}$ and any given random vector
$k_{\mathbf{x}_{j}}$ in the collection represents joint variations between all
of the $d$ point coordinates on the loci of the random vectors
$k_{\mathbf{x}_{i}}$ and $k_{\mathbf{x}_{j}}$, and the signed magnitude
$\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}_{i}}}}\left(\overrightarrow{k_{\mathbf{x}_{j}}}\right)$
of the random vector $k_{\mathbf{x}_{j}}$ along the locus of the random vector
$k_{\mathbf{x}_{i}}$ determines a certain portion
$\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$
of the random vector $k_{\mathbf{x}_{j}}$ that is distributed along the locus
of the random vector $k_{\mathbf{x}_{i}}$, at which point the distribution of
the random vector $k_{\mathbf{x}_{j}}$ is either positive or negative.

Accordingly, the expression
$\left\|k_{\mathbf{x}_{i}}\right\|\sum\nolimits_{j=1}^{N}\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}_{i}}}}\left(\overrightarrow{k_{\mathbf{x}_{j}}}\right)$
determines a condensed signed magnitude—along the locus of the random vector
$k_{\mathbf{x}_{i}}$—that represents a joint covariance
$\widehat{\operatorname*{cov}}\left(k_{\mathbf{x}_{i}}\right)$ of the random
vector $k_{\mathbf{x}_{i}}$ that is conditional on the relations
$\sum\nolimits_{j=1}^{N}\left\|k_{\mathbf{x}_{i}}\right\|\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x}_{i}}k_{\mathbf{x}_{j}}}$,
along with an expected location of the random vector $k_{\mathbf{x}_{i}}$ that
is conditional on how the loci of the $N$ vectors
$\left\\{k_{\mathbf{x}_{j}}\right\\}_{i=1}^{N}$ are distributed along the
locus of the random vector $k_{\mathbf{x}_{i}}$.

The condensed signed magnitude
$\left\|k_{\mathbf{x}_{i}}\right\|\sum\nolimits_{j=1}^{N}\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}_{i}}}}\left(\overrightarrow{k_{\mathbf{x}_{j}}}\right)$
along the locus of the random vector $k_{\mathbf{x}_{i}}$ can be written in
terms of signed magnitudes
$\left\|k_{\mathbf{x}_{i}}\right\|\cos\theta_{k_{\mathbf{x}_{j}}k_{\mathbf{x}_{i}}}$
along the loci of the $N$ random vectors $k_{\mathbf{x}_{j}}$

(14.3)
$\displaystyle\widehat{\operatorname*{cov}}\left(k_{\mathbf{x}_{i}}\right)$
$\displaystyle=\sum\nolimits_{j=1}^{N}\left\|k_{\mathbf{x}_{j}}\right\|\left[\left\|k_{\mathbf{x}_{i}}\right\|\cos\theta_{k_{\mathbf{x}_{j}}k_{\mathbf{x}_{i}}}\right]$
$\displaystyle=\sum\nolimits_{j=1}^{N}\left\|k_{\mathbf{x}_{j}}\right\|\operatorname{comp}_{\overrightarrow{k_{\mathbf{x}_{j}}}}\left(\overrightarrow{k_{\mathbf{x}_{i}}}\right)\text{,}$

so that the expected location and the joint covariance
$\widehat{\operatorname*{cov}}\left(k_{\mathbf{x}_{i}}\right)$ of the random
vector $k_{\mathbf{x}_{i}}$ are both determined by how the locus of the random
vector $k_{\mathbf{x}_{i}}$ is distributed along the loci of the $N$ random
vectors $\left\\{k_{\mathbf{x}_{j}}\right\\}_{i=1}^{N}$.

We now turn our attention to the core of the machine learning algorithm that
finds discriminant functions of minimum risk binary classification systems.

##  15 Inside the Wolfe-dual Principal Eigenspace

In this section of our treatise, we reveal elegant statistical relations and
deep-seated statistical interconnections between all of the principal
eigenaxis components and likelihood components that lie on both sides of the
primal novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ in
(12.11)

$\displaystyle\boldsymbol{\kappa}$
$\displaystyle=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}-\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$
$\displaystyle=\boldsymbol{\kappa}_{1}{\large-}\boldsymbol{\kappa}_{2}$

and the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}$ in (12.12)

$\displaystyle\boldsymbol{\psi}$
$\displaystyle=\sum\nolimits_{i=1}^{l_{1}}\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}+\sum\nolimits_{i=1}^{l_{2}}\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
$\displaystyle=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}$

of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
such that critical interconnections between intrinsic components of the system
determine an exclusive principal eigen-coordinate system
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ that is
the principal part of an equivalent representation of the pair of random
quadratic forms $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$, so that the structure
and behavior and properties of the geometric locus of the primal novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ are
symmetrically and equivalently related to the structure and behavior and
properties of the geometric locus of the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}$, at which
point the Wolfe-dual novel principal eigenaxis $\boldsymbol{\psi}$ is
symmetrically and equivalently related to the largest eigenvector
$\boldsymbol{\psi}_{\max}$ of the joint covariance matrix $\mathbf{Q}$ and the
inverted joint covariance matrix $\mathbf{Q}^{-1}$ associated with the pair of
random quadratic forms $\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$.

The Wolfe-dual eigenenergy functional in (12.9) is the core of the machine
learning algorithm that finds discriminant functions of minimum risk binary
classification systems.

In Section 20, it will be seen that the machine learning algorithm being
examined _transforms_ the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ in the Wolfe-dual
eigenenergy functional
$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2$
of a minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\mathbf{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$—which
is subject to the constraints $\boldsymbol{\psi}^{T}\mathbf{y}=0$ and
$\psi_{i\ast}>0$— _into_ a geometric locus of a novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ of the
system, so that the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$
satisfies the geometric locus of the decision boundary
$k_{\mathbf{s}}\boldsymbol{\kappa}+$ $\boldsymbol{\kappa}_{0}=0$ of the system
in terms of a critical minimum eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and a minimum expected
risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$,
at which point the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ and the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
exhibited by the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ are both
regulated by the total value of the scale factors $\psi_{1i\ast}$ and
$\psi_{2i\ast}$ for the components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
of the principal eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$ and the inverted joint covariance matrix
$\mathbf{Q}^{-1}$ associated with the pair of random quadratic forms
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$.

We now identify the essence of the Wolfe-dual eigenenergy functional
$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2$,
which is subject to the constraints $\boldsymbol{\psi}^{T}\mathbf{y}=0$ and
$\psi_{i\ast}>0$, where $y_{i}=\left\\{\pm 1\right\\}$, so that the structure
and behavior and properties of the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}$ are
symmetrically and equivalently related to the structure and behavior and
properties of the primal novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$.

###  15.1 Symmetrical and Equivalent Eigenenergies

Consider again the critical minimum eigenenergy constraint
$\gamma\left(\boldsymbol{\kappa}\right)=\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$
on the geometric locus of the primal novel principal eigenaxis
$\boldsymbol{\kappa}$ in (12.1), so that the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
and the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ exhibited by a minimum
risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\mathbf{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
are jointly minimized within the decision space $Z=Z_{1}\cup Z_{2}$ of the
system.

By the strong duality theorem [24, 47, 48, 54], we realize that the geometric
locus of the Wolfe-dual novel principal eigenaxis $\boldsymbol{\psi}$ is
subject to a critical minimum eigenenergy constraint that is symmetrically and
equivalently related to the critical minimum eigenenergy constraint on the
geometric locus of the primal novel principal eigenaxis
$\boldsymbol{\kappa}$—inside the Wolfe-dual principal eigenspace of
$\boldsymbol{\psi}$ and $\boldsymbol{\kappa}$, so that the Wolfe-dual novel
principal eigenaxis $\boldsymbol{\psi}$ and the primal novel principal
eigenaxis $\boldsymbol{\kappa}$ exhibit symmetrical and equivalent
eigenenergies
$\left\|Z|\boldsymbol{\psi}\right\|_{\min_{c}}^{2}\equiv\left\|Z|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$.

Thereby, the geometric locus of the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}$ is subject to a critical minimum eigenenergy constraint

(15.1)
$\lambda_{1}\left\|\boldsymbol{\psi}\right\|_{\min_{c}}^{2}=\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}\equiv\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$

at which point the random quadratic form
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}$ is
symmetrically and equivalently related to the critical minimum eigenenergy
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ exhibited by the geometric
locus of the primal novel principal eigenaxis $\boldsymbol{\kappa}$, so that
the random quadratic form
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}$, plus the
total allowed eigenenergy $\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$
and the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\right)$
exhibited by the geometric locus of the primal novel eigenaxis
$\boldsymbol{\kappa}$ jointly reach their minimum values.

Given the eigenenergy constraint on $\boldsymbol{\psi}$ in (15.1), it follows
that the Wolfe-dual eigenenergy functional

$\max\Xi\left(\boldsymbol{\psi}\right)=\mathbf{1}^{T}\boldsymbol{\psi}-\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi/}2\text{,}$

such that $\boldsymbol{\psi}^{T}\mathbf{y}=0$ and $\psi_{i\ast}>0$, is
_maximized_ by the largest eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$

(15.2)
$\mathbf{Q}\boldsymbol{\psi}_{\max}=\lambda_{1}\boldsymbol{\psi}_{\max}\text{,}$

at which point the random quadratic form
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}$ reaches it
_minimum_ value, so that the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
and the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}\right)$
exhibited by the geometric locus of the primal novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ are
jointly minimized.

Thereby, all of the principal eigenaxis components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}$, along with
all of the principal eigenaxis components
$\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ on the primal novel principal
eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$, are
subject to minimum length constraints—in accordance with the critical minimum
eigenenergy constraints on both $\boldsymbol{\psi}$ and $\boldsymbol{\kappa}$.

Given conditions expressed by Theorem 7.1 and the expression for the principal
eigenvector $\boldsymbol{\psi}_{\max}$ of the joint covariance matrix
$\mathbf{Q}$ in (12.12), it follows that the random quadratic form
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}$ has an
equivalent representation that is given by

(15.3)
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}\boldsymbol{=}\sum\nolimits_{i=1}^{l_{1}}\lambda_{1i}\left\|\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}\right\|^{2}+\sum\nolimits_{i=1}^{l_{2}}\lambda_{2i}\left\|\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}\right\|^{2}\text{,}$

where $\lambda_{1i}$ and $\lambda_{2i}$ are eigenvalues of the joint
covariance matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}$,
$\psi_{1i\ast}$ and $\psi_{2i\ast}$ are scale factors for principal eigenaxis
components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}$, and
$\left\|\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}\right\|^{2}$
and
$\left\|\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}\right\|$
are eigenenergies exhibited by corresponding principal eigenaxis components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on $\boldsymbol{\psi}=\boldsymbol{\psi}_{1}+\boldsymbol{\psi}_{2}$.

Given the equivalent representation for the random quadratic form in (15.3),
along with the eigenenergy constraint on $\boldsymbol{\psi}$ in (15.1)

$\lambda_{1}\left\|\boldsymbol{\psi}\right\|_{\min_{c}}^{2}=\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}\equiv\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}\text{,}$

it follows that the total allowed eigenenergy
$\lambda_{1}\left\|\boldsymbol{\psi}\right\|_{\min_{c}}^{2}$ exhibited by the
Wolfe-dual novel principal eigenaxis $\boldsymbol{\psi}$ and the total allowed
eigenenergy $\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ exhibited by
the primal novel principal eigenaxis $\boldsymbol{\kappa}$ are both regulated
by eigenvalues $\lambda_{1i}$ and $\lambda_{2i}$ of the joint covariance
matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$, where eigenvalues
$\lambda_{i}$ of the joint covariance matrix $\mathbf{Q}$ associated with
scale factors $\psi_{i}$ for which $\psi_{i}=0$ also regulate the total
allowed eigenenergies
$\lambda_{1}\left\|\boldsymbol{\psi}\right\|_{\min_{c}}^{2}$ and
$\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$ exhibited by
$\boldsymbol{\psi}$ and $\boldsymbol{\kappa}$.

We now begin identifying how the constrained geometric locus of the Wolfe-dual
novel principal eigenaxis

$\boldsymbol{\psi}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}+\sum\nolimits_{i=1}^{l_{2}}\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$

is symmetrically and equivalently related to the constrained geometric locus
of the primal novel principal eigenaxis

$\boldsymbol{\kappa}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}-\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$

inside the Wolfe-dual principal eigenspace of $\boldsymbol{\psi}$ and
$\boldsymbol{\kappa}$, where $l_{1}+l_{2}=$ $l$.

###  15.2 Statistical Pre-wiring of Important Generalizations

In this section of our treatise, we examine the vector algebra locus equation
$\boldsymbol{\psi}=\lambda_{1}^{-1}\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}$, so
that the geometric locus of the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}$ is related to the scaled principal eigenvector
$\lambda_{1}^{-1}\boldsymbol{\psi}_{\max}$ _acting_ on the joint covariance
matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$.

Thereby, we demonstrate that certain eigenfunction actions on the joint
covariance matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ _statistically_ _pre-wire
the important generalizations_ for a minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\mathbf{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$—such
that likely locations and likelihood values for each and every one of the
extreme points $\mathbf{x}_{1_{i\ast}}\mathbf{\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$ and
$\mathbf{x}_{2_{i\ast}}\mathbf{\sim}$ $p\left(\mathbf{x};\omega_{2}\right)$
are _statistically pre-wired within the geometric locus of the novel principal
eigenaxis_
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ of the
system—with _respect to_ and _in relation to_ each and every one of the
principal eigenaxis components $\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$
and $\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ and the correlated likelihood
components $\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ that lie on the geometric locus
of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$.

###  15.3 Eigenfunction Actions on Joint Covariance Matrices

Using the eigenvector relation in (15.2), it follows that the geometric locus
of a Wolfe-dual novel principal eigenaxis $\boldsymbol{\psi}$ satisfies the
vector algebra locus equation

(15.4) $\displaystyle\boldsymbol{\psi}$
$\displaystyle\triangleq\left(\begin{array}[c]{c}\psi_{1}\\\ \psi_{2}\\\
\vdots\\\
\psi_{N}\end{array}\right)=\frac{\psi_{1}}{\lambda_{1}}\begin{pmatrix}\left\|k_{\mathbf{x}_{1}}\right\|\left\|k_{\mathbf{x}_{1}}\right\|\cos\theta_{k_{\mathbf{x_{1}}}k_{\mathbf{x}_{1}}}\\\
\left\|k_{\mathbf{x}_{2}}\right\|\left\|k_{\mathbf{x}_{1}}\right\|\cos\theta_{k_{\mathbf{x}_{2}}k_{\mathbf{x}_{1}}}\\\
\vdots\\\
-\left\|k_{\mathbf{x}_{N}}\right\|\left\|k_{\mathbf{x}_{1}}\right\|\cos\theta_{k_{\mathbf{x}_{N}}k_{\mathbf{x}_{1}}}\end{pmatrix}+\cdots$
$\displaystyle+\cdots\frac{\psi_{N}}{\lambda_{1}}\begin{pmatrix}-\left\|k_{\mathbf{x}_{1}}\right\|\left\|k_{\mathbf{x}_{N}}\right\|\cos\theta_{k_{\mathbf{x}_{1}}k_{\mathbf{x}_{N}}}\\\
-\left\|k_{\mathbf{x}_{2}}\right\|\left\|k_{\mathbf{x}_{N}}\right\|\cos\theta_{k_{\mathbf{x}_{2}}k_{\mathbf{x}_{N}}}\\\
\vdots\\\
\left\|k_{\mathbf{x}_{N}}\right\|\left\|k_{\mathbf{x}_{N}}\right\|\cos\theta_{k_{\mathbf{x}_{N}}k_{\mathbf{x}_{N}}}\end{pmatrix}\text{,}$

so that the geometric locus of the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}$ is related to the scaled principal eigenvector
$\lambda_{1}^{-1}\boldsymbol{\psi}_{\max}$ _acting_ on the joint covariance
matrix $\mathbf{Q}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$, at which point each scale
factor $\psi_{i}$ for a principal axis of the principal eigenvector
$\boldsymbol{\psi}_{\max}$ is correlated with scalar projections
$\left\|k_{\mathbf{x}_{j}}\right\|\cos\theta_{k_{\mathbf{x_{i}}}k_{\mathbf{x}_{j}}}$
of a feature vector $k_{\mathbf{x}_{j}}$ onto a collection of $N$ signed $\pm
1$ feature vectors $k_{\mathbf{x}_{i}}$.

It will be seen that the vector algebra locus equation
$\boldsymbol{\psi}=\lambda_{1}^{-1}\boldsymbol{\psi}^{T}\mathbf{Q}$ in (15.4)
implements a vector-valued cost function, such that the eigenenergy exhibited
by both $\boldsymbol{\psi}$ and $\boldsymbol{\kappa}$ is minimized in
accordance with the eigenenergy condition
$\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}=\lambda_{\boldsymbol{1}}\left\|\boldsymbol{\psi}\right\|_{\min_{c}}^{2}\equiv\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$,
so that all of the principal eigenaxis components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on $\boldsymbol{\psi}$ and all of the principal eigenaxis components
$\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ on $\boldsymbol{\kappa}$ are
subject to minimum length constraints, at which point the random quadratic
form $\boldsymbol{\psi}_{\max}^{T}\mathbf{Q}\boldsymbol{\psi}_{\max}$ and the
total allowed eigenenergy $\left\|\boldsymbol{\kappa}\right\|_{\min_{c}}^{2}$
exhibited by the novel principal eigenaxis $\boldsymbol{\kappa}$ both reach
their minimum value.

We have previously demonstrated that the Wolfe-dual novel principal eigenaxis
$\boldsymbol{\psi}$ satisfies the Lagrangian relation

(15.5)
$\boldsymbol{\psi}=\mathbf{Q}^{-1}\left(\mathbf{1+\lambda}\right)+\lambda_{0}\mathbf{Q}^{-1}\boldsymbol{y}\text{,}$

where $\mathbf{\lambda}$ and $\lambda_{0}$ denote additional Lagrange
multipliers for the Lagrangian of the Wolfe-dual eigenenergy functional in
(12.9) [65, 67].

By the Lagrangian relation in (15.5), it follows that finding an equivalent
representation of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ in (12.9) requires finding
the values of the active scale factors $\psi_{i\ast}>0$ that are associated
with the inverted $\mathbf{Q}^{-1}$ joint covariance matrix of the random
quadratic form $\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$.

It also follows that the geometric locus of the Wolfe-dual novel principal
eigenaxis $\boldsymbol{\psi}$ is determined by covariance and distribution
information for extreme vectors $k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{i\ast}}}$ that is contained within the inverted joint
covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$, such that the
covariance and distribution information is conditional on relations between
each extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ or
$k_{\mathbf{x}_{2_{i\ast}}}$ and all of the feature vectors
$k_{\mathbf{x}_{i}}$ in the collection
$\left\\{k_{\mathbf{x}_{i}}\right\\}_{i=1}^{N}$ of training data.

Moreover, by Axiom 14.1 and the Lagrangian relation in (15.5), it follows that
each active scale factor $\psi_{1i\ast}>0$ and $\psi_{2i\ast}>0$ is related to
the inverted joint covariance matrix $\mathbf{Q}^{-1}$ so that each principal
eigenaxis component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on $\boldsymbol{\psi}$ is a function of covariance and distribution
information for a correlated extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{i\ast}}}$, at which point the information is contained
within the inverted joint covariance matrix $\mathbf{Q}^{-1}$ of the random
quadratic form $\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$—and is
conditional on relations between an extreme vector
$k_{\mathbf{x}_{1_{i\ast}}}$ or $k_{\mathbf{x}_{2_{i\ast}}}$ and all of the
feature vectors $k_{\mathbf{x}_{i}}$ in the collection
$\left\\{k_{\mathbf{x}_{i}}\right\\}_{i=1}^{N}$ of training data.

Correspondingly, given that the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$ in (15.5) vary continuously with the
elements of the inverted joint covariance matrix $\mathbf{Q}^{-1}$, it follows
that each active scale factor $\psi_{1i\ast}>0$ and $\psi_{2i\ast}>0$ is a
function of the eigenvalues of the inverted joint covariance matrix
$\mathbf{Q}^{-1}$, such that each principal eigenaxis component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on $\boldsymbol{\psi}$ is a function of covariance and distribution
information for a correlated extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{i\ast}}}$ that is represented by the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$.

Even more, using Axiom 14.1, the Lagrangian relation in (15.5) and the
expression for the primal novel principal eigenaxis $\boldsymbol{\kappa}$ in
(12.11)

$\boldsymbol{\kappa}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}-\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}\text{,}$

it follows that each scale factor $\psi_{1i\ast}$ and $\psi_{2i\ast}$ maps
covariance and distribution information for a correlated extreme vector
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$ onto the extreme
vector $k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$, where
the covariance and distribution information is represented by the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$.

Thereby, each principal eigenaxis component
$\psi_{1i\ast}k_{\mathbf{x}_{1i\ast}}$ and
$\psi_{2i\ast}k_{\mathbf{x}_{2i\ast}}$ on the primal novel principal eigenaxis
$\boldsymbol{\kappa}$ is a function of covariance and distribution
information— for a correlated extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{i\ast}}}$—that is contained within the inverted joint
covariance matrix $\mathbf{Q}^{-1}$, such that the covariance and distribution
information for any given extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ or
$k_{\mathbf{x}_{2_{i\ast}}}$ is represented by the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$ and is conditional on the entire
collection $\left\\{k_{\mathbf{x}_{i}}\right\\}_{i=1}^{N}$ of feature vectors
$k_{\mathbf{x}_{i}}$.

What is more, given the interrelations between (12.11), (15.4) and (15.5), it
follows that each scale factor $\psi_{1i\ast}$ is a function—of _all_ of the
covariance and distribution _information_ —for _all_ of the _extreme vectors_
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$ since each scale
factor $\psi_{1i\ast}$ satisfies the locus equation

$\psi_{1i\ast}=\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}-\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}\right)k_{\mathbf{x}_{1_{i\ast}}}\text{,}$

such that $k_{\mathbf{x}_{1_{j\ast}}}\equiv k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{j\ast}}}\equiv k_{\mathbf{x}_{2_{i\ast}}}$, so that the
scale factors $\psi_{1_{j\ast}}$ and $\psi_{2_{j\ast}}$ map covariance and
distribution information for correlated extreme vectors
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$ onto the extreme
vectors $k_{\mathbf{x}_{1_{j\ast}}}$ and $k_{\mathbf{x}_{2_{j\ast}}}$, at
which point the eigenvalue scaled dual locus of $\boldsymbol{\kappa}$

$\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}-\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}\right)$

contains _all_ of the covariance and distribution information for _all_ of the
extreme vectors $k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{i\ast}}}$—such that the covariance and distribution
information for any given extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ or
$k_{\mathbf{x}_{2_{i\ast}}}$ is represented by the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$—for a given collection
$\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}$ of feature vectors
$\mathbf{x}_{i}$.

Correspondingly, each scale factor $\psi_{2i\ast}$ is a function—of _all_ of
the covariance and distribution _information_ —for _all_ of the _extreme
vectors_ $k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$ since
each scale factor $\psi_{2i\ast}$ satisfies the locus equation

$\psi_{2i\ast}=\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}-\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}\right)k_{\mathbf{x}_{2_{i\ast}}}\text{,}$

such that $k_{\mathbf{x}_{1_{j\ast}}}\equiv k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{j\ast}}}\equiv k_{\mathbf{x}_{2_{i\ast}}}$, so that the
scale factors $\psi_{2_{j\ast}}$ and $\psi_{1_{j\ast}}$ map covariance and
distribution information for correlated extreme vectors
$k_{\mathbf{x}_{2_{i\ast}}}$ and $k_{\mathbf{x}_{1_{i\ast}}}$ onto the extreme
vectors $k_{\mathbf{x}_{2_{j\ast}}}$ and $k_{\mathbf{x}_{1_{j\ast}}}$, at
which point the eigenvalue scaled dual locus of $-\boldsymbol{\kappa}$

$\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}-\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}\right)$

contains _all_ of the covariance and distribution information for _all_ of the
extreme vectors $k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{i\ast}}}$—such that the covariance and distribution
information for any given extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ or
$k_{\mathbf{x}_{2_{i\ast}}}$ is represented by the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$—for a given collection
$\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}$ of feature vectors
$\mathbf{x}_{i}$.

###  15.4 Generating the Scale Factors

We now use (14.2), (15.4) and (15.5) to devise vector algebra locus equations
that determine values of the scale factors for the Wolfe-dual principal
eigenaxis components that lie on $\boldsymbol{\psi}$. Let there be
$l_{1}+l_{2}=l$ scale factors.

It will be seen that likely locations and likelihood values of extreme vectors
$\left\\{k_{\mathbf{x}_{1_{i\ast}}}\right\\}_{i=1}^{l_{1}}$ and
$\left\\{k_{\mathbf{x}_{2_{i\ast}}}\right\\}_{i=1}^{l_{2}}$ are
_statistically_ “ _pre-wired_ ” within the components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
of the principal eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$ and the inverted joint covariance matrix
$\mathbf{Q}^{-1}$ associated with the pair of random quadratic forms
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$.

####  15.4.1 Scale Factors for Class $\omega_{1}$

Let $i=1:l_{1}$, such that each extreme vector $k_{\mathbf{x}_{1i\ast}}$ that
belongs to class $\omega_{1}$ is correlated with a Wolfe-dual principal
eigenaxis component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{1}$ of the principal eigenvector
$\boldsymbol{\psi}_{\max}$. Now take the extreme vector
$k_{\mathbf{x}_{1i\ast}}$ that is correlated with the Wolfe-dual principal
eigenaxis component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$.

Using (14.2), (15.4) and (15.5), it follows that the scale factor
$\psi_{1i\ast}$ for the principal eigenaxis component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{1}$ is determined by the vector algebra
locus equation

(15.6) $\displaystyle\psi_{1i\ast}$
$\displaystyle=\lambda_{1}^{-1}\left\|k_{\mathbf{x}_{1_{i\ast}}}\right\|\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}\left\|k_{\mathbf{x}_{1_{j\ast}}}\right\|\cos\theta_{k_{\mathbf{x}_{1_{i\ast}}}k_{\mathbf{x}_{1_{j\ast}}}}$
$\displaystyle-\lambda_{1}^{-1}\left\|k_{\mathbf{x}_{1_{i\ast}}}\right\|\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}\left\|k_{\mathbf{x}_{2_{j\ast}}}\right\|\cos\theta_{k_{\mathbf{x}_{1_{i\ast}}}k_{\mathbf{x}_{2_{j\ast}}}}\text{,}$

such that $k_{\mathbf{x}_{1_{j\ast}}}\equiv k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{j\ast}}}\equiv k_{\mathbf{x}_{2_{i\ast}}}$, so that the
scale factors $\psi_{1_{j\ast}}$ and $\psi_{2_{j\ast}}$ map covariance and
distribution information for correlated extreme vectors
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$ onto the extreme
vectors $k_{\mathbf{x}_{1_{j\ast}}}$ and $k_{\mathbf{x}_{2_{j\ast}}}$ in such
a manner that the eigenvalue scaled dual locus of $\boldsymbol{\kappa}$

$\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}-\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}\right)$

is mapped onto the extreme vector $k_{\mathbf{x}_{1_{i\ast}}}$ in the
following manner

$\psi_{1i\ast}=\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}-\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}\right)k_{\mathbf{x}_{1_{i\ast}}}\text{,}$

so that the scaled geometric locus of the novel principal eigenaxis
$\lambda_{1}^{-1}\boldsymbol{\kappa}$ contains all of the covariance and
distribution information for all of the extreme vectors
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$—relative to the
eigenvalues $\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the
inverted joint covariance matrix $\mathbf{Q}^{-1}$ for a given collection
$\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}$ of feature vectors
$\mathbf{x}_{i}$ —at which point the likelihood component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
on side $\boldsymbol{\psi}_{1}$ contains equivalent covariance and
distribution information for the extreme point $\mathbf{x}_{1_{i\ast}}$ that
is normalized relative to length

$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}\equiv\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}-\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}\right)k_{\mathbf{x}_{1_{i\ast}}}\text{.}$

Thereby, the likely location of the extreme point $\mathbf{x}_{1_{i\ast}}$
within the decision space $Z=Z_{1}\cup Z_{2}$ of the minimum risk binary
classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is determined by the geometric locus of the principal eigenaxis component
$\psi_{1i\ast}k_{\mathbf{x}_{1i\ast}}$, whereas the likelihood value of the
extreme point $\mathbf{x}_{1_{i\ast}}$ is determined by the statistical
contents of the correlated likelihood component
$\psi_{1i\ast}k_{\mathbf{x}_{1i\ast}}$.

####  15.4.2 Scale Factors for Class $\omega_{2}$

Let $i=1:l_{2}$, such that each extreme vector $k_{\mathbf{x}_{2i\ast}}$ that
belongs to class $\omega_{2}$ is correlated with a Wolfe principal eigenaxis
component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{2}$ of the principal eigenvector
$\boldsymbol{\psi}_{\max}$. Now take the extreme vector
$k_{\mathbf{x}_{2i\ast}}$ that is correlated with the Wolfe-dual principal
eigenaxis component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$.

Using (14.2), (15.4) and (15.5), it follows that the scale factor
$\psi_{2i\ast}$ for the principal eigenaxis component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{2}$ is determined by the vector algebra
locus equation

(15.7) $\displaystyle\psi_{2i\ast}$
$\displaystyle=\lambda_{1}^{-1}\left\|k_{\mathbf{x}_{2_{i\ast}}}\right\|\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}\left\|k_{\mathbf{x}_{2_{j\ast}}}\right\|\cos\theta_{k_{\mathbf{x}_{2_{i\ast}}}k_{\mathbf{x}_{2_{j\ast}}}}$
$\displaystyle-\lambda_{1}^{-1}\left\|k_{\mathbf{x}_{2_{i\ast}}}\right\|\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}\left\|k_{\mathbf{x}_{1_{j\ast}}}\right\|\cos\theta_{k_{\mathbf{x}_{2_{i\ast}}}k_{\mathbf{x}_{1_{j\ast}}}}\text{,}$

such that $k_{\mathbf{x}_{1_{j\ast}}}\equiv k_{\mathbf{x}_{1_{i\ast}}}$ and
$k_{\mathbf{x}_{2_{j\ast}}}\equiv k_{\mathbf{x}_{2_{i\ast}}}$, so that the
scale factors $\psi_{2_{j\ast}}$ and $\psi_{1_{j\ast}}$ map covariance and
distribution information for correlated extreme vectors
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$ onto the extreme
vectors $k_{\mathbf{x}_{2_{j\ast}}}$ and $k_{\mathbf{x}_{1_{j\ast}}}$ in such
a manner that the eigenvalue scaled dual locus of $-\boldsymbol{\kappa}$

$\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}-\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}\right)$

is mapped onto the extreme vector $k_{\mathbf{x}_{2_{i\ast}}}$ in the
following manner

$\psi_{2i\ast}=\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}-\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}\right)k_{\mathbf{x}_{2_{i\ast}}}\text{,}$

so that the signed and scaled geometric locus of the novel principal eigenaxis
$-\lambda_{1}^{-1}\boldsymbol{\kappa}$ contains all of the covariance and
distribution information for all of the extreme vectors
$k_{\mathbf{x}_{1_{i\ast}}}$ and $k_{\mathbf{x}_{2_{i\ast}}}$—relative to the
eigenvalues $\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the
inverted joint covariance matrix $\mathbf{Q}^{-1}$ for a given collection
$\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{N}$ of feature vectors
$\mathbf{x}_{i}$ —at which point the likelihood component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
on side $\boldsymbol{\psi}_{2}$ contains equivalent covariance and
distribution information for the extreme point $\mathbf{x}_{2_{i\ast}}$ that
is normalized relative to length

$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}\equiv\lambda_{1}^{-1}\left(\sum\nolimits_{j=1}^{l_{2}}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}-\sum\nolimits_{j=1}^{l_{1}}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}\right)k_{\mathbf{x}_{2_{i\ast}}}\text{.}$

Thereby, the likely location of the extreme point $\mathbf{x}_{2_{i\ast}}$
within the decision space $Z=Z_{1}\cup Z_{2}$ of the minimum risk binary
classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$
is determined by the geometric locus of the principal eigenaxis component
$\psi_{2i\ast}k_{\mathbf{x}_{2_{i\ast}}}$, whereas the likelihood value of the
extreme point $\mathbf{x}_{2_{i\ast}}$ is determined by the statistical
contents of the correlated likelihood component
$\psi_{2i\ast}k_{\mathbf{x}_{2_{i\ast}}}$.

Given the locus equations in (15.6) and (15.7), we realize that likely
locations and likelihood values of extreme vectors
$\left\\{k_{\mathbf{x}_{1_{i\ast}}}\right\\}_{i=1}^{l_{1}}$ and
$\left\\{k_{\mathbf{x}_{2_{i\ast}}}\right\\}_{i=1}^{l_{2}}$ are
_statistically_ “ _pre-wired_ ” within the components
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
and
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
of the principal eigenvector $\boldsymbol{\psi}_{\max}$ of the joint
covariance matrix $\mathbf{Q}$ and the inverted joint covariance matrix
$\mathbf{Q}^{-1}$ associated with the pair of random quadratic forms
$\boldsymbol{\psi}^{T}\mathbf{Q}\boldsymbol{\psi}$ and
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$.

###  15.5 Regulation of Total Allowed Eigenenergy and Risk

Given the expressions satisfied by the scale factors $\psi_{1i\ast}$ and
$\psi_{2i\ast}$ for the dual components $\psi_{1i\ast}k_{\mathbf{x}_{1i\ast}}$
and $\psi_{2i\ast}k_{\mathbf{x}_{2_{i\ast}}}$ of the geometric locus of the
primal novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ in
(15.6) and (15.7), it follows that the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$ regulate the total
allowed eigenenergies
$\left\|\psi_{1_{i\ast}}k_{\mathbf{x}_{1i\ast}}\right\|_{\min_{c}}^{2}$ and
$\left\|\psi_{2_{i\ast}}k_{\mathbf{x}_{2i\ast}}\right\|_{\min_{c}}^{2}$
exhibited by the principal eigenaxis components
$\psi_{1i\ast}k_{\mathbf{x}_{1i\ast}}$ and
$\psi_{2i\ast}k_{\mathbf{x}_{2_{i\ast}}}$ on the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$, along
with the expected risks or counter risks
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\psi_{1_{i\ast}}k_{\mathbf{x}_{1i\ast}}\right\|_{\min_{c}}^{2}\right)$
and
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\psi_{2_{i\ast}}k_{\mathbf{x}_{2i\ast}}\right\|_{\min_{c}}^{2}\right)$
exhibited by the likelihood components $\psi_{1i\ast}k_{\mathbf{x}_{1i\ast}}$
and $\psi_{2i\ast}k_{\mathbf{x}_{2_{i\ast}}}$ on the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$.

Thereby, the total allowed eigenenergy
$\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}$
and the expected risk
$\mathfrak{R}_{\mathfrak{\min}}\left(\left\|\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}\right\|_{\min_{c}}^{2}\right)$
exhibited by the geometric locus of the novel principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}-\boldsymbol{\kappa}_{2}$ are
regulated by the eigenvalues
$\lambda_{N}^{-1}\leq\mathbf{\ldots}\leq\lambda_{1}^{-1}$ of the inverted
joint covariance matrix $\mathbf{Q}^{-1}$ of the random quadratic form
$\boldsymbol{\psi}^{T}\mathbf{Q}^{-1}\boldsymbol{\psi}$ .

We now devise normalized conditional density estimates for extreme points.

###  15.6 Normalized Conditional Density Estimates

Given (14.3) and (15.6), it follows that the geometric locus of each principal
eigenaxis component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{1}$ of the principal eigenvector
$\boldsymbol{\psi}_{\max}$ is determined by how the first and second order
vector components of a correlated extreme vector
$k_{\mathbf{x}_{1_{i_{\ast}}}}$ are symmetrically distributed along the
eigenvalue scaled loci of $l$ signed and scaled extreme vectors
$\left\\{\lambda_{1}^{-1}\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}\right\\}_{j=1}^{l_{1}}$
and
$\left\\{-\lambda_{1}^{-1}\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}\right\\}_{j=1}^{l_{2}}$,
such that each signed and scaled extreme vector
$\psi_{1_{j\ast}}k_{\mathbf{x}_{1_{j\ast}}}$ and
$-\psi_{2_{j\ast}}k_{\mathbf{x}_{2_{j\ast}}}$ represents a symmetrically
balanced distribution of the $l$ scaled extreme vectors
$\left\\{\psi_{{}_{k\ast}}k_{\mathbf{x}_{k\ast}}\right\\}_{k=1}^{l}$ that is
conditional on the entire collection
$\left\\{k_{\mathbf{x}_{i}}\right\\}_{i=1}^{N}$ feature vectors
$k_{\mathbf{x}_{i}}$.

Therefore, each principal eigenaxis component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{1}$ constitutes a normalized conditional
density estimate for a correlated extreme point
$\mathbf{x}_{1_{i\ast}}$—represented by a distribution of first and second
degree coordinates of the extreme point $\mathbf{x}_{1_{i\ast}}$—that is
determined by the locus equation in (15.6).

Thereby, each likelihood component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
on that lies on side $\boldsymbol{\psi}_{1}$ represents the distribution of a
correlated extreme point $\mathbf{x}_{1_{i\ast}}$ within the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
such that the likelihood component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
determines a conditional density estimate and a conditional likelihood
value—both of which are normalized relative to length—for the extreme point
$\mathbf{x}_{1_{i\ast}}$.

Correspondingly, given (14.3) and (15.7), it follows that the geometric locus
of each principal eigenaxis component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{2}$ of the principal eigenvector
$\boldsymbol{\psi}_{\max}$ constitutes a normalized conditional density
estimate for a correlated extreme point $\mathbf{x}_{2_{i\ast}}$—represented
by a distribution of first and second degree coordinates of the extreme point
$\mathbf{x}_{2_{i\ast}}$—that is determined by the locus equation in (15.7).

Thereby, each likelihood component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{2}$ represents the distribution of a
correlated extreme point $\mathbf{x}_{2_{i\ast}}$ within the decision space
$Z=Z_{1}\cup Z_{2}$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\boldsymbol{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
such that the likelihood component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
determines a conditional density estimate and a conditional likelihood
value—both of which are normalized relative to length—for the extreme point
$\mathbf{x}_{2_{i\ast}}$.

We are now in a position to devise conditional probability density functions
for class $\omega_{1}$ and class $\omega_{2}$.

###  15.7 Conditional Probability Density Functions

Given that each principal component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{1}$ also represents a likelihood
component
$\psi_{1i\ast}\frac{k_{\mathbf{x}_{1i\ast}}}{\left\|k_{\mathbf{x}_{1i\ast}}\right\|}$
that determines a normalized conditional density estimate and a normalized
conditional likelihood value for a correlated extreme point
$\mathbf{x}_{1_{i\ast}}$, it follows each principal eigenaxis component
$\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ that lies on side
$\boldsymbol{\kappa}_{1}$ also represents a likelihood component
$\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ that determines a conditional
density estimate and a conditional likelihood value for a correlated extreme
point $\mathbf{x}_{1_{i\ast}}$—both of which are conditional on all of the
likelihood components $\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$-\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ that lie on the novel principal
eigenaxis $\boldsymbol{\kappa}$.

Correspondingly, given that each principal component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
that lies on side $\boldsymbol{\psi}_{2}$ also represents a likelihood
component
$\psi_{2i\ast}\frac{k_{\mathbf{x}_{2i\ast}}}{\left\|k_{\mathbf{x}_{2i\ast}}\right\|}$
that determines a normalized conditional density estimate and a normalized
conditional likelihood value for a correlated extreme point
$\mathbf{x}_{2_{i\ast}}$, it follows that each principal eigenaxis component
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ that lies on side
$\boldsymbol{\kappa}_{2}$ also represents a likelihood component
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ that determines a conditional
density estimate and a conditional likelihood value for a correlated extreme
point $\mathbf{x}_{2_{i\ast}}$—both of which are conditional on all of the
likelihood components $\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ and
$-\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ that lie on the signed novel
principal eigenaxis $-\boldsymbol{\kappa}$.

Thereby, it follows that principal eigenaxis components
$\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ and correlated likelihood
components $\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$ and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ are identically and symmetrically
distributed over both sides $\boldsymbol{\kappa}_{1}$ and
$\boldsymbol{\kappa}_{2}$ of the geometric locus of the primal novel principal
eigenaxis

$\displaystyle\boldsymbol{\kappa}$
$\displaystyle=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}-\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$
$\displaystyle=\boldsymbol{\kappa}_{1}{\large-}\boldsymbol{\kappa}_{2}\text{,}$

so that the dual locus of
$\boldsymbol{\kappa}_{1}=\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$
is a conditional probability density function
$p\left(\mathbf{x}_{1_{i\ast}}|\boldsymbol{\kappa}_{1}\right)$ that determines
distributions of extreme points $\mathbf{x}_{1_{i\ast}}$ that are conditional
on the likelihood components $\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$and
$-\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ on $\boldsymbol{\kappa}$, and
the dual locus of
$\boldsymbol{\kappa}_{2}=\sum\nolimits_{i=1}^{l_{2}}\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$
is a conditional probability density function
$p\left(\mathbf{x}_{2_{i\ast}}|\boldsymbol{\kappa}_{2}\right)$ that determines
distributions of extreme points $\mathbf{x}_{2_{i\ast}}$ that are conditional
on the likelihood components $-\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}$and
$\psi_{2_{i\ast}}k_{\mathbf{x}_{2_{i\ast}}}$ on $-\boldsymbol{\kappa}$.

We now devise conditional probability functions for class $\omega_{1}$ and
class $\omega_{2}$.

###  15.8 Conditional Probability Function for Class $\omega_{1}$

Let the side $\boldsymbol{\kappa}_{1}$ of the geometric locus of the novel
principal eigenaxis
$\boldsymbol{\kappa}=\boldsymbol{\kappa}_{1}{\large-}\boldsymbol{\kappa}_{2}$
be a conditional probability density function
$p\left(\mathbf{x}_{1_{i\ast}}|\boldsymbol{\kappa}_{1}\right)$ for class
$\omega_{1}$, such that feature vectors $\mathbf{x}_{1_{i}}\in\mathbb{R}^{d}$
that belong to class $\omega_{1}$ are generated by a certain probability
density function $\mathbf{\mathbf{x}_{1_{i}}\sim}$
$p\left(\mathbf{x};\omega_{1}\right)$.

Then the integral of the conditional probability density function
$p\left(\mathbf{x}_{1_{i\ast}}|\boldsymbol{\kappa}_{1}\right)$ for class
$\omega_{1}$

$\displaystyle P\left(\mathbf{x}_{1_{i\ast}}|\boldsymbol{\kappa}_{1}\right)$
$\displaystyle=\int_{Z}\left(\sum\nolimits_{i=1}^{l_{1}}\psi_{1_{i\ast}}k_{\mathbf{x}_{1_{i\ast}}}\right)d\boldsymbol{\kappa}_{1}=\int_{Z}p\left(k_{\mathbf{x}_{1i\ast}}|\boldsymbol{\kappa}_{1}\right)d\boldsymbol{\kappa}_{1}$
$\displaystyle=\int_{Z}\boldsymbol{\kappa}_{1}d\boldsymbol{\kappa}_{1}=\int_{Z_{1}}\boldsymbol{\kappa}_{1}d\boldsymbol{\kappa}_{1}+\int_{Z_{2}}\boldsymbol{\kappa}_{1}d\boldsymbol{\kappa}_{1}$
$\displaystyle=\frac{1}{2}\left\|\boldsymbol{\kappa}_{1}\right\|^{2}+C=\left\|\boldsymbol{\kappa}_{1}\right\|^{2}+C_{1}\text{,}$

over the decision space $Z$ of the minimum risk binary classification system
$k_{\mathbf{s}}\boldsymbol{\kappa}+\mathbf{\kappa}_{0}\overset{\omega_{1}}{\underset{\omega_{2}}{\gtrless}}0$,
determines the conditional probability
$P\left(\mathbf{x}_{1_{i\ast}}|\boldsymbol{\kappa}_{1}\right)$ of observing a
set $\left\\{\mathbf{x}_{1_{i\ast}}\right\\}_{i=1}^{l_{1}}$ of $l_{1}$ extreme
points $\mathbf{x}_{1_{i\ast}}$ located within localized areas of the decision
space $Z=Z_{1}\cup Z_{2}$ of the system, so that the probability of finding an