#  Bucketed PCA Neural Networks with Neurons Mirroring Signals ††thanks:
Contact Email<EMAIL_ADDRESS>URL: alum.mit.edu/www/jhshen

Jackie Jianhong Shen

Financial Services
New York City, USA

###### Abstract

The bucketed PCA neural network (PCA-NN) with transforms is developed here in
an effort to benchmark deep neural networks (DNN’s), for problems on
supervised classification. Most classical PCA models apply PCA to the entire
training data set to establish a reductive representation and then employ non-
network tools such as high-order polynomial classifiers. In contrast, the
bucketed PCA-NN applies PCA to individual buckets which are constructed in two
consecutive phases, as well as retains a genuine architecture of a neural
network. This facilitates a fair apple-to-apple comparison to DNN’s, esp. to
reveal that a major chunk of accuracy achieved by many impressive DNN’s could
possibly be explained by the bucketed PCA-NN (e.g., 96% out of 98% for the
MNIST data set as an example). Compared with most DNN’s, the three building
blocks of the bucketed PCA-NN are easier to comprehend conceptually - PCA,
transforms, and bucketing for error correction. Furthermore, unlike the
somewhat quasi-random neurons ubiquitously observed in DNN’s, the PCA neurons
resemble or mirror the input signals and are more straightforward to decipher
as a result.

Keywords: Interpretable AI, Supervised Learning, Neurons, DNN, Mirroring, PCA,
Transforms, Bucketing, Error Correction, MNIST Data, Benchmark Models, SR
11-7.

Attention: The current work is designed to be published exclusively in the
Social Sciences Research Network (SSRN) or arXiv preprint server. Its
commercial or open-journal publication is prohibited without the prior consent
of the author.

###### Contents

  1. 1 Introduction
  2. 2 Some Quasi Motivations from Neuroscience
  3. 3 Data Set
  4. 4 The Bucketed PCA Neural Network
    1. 4.1 Raw PCA Neurons and Neural Networks
      1. 4.1.1 PCA of Digit Images
      2. 4.1.2 PCA Neurons and Networks
      3. 4.1.3 Performance of the Raw PCA Neural Network
    2. 4.2 Neuron Transforms
      1. 4.2.1 Signal Transforms
      2. 4.2.2 Duality and Neuron Transforms
      3. 4.2.3 Transformed PCA-NN
      4. 4.2.4 Performance of the Transformed PCA-NN
    3. 4.3 Bucketing and Error Correction
      1. 4.3.1 Systematic Factors behind Erroneous Buckets
      2. 4.3.2 The Bucketed PCA-NN for Error Correction
      3. 4.3.3 Performance of the Bucketed PCA-NN, and the Transformed
  5. 5 Conclusion

##  1 Introduction

We propose a neural network system directly based on the principal component
analysis (PCA) for supervised classification. The three main components are:

  1. (a)

direct neuron construction via PCA,

  2. (b)

neuron transformation, and

  3. (c)

error correction via bucketed PCA.

They are universally applicable to problems in supervised classification,
though the second component on transformation may have to depend on the
specific characteristics of the input signal classes.

There are two primary objectives of developing such non-mainstream neural
networks.

  1. (A)

To benchmark mainstream deep neural networks (DNN) [2, 4]. Traditional
parametric models, e.g., linear or logistic regressions, can also offer
benchmarking to DNN’s but are often less insightful due to major differences
in architecture. PCA neural networks (PCA-NN) explored in this work are
genuine neural networks although constructed very differently from the
mainstream DNN’s. Model benchmarking is a crucial tool for effective model
validation, e.g., as in the dogmatic supervisory statement “SS 11-7” by the
United States Federal Reserve Board, for validating all models adopted by the
financial industry in the USA.

Practitioners are often impressed by the high accuracy rates of DNN’s but also
keep wondering how much can be directly interpreted away from the somewhat
black-box nature of DNN’s. The current work shows that at least up to 96.00%
of the accuracy rate can be interpreted by the bucketed PCA-NN framework, as
applied to the celebrated MNIST digit data set.

  2. (B)

To construct neural networks with neurons that are much easier to decipher.
This is pertinent to the new initiative of creating “interpretable AI.” The
author, however, is not in a position to defend or denounce such informal
movement.

In the financial industry for instance, from many Chief Risk Officers for
Models (CRO-M’s) to front desk quantitative traders, the demand for some level
of interpretability is almost ubiquitously on the table. Should a trader sell
a very illiquid municipal bond at the price tag of $94.15 as projected by a
DNN for a notional of $600 MM? When the entire open market has not observed or
reported a single trade for three days, can a DNN speak a bit of human
language to explain how $94.15 has been derived? When the financial stake is
high, the human mind seems to habitually seek comfort from rationality and
interpretability.

The neurons in a typical DNN often appear quasi-random in terms of spatial or
correlation patterns, and do not possess the look and feel that resonate with
the input signals or their intrinsic characteristics (e.g., see Figure 1). In
contrast, the PCA neurons naturally mirror the common structures (or the most
significant as expressed by variances or principal values) of the input
signals and their population variations.

PCA [9, 10] has found numerous applications in machine learning. In most
classical works (e.g., [7, 8]), PCA offers a universal way to project complex
ensembles of signals onto lower-dimensional feature spaces. Further
classification schemes are then applied in these simplified spaces, e.g.,
quadratic or more general polynomial classifiers. The PCA-NN explored herein
differs in three aspects.

  1. (1)

First, one primary objective of the PCA-NN is to preserve the genuine
architecture of a neural network. This allows apple-to-apple benchmarking with
DNN’s. As a result, PCA is not merely used as a tool to project onto lower-
dimensional feature spaces, but also as a tool to produce interpretable
neurons. These neurons can then be transformed under a generic framework.

  2. (2)

The second salient feature of the PCA-NN is that PCA is not applied outright
to the entire training set, but only to individual “buckets” and in two
sequential phases. The second phase is of particular interest since it
achieves the function of error correction, as analogous to error correction in
Kalman filters.

  3. (3)

Finally, the PCA-NN requires no extra non-network classifying schemes such as
the polynomial classifiers. PCA-NN maintains the genuine architecture of a
neural network, i.e., only involving weighted sums (as a classical model of
perceptual neurons), univariate “activation” functions, and max and argmax.

In the current work, the bucketed PCA-NN has been illustrated entirely on the
MNIST data set of handwritten digits. It achieves a final accuracy score of
$96.00^{+}$%, but with three fully interpretable components of design. its
performance on other data sets may depend on the specific characteristics of
the signal classes. The framework of PCA-NN, however, should be universally
applicable to classification problems. At a minimum, it shall always provide a
genuine benchmarking neural network to DNN’s, as well as unveil a major chunk
of the accuracy rate achieved by DNN’s.

For the rest of the paper, the three components in the bucketed PCA-NN and
their performance will be presented in full details.

Finally, we emphasize that the current work is only based on low-level vision,
i.e., relying only on pixels and their gray values. More robust classification
frameworks have to be based on high-level vision toolkit such as hierarchical
structuring or context-driven grammars (e.g., [5]).

##  2 Some Quasi Motivations from Neuroscience

One important aspect of an interpretable neural network is to construct
neurons that are easy to understand. Such neurons usually capture or mirror
certain characteristics of the input signal classes.

Two results in neuroscience might support the effort of seeking neurons or
network designs that mirror or are directly “shaped” by stimulus signals. Such
a view, however, remains more speculative than scientific unless further
neuroscientific evidences are unveiled.

  1. (1)

Mirror neurons or systems are ones that can be activated at the sight or
hearing of other people’s selected actions, though it is still a developing
and debating subject. Actions like yawning, for example, are surprisingly
contagious among people confined in a small space.

Such scenes are often highly selective among hundreds of thousands of image
frames projected onto the retinas each day. The almost automated and
subconscious mirroring reactions may suggest that the human brain codes and
stores these specific scenes or related features directly, instead of relying
on generic neural wiring or systems.

One speculative analogy is to store a unit feature vector
${\boldsymbol{w}}=(w_{1},\ldots,w_{m})\in{\mathbb{R}}^{m}$ in the canonical
Euclidean space, as a neuron or its weights. When any (potentially pre-
processed) unit vector
${\boldsymbol{v}}=(v_{1},\ldots,v_{m})\in{\mathbb{R}}^{m}$ emerges as a
stimulus signal, the magnitude of the response $z$ via the Euclidean inner
product -

$z=\big{\langle}{\boldsymbol{w}},\,{\boldsymbol{v}}\big{\rangle}=\sum_{j=1}^{m}w_{j}*v_{j},$

can immediately indicate if the pattern ${\boldsymbol{w}}$ has been observed
in ${\boldsymbol{v}}$ (say, via the shifted Heaviside activation
$H(z-(1-\delta))$ for some tolerance threshold $\delta$). This is done in the
genuine fashion of an artificial neuron, i.e., via a weighted sum and an
activation modulation. More remarkably, the “neuron” ${\boldsymbol{w}}$
stores, encodes, or mirrors the type of signals it is supposed to capture.

  2. (2)

The discoveries of the Nobel Laureates Hubel and Wiesel [3] on the behavior of
the simple cells of cats. Unlike the quasi-random patterns of most
artificial/computational neurons, these neurons are only activated by certain
regular spatial patterns, i.e., by specific orientations and spatial
oscillations of input stimuli. The summation behavior (on activation and
prohibition) in their visual fields suggests that these neurons more or less
mirror the edges, edgelets, or linear oscillations that exist ubiquitously in
natural scenes and are crucial bottom-up visual cues for high-level visual
recognition [1, 6]. Such findings support the speculation that neurons can be
“shaped” by nature and may directly encode or mirror frequent and important
external stimuli.

##  3 Data Set

The framework of the bucketed PCA-NN is universally applicable to problems on
supervised classification. To develop the main ideas, we focus on the
celebrated digit-recognition data set by NIST (National Institute of Standards
and Technology, USA). Due to some modifications, it is now commonly referred
to as the MNIST data set.

The MNIST data set is summarized as follows.

  1. (1)

Each data point is a monochrome image of 28 by 28 pixels, capturing a
handwritten digit from $\\{0,1,\ldots,9\\}$.

  2. (2)

There are totally 60,000 train images, and 10,000 test images.

  3. (3)

In Python, based on Tensorflow and Keras it can be loaded via a command line
like:

⬇

from keras.datasets import mnist

(trainImages, trainLabels), (testImages, testLabels) \

= mnist.load_data()

Using two layers of neurons of sizes (400, 10) with full connectivity and some
other common network features, a DNN could achieve an impressive accuracy rate
of 98.15% for the test data set.

The first 10 neurons for such a DNN implementation are plotted as monochrome
images in Figure 1, using weights as pixel values. As common to DNN’s, the
individual neurons do not relate intuitively well to the input handwritten
digits. They demonstrate some quasi-random patterns produced by the weights
optimization procedure.

Figure 1:  The first 10 neurons of a typical DNN implementation for MNIST. The
somewhat quasi-random patterns resulted from weights optimization of DNN do
not relate well to the input signal classes, e.g., the 10 digits in the
current study.

##  4 The Bucketed PCA Neural Network

As mentioned earlier in the Introduction, readers should be aware of the
difference from the earlier PCA works (e.g., [7, 8]), where PCA components are
first constructed from the entire training data set and then a high-order
polynomial classifier is deployed based on the reductive PCA representation.
In the current work, PCA is applied to each bucket of training samples, and
bucketing takes two consecutive steps. Furthermore, it also intends to
maintain the authentic architecture of neural networks in order to make fair
apple-to-apple comparison with DNN’s.

To summarize the entire effort, the current work helps reveal how much of
DNN’s impressive accuracy rates can be interpreted by classical frameworks
like the present, e.g., at least 96.0% out of 98.0% for the MNIST data set.

###  4.1 Raw PCA Neurons and Neural Networks

####  4.1.1 PCA of Digit Images

Let $\Omega$ denote the collection of training samples, e.g., the 60,000 digit
images in MNIST. A generic individual sample is denoted by ${\boldsymbol{x}}$.
Let $\alpha\in{\mathcal{A}}$ denote all supervised labels or classes, i.e.,

${\boldsymbol{x}}\in\Omega\longrightarrow\alpha=\phi({\boldsymbol{x}})\in{\mathcal{A}},$

where $\phi$ denotes the supervision. For MNIST, ${\mathcal{A}}$ consists of
the 10 single digits:

${\mathcal{A}}=\\{\text{`0'},\text{`1'},\ldots,\text{`9'}\\}.$

Given a supervision $\phi$, the sample set can be partitioned into single-
label buckets $\Omega_{\alpha}$’s:

$\Omega=\bigcup_{\alpha\in{\mathcal{A}}}\Omega_{\alpha},\qquad\mbox{with}\quad\Omega_{\alpha}=\\{{\boldsymbol{x}}\in\Omega\mid\phi({\boldsymbol{x}})=\alpha\\}.$

For MNIST, each sample data point is a monochrome 2D image of a handwritten
digit:

${\boldsymbol{x}}\in\Omega\subset[0,1]^{28\times
28}\subset{\mathbb{R}}^{28\times 28}.$

A pixel value $x_{ij}=1$ stands for the brightest gray scale while $x_{ij}=0$
for the dimmest. The background against which digits are handwritten is
typically zero or close to.

Since the perception of a digit is gray-scale invariant, in the current work
each image ${\boldsymbol{x}}\in\Omega$ is first normalized to a unit vector:

${\boldsymbol{x}}\longrightarrow\frac{{\boldsymbol{x}}}{{\left\|{\boldsymbol{x}}\right\|_{2}}},\qquad\mbox{with}\quad{\left\|{\boldsymbol{x}}\right\|_{2}}^{2}=\mathrm{trace}({\boldsymbol{x}}{\boldsymbol{x}}^{T})=\sum_{i,j}x_{ij}^{2}.$

In this way, all sample digit images in MNIST live on the unit hyper-sphere
${\mathbb{S}}^{28\times 28-1}$. This helps improve both data visualization and
the interpretibility of the PCA approach (See Figure 2 for a visual
illustration).

Figure 2:  After normalization, all digit images live on the unit hyper-sphere
${\mathbb{S}}^{28\times 28-1}$.

In order to call the numerical SVD decomposor in Python:

⬇

numpy.linalg.svd(),

one linearly stretches the 2D image arrays to 1D row vectors via:

${\boldsymbol{x}}\longrightarrow X={\boldsymbol{x}}.\mathrm{reshape}(28\times
28).$

For any given digit $\alpha\in{\mathcal{A}}$, the associated sample subspace
$\Omega_{\alpha}$ can be numerically enumerated as a
$|\Omega_{\alpha}|\times(28\times 28)$ matrix:

$M_{\alpha}=\begin{pmatrix}X_{1}\\\ X_{2}\\\ \vdots\\\
X_{|\Omega_{\alpha}|}\end{pmatrix}$

Calling the SVD decomposer, the subspace is decomposed via:

$M_{\alpha}=U^{T}\cdot D\cdot V.$

Here $D=\mathrm{Diag}(\sigma_{1},\ldots,\sigma_{28\times 28})$ stores all the
singular values in descending order, and $V$ stores an orthonormal basis in
${\mathbb{R}}^{28\times 28}$ (as row vectors for convenience):

$V=\\{V_{i}\mid 1\leq i\leq 28\times 28\\}.$

Notice that $|\Omega_{\alpha}|\gg 28\times 28$ for MNIST. Finally, for any
image sample ${\boldsymbol{x}}$ or $X$ in $\Omega_{\alpha}$,

$X\in\mathrm{span}\\{V_{1},\ldots,V_{28\times 28}\\}.$

####  4.1.2 PCA Neurons and Networks

As typical for many real-world signals, the singular values decay to zero
fast, which gives rise to the most common way of weeding out noises from real
signal structures [9, 10]. Let $r\in(0,100\%)$ denote a tail truncation
parameter, e.g., $r=20\%$. For a given PCA with $K$ descending singular
values, define the $r$-truncation index: $K_{r}=K$ if $\sigma_{K}^{2}>r$, and
otherwise,

$K_{r}=\min\left\\{k:\;\sum_{i=k+1}^{K}\sigma_{i}^{2}\leq r\right\\}.$

For digit classification, let $K_{r}(\alpha)$ denote the $K_{r}$ index for the
PCA associated with the matrix $M_{\alpha}$ as defined above (for
$\Omega_{\alpha}$). Then define the neuron set under a given truncation level
$r$ by:

${\mathcal{N}}_{\alpha}=\left\\{{\boldsymbol{v}}_{k}=V_{k}.\mathrm{reshape}((28,28)):\;1\leq
k\leq K_{r}(\alpha)\right\\}.$ (1)

Each neuron in ${\mathcal{N}}_{\alpha}$ is generically denoted by
${\boldsymbol{v}}^{(\alpha)}_{k}$ to indicate the $\alpha$-dependency. Also
the neurons have been reshaped to the original image dimension of $28$ by $28$
pixels.

The raw PCA neural network (PCA-NN) is built upon all these neurons:

${\mathcal{N}}=\bigcup_{\alpha\in{\mathcal{A}}}\cal N_{\alpha}.$

The total number of neurons is:

$|{\mathcal{N}}|=\sum_{\alpha\in{\mathcal{A}}}|N_{\alpha}|=\sum_{\alpha\in{\mathcal{A}}}K_{r}(\alpha),$

under a given tail truncation parameter $r$.

The PCA-NN is comprised of the following two layers.

  1. (A)

(First Layer) On the frontal layer directly facing an input image sample
${\boldsymbol{x}}$, the response of neuron ${\boldsymbol{v}}^{(\alpha)}_{k}$
is the weighted sum or the canonical Euclidean inner product:

$z_{k}^{(\alpha)}=\big{\langle}{\boldsymbol{x}},\,{\boldsymbol{v}}^{(\alpha)}_{k}\big{\rangle}=\sum_{1\leq
i,j\leq 28}v^{(\alpha)}_{k,ij}\cdot x_{ij}.$ (2)

For each such neuron, the activation function is a square: $\psi(z)=z^{2}$.

  2. (B)

(Second Layer) The second layer consists of $|{\mathcal{A}}|=10$ aggregating
neurons: for each $\alpha\in{\mathcal{A}}$, the neuron $u_{\alpha}$ computes
the following response:

$u_{\alpha}=\sum_{1\leq k\leq
K_{r}(\alpha)}1.0*\psi\left(z_{k}^{(\alpha)}\right)=\sum_{1\leq k\leq
K_{r}(\alpha)}\big{\langle}{\boldsymbol{x}},\,{\boldsymbol{v}}^{(\alpha)}_{k}\big{\rangle}^{2}.$
(3)

In term of weights on the outputs of the first layer, it can also be written
as:

$u_{\alpha}=\sum_{\beta\in{\mathcal{A}},1\leq k\leq
K_{r}(\beta)}w_{\beta,k}^{(\alpha)}\cdot\psi\left(z_{k}^{(\beta)}\right),$ (4)

where $\displaystyle w_{\beta,k}^{(\alpha)}=1.0$ when $\beta=\alpha$, and 0
otherwise.

The final classifier for a given sample ${\boldsymbol{x}}$ is directly defined
by the argmax:

$\hat{\alpha}({\boldsymbol{x}})=\mathrm{argmax}\\{u_{\alpha}:\;\alpha\in{\mathcal{A}}\\}.$
(5)

Figure 3:  The raw PCA Neural Network (PCA-NN). When the PCA tail truncation
is set to $r=20.0\%$ for instance, the average $K_{r}(\alpha)$ is only about
13 (neurons) for MNIST. In contrast with most classical PCA works, PCA-NN
applies PCA to each bucket, instead of the entire training set. It also relies
only on the genuine neural network aggregator or synthesizer $\max$ or
$\mathrm{argmax}$, instead of separate high-order polynomial classifiers. Also
stay tuned that these raw buckets will be further refined.

####  4.1.3 Performance of the Raw PCA Neural Network

We call such a neural network the raw PCA-NN. The performance of the raw PCA-
NN is moderate.

  1. (a)

The (out-of-sample) accuracy rate for the 10,000 MNIST test images is 93.49%.

  2. (b)

The (in-sample) accuracy rate for the 60,000 MNIST training images is 93.50%.

While the performance may seem less appetizing compared with DNN’s, the raw
PCA-NN does possess the genuine 2-layer neural network structure and hence
offers an apple-to-apple benchmark to the DNN’s and their impressive
performance. It shows at least that a major chunk (e.g., 93.50% out of 98.0%+)
of DNN’s performance can be easily interpretable. This will be further
improved later on.

The raw PCA-NN does not explicitly rely on forward or backward optimization.
Instead, it only relies on the PCA decomposer, which itself is certainly an
implicit classical optimizer - seeking the least-dimensional subspaces where
the samples are maximally concentrated. For instance, in the case of setting a
tail truncation level at $r=20\%$ (i.e., retaining the 80% of the principal
spectra), the average dimension for the 10 digits is only about 13, i.e.,
seeking 13 neurons and their spanned spaces to efficiently represent the key
features of about 6,000 samples for each digit.

The first 4 PCA neurons for digit “8” have been plotted in Figure 4 and can be
easily interpreted as demonstrated in the caption.

Figure 4:  The first 4 PCA neurons ${\boldsymbol{v}}_{1:4}^{(\alpha)}$ for
digit $\alpha=8$. Unlike the quasi-random patterns of neurons in most DNN’s
(e.g., Figure 1), these neurons are easily interpretable. The first one (from
the left) bears the highest principal value and captures the average or most
common features of the given digit. It corresponds to the clustering center of
a given digit population on the hyper-sphere, as illustrated in Figure 2. The
second one captures the angular or rotational variations in handwriting. The
third neuron captures horizontal displacement or thickening in handwriting,
while the fourth neuron captures vertical displacement or variation.

###  4.2 Neuron Transforms

####  4.2.1 Signal Transforms

Neuron transform is a way to recycle existing neurons for extracting further
signal features. Convolutional neural network [2], for instance, is a well-
known example in which a base neuron or set is transformed via a series of
translations to participate in the same network.

Transforms are certainly signal specific, and different signal classes may
embrace different transforms. Following the earlier notation, let $\Omega$
denote the signal sample space, ${\boldsymbol{x}}\in\Omega$ a generic sample,
and ${\mathcal{A}}$ the label set. Let $\phi:\Omega\to{\mathcal{A}}$ denote
the ground-truth supervision or classification. A classification is said to be
invariant under a transform group (or collection) ${\mathsf{G}}$, if

$\phi({\mathsf{g}}{\boldsymbol{x}})=\phi({\boldsymbol{x}}),\qquad\mbox{for
any}\quad{\boldsymbol{x}}\in\Omega,\;{\mathsf{g}}\in{\mathsf{G}}.$ (6)

For digit recognition, the 2D Euclidean group ${\mathsf{E}}$ over
${\mathbb{R}}^{2}$ naturally comes to mind. Let
${\boldsymbol{q}}=(z_{1},z_{2})$ denote an arbitrary pixel in
${\mathbb{R}}^{2}$. Any pair of data

${\mathsf{g}}=({\boldsymbol{a}},Q)=({\boldsymbol{a}}\in{\mathbb{R}}^{2},\quad
Q\;\mbox{a 2D orthogonal matrix})$

naturally defines a transform on any given image
${\boldsymbol{x}}={\boldsymbol{x}}({\boldsymbol{q}})$ by:

${\mathsf{g}}{\boldsymbol{x}}({\boldsymbol{q}}):={\boldsymbol{x}}(Q^{-1}{\boldsymbol{q}}+{\boldsymbol{a}}),\qquad\mbox{i.e.,
moving the image in rigid motions.}$

But common sense reminds us the following.

  1. (i)

Digit recognition is not reflection invariant. These three digits
$\\{0,1,8\\}$ are approximately reflection invariant under horizontal or
vertical reflections. But all the rest are not.

  2. (ii)

Digit recognition is rotationally invariant only to certain degrees. For
instance, when the digit “6” is rotated by 180 degrees, it actually becomes
another digit “9.” Similarly, the digit “1” only becomes the Mandarin
character for “1” after a 90-degree rotation.

In addition, the MNIST data set has largely been normalized by framing all
digits around the center of the image domain. Hence translation (via
${\boldsymbol{a}}$ in the above definition) also would not make much
contribution. Such reckoning suggests that one should focus only on small
rotations $Q=R_{\theta}$, with $|\theta|\leq U$:

$R_{\theta}{\boldsymbol{q}}=R_{\theta}(z_{1},z_{2})=(z_{1}\cos\theta-
z_{2}\sin\theta,\;z_{1}\sin\theta+z_{2}\cos\theta).$

This translates to the rotation operator $R_{\theta}$ on any image sample
${\boldsymbol{x}}={\boldsymbol{x}}(z_{1},z_{2})$:

$R_{\theta}{\boldsymbol{x}}({\boldsymbol{q}}):={\boldsymbol{x}}(R_{-\theta}{\boldsymbol{q}}),\qquad\mbox{or}\quad
R_{\theta}{\boldsymbol{x}}(x,y):={\boldsymbol{x}}(z_{1}\cos\theta+z_{2}\sin\theta,\;-z_{1}\sin\theta+z_{2}\cos\theta).$
(7)

For instance, $R_{20^{\circ}}{\boldsymbol{x}}$ rotates an image
${\boldsymbol{x}}$ by $20^{\circ}$ counterclockwise.

The angle range $U$ can be empirically defined, e.g., $U=30^{\circ}$. In
reality, people’s writing habits, including left-right handedness and hand-
paper alignment, result in the natural dispersion of the orientations of a
given digit within a moderate range.

####  4.2.2 Duality and Neuron Transforms

For the purpose of discussion, it is assumed that image samples
${\boldsymbol{x}},{\boldsymbol{y}}$ are defined on the entire 2D visual field
$(z_{1},z_{2})\in{\mathbb{R}}^{2}$. The reception of an input image
${\boldsymbol{x}}$ by a neuron ${\boldsymbol{v}}$ is defined by the weighted
sum, or more formally, the inner product in $L^{2}({\mathbb{R}}^{2})$:

$\big{\langle}{\boldsymbol{x}},\,{\boldsymbol{v}}\big{\rangle}=\int_{{\mathbb{R}}^{2}}{\boldsymbol{x}}(z_{1},z_{2}){\boldsymbol{v}}(z_{1},z_{2})\;dz_{1}dz_{2}=\int_{{\mathbb{R}}^{2}}{\boldsymbol{x}}({\boldsymbol{q}}){\boldsymbol{v}}({\boldsymbol{q}})\;d{\boldsymbol{q}}.$

The duality for bounded linear operators refers to:

$\big{\langle}R_{\theta}{\boldsymbol{x}},\,{\boldsymbol{v}}\big{\rangle}=\int_{{\mathbb{R}}^{2}}{\boldsymbol{x}}\left(R_{-\theta}{\boldsymbol{q}}\right){\boldsymbol{v}}({\boldsymbol{q}})d{\boldsymbol{q}}=\int_{{\mathbb{R}}^{2}}{\boldsymbol{x}}\left(\hat{\boldsymbol{q}}\right){\boldsymbol{v}}(R_{\theta}\hat{\boldsymbol{q}})d\hat{\boldsymbol{q}}=\big{\langle}{\boldsymbol{x}},\,R_{-\theta}{\boldsymbol{v}}\big{\rangle},$
(8)

or simply $R_{\theta}^{\ast}=R_{-\theta}$ in $L^{2}({\mathbb{R}}^{2})$. Such
duality is standard in functional analysis but can be very useful for neural
network design for the following reasons.

  1. (a)

There is no need in the network to perform direct transforms on input signals.
Instead, under the duality principal, it is sufficient to only insert new
neurons transformed from existing ones.

  2. (b)

The duality principal even extends to more general transforms:

${\mathsf{g}}{\boldsymbol{x}}({\boldsymbol{q}}):={\boldsymbol{x}}(\Phi^{-1}{\boldsymbol{q}}),$

where $\Phi(\cdot)$ is a diffeomorphism of the 2D visual field
${\mathbb{R}}^{2}$ (e.g., in the context of deformable templates in [5]). It
can be easily shown that:

$\big{\langle}{\mathsf{g}}{\boldsymbol{x}}({\boldsymbol{q}}),\,{\boldsymbol{v}}({\boldsymbol{q}})\big{\rangle}=\big{\langle}{\boldsymbol{x}}(\hat{\boldsymbol{q}}),\,{\mathsf{g}}^{\ast}{\boldsymbol{v}}(\hat{\boldsymbol{q}})\big{\rangle},\qquad\mbox{with}\quad{\mathsf{g}}^{\ast}{\boldsymbol{v}}(\hat{\boldsymbol{q}}):={\boldsymbol{v}}(\Phi\hat{\boldsymbol{q}})\cdot
J_{\hat{\boldsymbol{q}}}\Phi,$ (9)

where $J$ is the Jacobian volume multiplier in ${\mathbb{R}}^{2}$. Then to
properly capture $\Phi$-transformed image signals, it is sufficient to only
insert the dual neurons ${\mathsf{g}}^{\ast}{\boldsymbol{v}}$ into the
existing neural networks.

####  4.2.3 Transformed PCA-NN

In actual numerical implementation, once a set of transforms ${\mathsf{g}}\in
G$ have been chosen, following the earlier settings in Eqn. (1), (2), and (4),
one can augment the raw PCA-NN by the following approach.

  1. (a)

For each transform ${\mathsf{g}}$, under duality each neuron
${\boldsymbol{v}}_{k}^{(\alpha)}$ is transformed to a new one
$\displaystyle{\mathsf{g}}^{\ast}{\boldsymbol{v}}_{k}^{(\alpha)}$, denoted by
$\displaystyle{\boldsymbol{v}}_{k}^{({\mathsf{g}},\alpha)}$ for convenience.
Then each neural set ${\mathcal{N}}_{\alpha}$ defined in Eqn. (1) is
transformed accordingly to ${\mathcal{N}}^{({\mathsf{g}})}_{\alpha}$.

  2. (b)

For a given sample signal ${\boldsymbol{x}}\in\Omega$, following Eqn. (2) and
(4), one obtains the responses of the first and second layers:

$z_{k}^{({\mathsf{g}},\alpha)},\quad u_{\alpha}^{({\mathsf{g}})}.$

  3. (c)

Then for each label or class $\alpha\in{\mathcal{A}}$, the final
$\alpha$-score for a given signal ${\boldsymbol{x}}$ is:

$U_{\alpha}=\max\\{u_{\alpha}^{({\mathsf{g}})}:\;{\mathsf{g}}\in G\\}.$

For convenience, it is assumed that the transform set $G$ also contains the
identity transform so that the original raw PCA-NN response $u_{\alpha}$ is
also included. Then the final classifier $\hat{\alpha}_{G}$ is defined
similarly as in Eqn. (5):

$\hat{\alpha}_{\tiny
G}({\boldsymbol{x}})=\mathrm{argmax}\\{U_{\alpha}:\;\alpha\in{\mathcal{A}}\\}.$
(10)

We call this augmented neural network the $G$-transformed PCA-NN.

Figure 5:  The transformed PCA-NN, as illustrated via a single (non-identity)
transform ${\mathsf{g}}\in G$. (See Eqn.(10) and also compare with Figure 3.)
The transformed neurons can capture transformed input signals without directly
turning to inverse transforms on the signals, while maintaining the genuine
architecture of a neural network.

####  4.2.4 Performance of the Transformed PCA-NN

Once a range of angles $(-U,U)$ is set (e.g., $U=30^{\circ}$ as mentioned
earlier), one can discretize it via a chosen gap, e.g.,
$\Delta\theta=3^{\circ}$ or $5^{\circ}$:

$\theta_{-M}<\cdots<\theta_{-1}<\theta_{0}=0.0<\theta_{1}<\cdots<\theta_{M}.$

Here $\theta_{0}=0.0$ corresponds to the identity operator, i.e., original raw
PCA-NN. It is possible to select the best combination simply via examining the
associated classification accuracy. For the present work, we eventually settle
to the following transform set:

$G=\left\\{\theta_{-1}=-12.0^{\circ},\quad\theta_{0}=0.0,\quad\theta_{1}=12.0^{\circ}\right\\}.$

The moderate degree of 12 may indicate the average angular variation among
handwritten digits.

The performance improvement is as follows:

  1. (i)

For training samples (of 60,000), the accuracy is 94.11%, which is a 61 basis
point (bps) improvement from 93.50% of the raw PCA-NN.

  2. (ii)

For test samples (of 10,000), the accuracy rate is 94.09%, which is an
improvement of 60 bps from 93.49% of the raw PCA-NN. It is consistent with the
in-sample training data.

Later on we shall see that combined with the last component of bucketing and
error correction, the transformation method will eventually contribute to an
ultimate accuracy rate above 96.0% - an improvement of totally 250 basis
points from the raw PCA-NN.

###  4.3 Bucketing and Error Correction

The third and also the last stage of the PCA-NN design involves bucketing and
error correction.

####  4.3.1 Systematic Factors behind Erroneous Buckets

Error correction is a key step in Kalman Filtering. The underlying philosophy
however could be universally applicable, i.e., embedded within prediction
errors is usually some useful information about a hidden target and hence such
errors can be used for prediction improvement a posteriori. The third and also
the last component of the PCA-NN follows exactly this spirit.

Let $\alpha,\beta\in{\mathcal{A}}$ denote two labels (or digits in the current
application). Given a learning network ${\mathcal{N}}$ which outputs
$\hat{\phi}({\boldsymbol{x}})\in{\mathcal{A}}$ for any input sample
${\boldsymbol{x}}\in\Omega$. Let $\phi({\boldsymbol{x}})$ denote the
supervised labeling or ground-truth classification. When
$\hat{\phi}({\boldsymbol{x}})\neq\phi({\boldsymbol{x}})$, the network
${\mathcal{N}}$ produces a classification error.

Let the bucket $\Omega_{\beta\mid\alpha}$ denote all the training samples that
are supervised as $\alpha$ but classified by ${\mathcal{N}}$ as $\beta$, i.e.,

$\begin{split}\Omega_{\beta\mid\alpha}&=\\{{\boldsymbol{x}}\in\Omega:\;\phi({\boldsymbol{x}})=\alpha,\;\hat{\phi}({\boldsymbol{x}})=\beta\\}\\\
&=\\{{\boldsymbol{x}}\in\Omega_{\alpha}:\;\hat{\phi}({\boldsymbol{x}})=\beta\\}.\end{split}$
(11)

Then for any $\alpha\in{\mathcal{A}}$, $\Omega_{\alpha\mid\alpha}$ is a
correct bucket while any $\Omega_{\beta\mid\alpha}$ is an error bucket when
$\beta\neq\alpha$. Clearly,

$\Omega_{\alpha}=\bigcup_{\beta\in{\mathcal{A}}}\Omega_{\beta\mid\alpha}.$

An effective network ${\mathcal{N}}$ should see high concentrations on the
correct buckets $\Omega_{\alpha\mid\alpha}$’s.

When the total training population $\Omega$ is substantive (e.g., 60,000 for
MNIST), some erroneous buckets $\Omega_{\beta\mid\alpha}$’s could still
contain statistically meaningful pools of samples. For digit recognition in
MNIST, for instance, $\Omega_{4\mid 7}$ \- all images of digit “7” that are
misclassified as “4” by the raw PCA-NN, could still contain a few hundreds of
samples out of the entire training set.

There often exists a systematic driver that leads to so many misclassified
samples. Take $\Omega_{4\mid 7}$ for example. Some people (esp. in Continental
Europe) habitually strike through “7” to differentiate it from the digit “1”
in handwriting. As demonstrated in Figure 6, in combination with another habit
of “hooking” the top horizontal line of “7,” this could create the perfect
recipe for confusion with the digit “4.”

Figure 6:  Some examples in the error bucket $\Omega_{4\mid 7}$, i.e., those
supervised as “7” but classified as “4” by the raw PCA-NN. It demonstrates a
systematic deviation due to the habit of striking through the digit “7,” a
perfect recipe for creating confusion with the digit “4” (in low-level
vision).

####  4.3.2 The Bucketed PCA-NN for Error Correction

A general classification problem usually does not attempt to unearth the exact
physical causes behind such sizable error buckets. Instead, one holds the
general assumption that any sizable error bucket may result from a yet-to-be
characterized systematic pattern (e.g., a “7” with a through-strike), or more
generally, a new statistical mode. It implies that the associated label
$\alpha\in{\mathcal{A}}$ may be multimodal and the raw PCA-NN only captures
the main mode.

This inspires the following bucketed PCA-NN, as improved from the raw PCA-NN.
First, the neuron sets for “sizable” or substantial buckets are constructed.
for each label $\alpha\in{\mathcal{A}}$ do for each predicted label
$\beta\in{\mathcal{A}}$ (by the raw PCA-NN) do if
$\\#\Omega_{\beta\mid\alpha}<M$ then skip  else construct PCA neuron set
${\mathcal{N}}_{\beta\mid\alpha}$ for $\Omega_{\beta\mid\alpha}$ end if end
for end for The cutoff $M$ is to ignore buckets that are insignificant in
terms of size. For the present work, we have chosen $M=20$ heuristically. In
general it can be chosen in proportion to the size of the overall training
pool. In addition, the bucketed PCA neuron set
${\mathcal{N}}_{\beta\mid\alpha}$ is constructed exactly as the raw PCA-NN,
with a spectral tail cutoff $r$ (e.g., $r=20\%$ as consistent with the raw).

For convenience, a generic PCA neuron from a PCA bucket
${\mathcal{N}}_{\beta\mid\alpha}$ is denoted by
${\boldsymbol{v}}_{k}^{(\beta\mid\alpha)}$.

Also notice that for each label $\alpha\in{\mathcal{A}}$, the “diagonal”
bucket $\Omega_{\alpha\mid\alpha}$ contains most of the samples from
$\Omega_{\alpha}$, and its neural set ${\mathcal{N}}_{\alpha\mid\alpha}$
captures the prevailing features of the main mode of $\alpha$.

Once the bucketed neuron sets are constructed, the bucketed PCA-NN classifier
is then defined exactly as the raw PCA-NN.

input a sample ${\boldsymbol{x}}$ to be classified  for each label
$\alpha\in{\mathcal{A}}$ do for each non-empty bucketed PCA neural set
${\mathcal{N}}_{\beta\mid\alpha}$ do calculate the neuron-aggregated response
$u_{\beta\mid\alpha}$ (similar to $u_{\alpha}$ in Eqn. (3))  end for calculate
the bucket-aggregated response via:
$u_{\alpha}=\max\\{u_{\beta\mid\alpha}:\;\beta\\}$ end for project the label
for ${\boldsymbol{x}}$:
$\hat{\alpha}({\boldsymbol{x}}):=\mathrm{argmax}\\{u_{\alpha}:\;\alpha\in{\mathcal{A}}\\}$

Notice that the bucketed PCA-NN maintains the authentic structure of a neural
network, and is structurally identical to the transformed PCA-NN as depicted
in Figure 5, as long as the neurons
${\boldsymbol{v}}_{k}^{({\mathsf{g}},\alpha)}$ ’s are replaced by the neurons
${\boldsymbol{v}}_{k}^{(\beta\mid\ \alpha)}$ ’s for each PCA neuron bucket
${\mathcal{N}}_{\beta\mid\alpha}$. Hence we have omitted the illustrative
figure.

####  4.3.3 Performance of the Bucketed PCA-NN, and the Transformed

The performance for the bucketed PCA-NN is as follows.

  1. (i)

For training images, compared with the accuracy of 93.50% of the raw PCA-NN,
the bucketed PCA-NN achieves 96.23%, with an improvement of 273 basis points.

  2. (ii)

For test images, compared with the accuracy of 93.49% of the raw PCA-NN, the
bucketed PCA-NN achieves 94.93%, with an improvement of 144 basis points.

The improvement is more salient for in-sample training images than for the
test. After all, the bucketed neurons have been derived from the training
images.

Finally, we apply the transform method of the preceding subsection to the
bucketed PCA-NN. Assuming that the transform set (or group) is ${\mathsf{G}}$,
the neural network can then be summarized easily using the same pseudo code
for the bucketed PCA-NN. As in the previous discussion, ${\mathsf{G}}$ is
assumed to contain the identity transform.

input a sample ${\boldsymbol{x}}$ to be classified  for each label
$\alpha\in{\mathcal{A}}$ do for each non-empty bucketed PCA neural set
${\mathcal{N}}_{\beta\mid\alpha}$ do for each transform
${\mathsf{g}}\in{\mathsf{G}}$ do calculate the neuron-aggregated response
$u^{({\mathsf{g}})}_{\beta\mid\alpha}$ for ${\mathsf{g}}$-transformed neuron
set ${\mathcal{N}}_{\beta\mid\alpha}^{({\mathsf{g}})}$ end for calculate the
transform-aggregated response via:
$u_{\beta\mid\alpha}=\max\\{u^{({\mathsf{g}})}_{\beta\mid\alpha}:\;{\mathsf{g}}\in{\mathsf{G}}\\}$
end for calculate the bucket-aggregated response via:
$u_{\alpha}=\max\\{u_{\beta\mid\alpha}:\;\beta\\}$ end for project the label
for ${\boldsymbol{x}}$:
$\hat{\alpha}({\boldsymbol{x}}):=\mathrm{argmax}\\{u_{\alpha}:\;\alpha\in{\mathcal{A}}\\}$

We call this final neural network

the Bucketed PCA-NN with Transforms.

Notice that it carries the genuine structure of a neural network, and that the
network flow is identical to what is depicted in Figure 5 \- by substituting
any raw neuron set ${\mathcal{N}}_{\alpha}$ with its (non-empty) refined
buckets ${\mathcal{N}}_{\beta\mid\alpha}$ ’s, and also electing
$u_{\alpha}=\max_{\beta}\;\\{u_{\beta\mid\alpha}\\}$ as the delegate of
label/class $\alpha$ for the final $\mathrm{argmax}$-layer.

For the bucketed PCA-NN with transforms, the ultimate performance is as
follows.  (i) For training images, compared with the accuracy of 93.50% of the
raw PCA-NN, the bucketed PCA-NN with transforms achieves 96.99%, with a total
improvement of 349 basis points. The transforms (via two rotations) have thus
contributed to an improvement of 76 basis points, as compared with 96.23% of
the bucketed PCA-NN (without transforms). (ii) For test images, compared with
the accuracy of 93.49% of the raw PCA-NN, the bucketed PCA-NN with transforms
achieves 96.12%, with a total improvement of 263 basis points. The transforms
(via two rotations) have thus contributed to an improvement of 119 basis
points, as compared with 94.93% of the bucketed PCA-NN (without transforms).

##  5 Conclusion

To conclude, the current work has demonstrated that a major chunk (i.e., about
96.00+%) of the impressive accuracy rates from various neural networks is
interpretable. The bucketed PCA-NN has relied on the three interpretable
components for network design - PCA, neuron transformation, and error
correction via bucketing.

It is worth emphasizing that our primary objective is to construct genuine
neural networks without using non-network frameworks such as high-order
polynomial classification (as in other classical PCA-related efforts). This
allows a genuine apple-to-apple comparison with neural-network classifiers,
including DNN’s.

## Acknowledgments

The project has been stretched on and off from 2019 to 2021 through a
prolonged and challenging period of pandemic and unemployment. The author is
very grateful to all the friends and former colleagues who have
unconditionally encouraged and supported him at the personal level, esp. Rafa
Santander (my former manager and friend at the Santander Bank, USA), Andreza
Barbosa (Goldman Sachs), Andrew Ang (BlackRock), Huazhang Luo, Xiangwei Liu,
Dan Pirjol, as well as my dear recruiter Lou Rodriguez. Years later this brief
note could become a fragrant rose petal reminding me about the beauty of the
seemingly unpredictable dynamics of life and our era.

## References

  * [1] T. F. Chan and J. Shen.  Image Processing and Analysis - Variational, PDE, Wavelet, and Stochastic Methods.  Soc. Applied Industrial Math. (SIAM), 2005.
  * [2] I. Goodfellow, Y. Bengio, and A. Courville.  Deep Learning.  The MIT Press, 2016.
  * [3] D. H. Hubel and T. N. Wiesel.  Receptive fields of single neurons in the cat’s striate cortex.  J. Physiology, 124(3):574–591, 1959.
  * [4] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  Proc. of The IEEE, 86(11):2278–2324, 1998.
  * [5] D. Mumford and A. Desolneux.  Pattern Theory.  A K Peters, Ltd, Natick, Massachusetts, USA, 2010.
  * [6] M. Nitzberg, D. Mumford, and T. Shiota.  Filtering, Segmentation, and Depth.  Lecture Notes in Comp. Sci., Vol. 662. Springer-Verlag, Berlin, 1993.
  * [7] R. Ott.  Construction of quadratic polynomial classifiers.  Proc. Int’l Conf. Pattern Recog., pages 161–165, 1976.
  * [8] J. Schürmann.  A multi-font word recognition system for postal address reading.  IEEE Trans. Computers, C-27(81):721–732, 1978.
  * [9] G. Strang.  Introduction to Linear Algebra.  Wellesley-Cambridge Press, 4th edition, 2009.
  * [10] G. Strang.  Linear Algebra and Learning from Data.  Wellesley-Cambridge Press, 2019.