# Entropy Regularization Series: A Simple Method for Differentiated
Unsupervised Learning

Sue Sin Chong
Tsinghua University
Institution1 address
<EMAIL_ADDRESS>
###### Abstract

Is it possible to train several classifiers to perform meaningful
crowdsourcing to produce a better prediction label set without any ground-
truth annotation? A recent work which has achieved state-of-the-art results on
unsupervised classification, by combining end-to-end and contrastive methods.
In this paper, we will attempt to modify the contrastive learning objectives
to train several neural network to coordinate to produce a state-of-the-art
prediction on the CIFAR10 and CIFAR100-20 task. This paper will present a
remarkably simple method to modify a single unsupervised classification
pipeline to generate a series of neural networks with to learn different
features of a same dataset. With a set of neural networks that converge on
different set of features on the same dataset, instead of painting a class as
a generic template, it is now possible to characterize a class with a set of
more specific and finer features. The Entropy Regularization Series (ERS),
when added upon the contrastive learning objective functions, gives users a
knob to tweak the entropy state of the output space of unsupervised learning,
thereby diversifying the types of neural networks that can be generated by the
same pipeline. Exploiting the multiple converging points, the ERS can serve as
a self-filter for noisy and uncertain samples, it is able to separate certain
samples from sample where there are confusion and separate certain classes
from classes where there are confusion. The ERS is a sensitive knob to tweak
decision boundaries, and has proven to be able to produce classifiers that
beat state-of-the-art at contrastive learning stage. Experiments show that ERS
can produce different neural networks, that each have accuracy that is
comparable to the state-of-the-art, yet have each have learnt a different set
of features from the original dataset. It allows users of unsupervised
classification to in essence perform search in the output space entropy of
contrastive learning.

Unsupervised Learning, Diversified Feature Set, Entropy Regularization,
Convergence

##  1 Introduction

How to make unsupervised algorithms to converge and learn different features
at will at the same time has always been difficult. Recently, a combination of
representation learning method and end-to-end learning has given promising
results on the coarse classification of reasonably large datasets such as
CIFAR100 [wvangansbeke2020scan]. [wvangansbeke2020scan]’s three-step process
involves 2 contrastive learning stages and a fine-tuning stage.

Recently, [JunGuo_AAAI_2020] works on feature selection in unsupervised
learning. [pretextInvariant2020], [pretrain2019Uncurated] works on making the
pretext training phase of unsupervised learning more robust. The properties of
similarity search vectors and application in unsupervised image classification
task, has also been much studied in [alex2018spreading], [spreadLocal2019] .
The integration of noise and feature into the unsupervised learning pipeline
has also proven to be useful [bojanowski2017unsupervised].
[jain2016approximate] [Zhang2015SparseCQ] has studied the possibility of
searching in a quantized sparse space representation. [he2019momentum] has
studied momentum of the unsupervised clustering process. Most recently,
[chen2020simple] introduced a simple contrastive learning framework. Whereas
[chuang2020debiased] makes the contrastive learning stage robust. And
[asano2020self] is a fine-tuning technique that has proven to be very
effective in the final stages of unsupervised learning. Whereas
[vo2019unsupervised] attempts to find new objects or features as an
optimization objective.

Entropy regularization is an important technique in machine learning, and has
many applications. [pereyra2017regularizing], [cuturi2013sinkhorn],
[Liong2015DeepHF] all utilized entropy regularization enhance unsupervised
learning, entropy regularization can either maximize marginal entropy of bits
or speed-up classification. In searching, [subic2017] also utilized entropy
regularization on one-hot codes. Whereas [entropy2015Spring] has studied the
mass-spring-damper system without a singular kernel, which might be able to
play the role of controller in unsupervised learning.

###  1.1 Related Work

Out of Distribution (OOD) Detection is an increasing prominent field in
machine learning. [pidhorskyi2018generative] is a method for generating state-
of-the-art OOD detector with adversarial method. [lee2018simple] used a method
based on Gaussian to extract features from a trained neural network to train
an OOD detector with a relatively simple tools. Via [lee2018simple], one is
able to link the certainty of features learnt to OOD classifier accuracy.

###  1.2 Motivation

The ability to check for feature that has not been detected by an unsupervised
learner but also an important feature of the dataset is important. It serves
as a fill-in-the-blank check for an unsupervised classifier. It allows us to
paint a class as a set of more specific and finer and at the same time correct
features, instead of having to accept a class as a very generic template of
features.

Secondly, the ability to separate weakly-classified samples from well-
classified samples only by using similarly unsupervised-trained neural network
or pipeline, allows the unsupervised learner to self-check, and improve. When
it is hard to differentiate, it will be very helpful to be able to have neural
networks trained with different feature emphasis to vote on a particular
sample. This motivates us to modify entropy of the output space to train
neural networks which has different correctness emphasis to help partition the
dataset into easy-to-classify and possibly-hard-to-classify samples.

##  2 Preliminaries

We attempt to introduce the possibility of searching in the entropy space in
the unsupervised image classification task. This work attempts to combine the
best of representation learning, end-to-end learning and the properties of
spreading vector and entropy control to give users a simple yet intuitive way
to manage the entropy in output space of unsupervised learning.

In this paper, we present a remarkably simple method for training reasonably
good unsupervised classifiers which automatically learns different features of
the dataset. We view the problem of improving unsupervised classification as
the unsupervised and automatic extraction of different features. We present a
framework where it is possible to generate series of neural networks which
have different and complementing feature emphasis simply by tweaking
constants. Our objective is to maximize the number of possibilities in output
space entropy distribution of converging neural networks, which allows for a
large variety of decision boundaries and therefore a highly diversified set of
learnt features.

To tackle the task of unsupervised entropy management, we developed an
approach of involving entropy and entropy regularization terms in objective
functions throughout the contrastive learning stage, called the entropy
regularization series (ERS). The ERS is a set of regularization terms to be
added on unsupervised learning objective functions . ERS offers a simple-to-
implement yet sensitive knob to unsupervised classification entropy
management, which allows for differentiated unsupervised learning. This work
presents the array of rich modification options presented by adding entropy
regularization term, and giving insight to the possibility of automatically
identify various features of different classes by training multiple neural
networks. We show that the probability of finding a neural network that has
learnt different feature set is high, and that generating a stable neural
network which has learnt a different feature set is very convenient.

The main contributions of our paper are as follows:

  * •

We identified a method to manage entropy in the unsupervised classification
problem, the ERS method.

  * •

ERS is simple to implement and use, and almost always guarantee to learn
different feature sets from the same data. Without ERS, unsupervised learning
always converges on to the same set of feature, reproducably.

  * •

ERS method can produce networks that have classification accuracy which
compares to state-of-the-art, but with different feature emphasis. It can also
serve as a uniqueness of convergence check on an unsupervised classification
pipeline.

  * •

Series of neural network trained with ERS can self-check to filter for samples
where there are second opinions.

  * •

We conduct experiments on CIFAR100-20 task. And shown that finding a neural
network which learnt the features of a variety of subclasses is easy.

  * •

ERS can mine for neural networks which can be better trained as out-of-
distribution detectors.

In the following section, we will discuss the definition and implications of
solving the problem of diversified unsupervised learning with ERS.

##  3 Problem Definition

Given a unsupervised classification pipeline $Process$, how to automatically
train series of neural networks which detect different features and are each
independently a reasonable classifier.

###  3.1 Applications

ERS is a natural filter for weakly-classified samples in unsupervised
learning. With a series of neural network which each converge to a different
set of correct features, it can immediately identify weakly-classified
samples. Secondly, ERS checks for convergence in features learnt in
unsupervised classification pipelines. When the dataset has multiple ways of
matching different features to the same set of classes, it is possible for ERS
to list the different mappings between features and classes. It serves as a
neighborhood exploration tool which allows for grid-searching with constants
on some arbitrary entropy structure in the output space.

###  3.2 Implications

When a $Process$ is sufficient to produce a reasonable classifier by
unsupervised learning on a dataset. By using ERS, we can exhaust the feature
discovery possibilities of the target $Process$. If there exist any subclass
within a super-class, the ERS will effectively mine for sub-class
representations in the original dataset as there will very likely exist a
neural network which converge to an alternative set of features, it serves as
a checker for the uniqueness of feature convergence in a unsupervised
classification process. This is particularly useful when the actual number of
classes is unknown.

The first section will discuss the ERS method on Unsupervised Semantic
Clustering pipeline. Whereas the second section will discuss reasoning about
entropy regularization, and the convergence of neural network trained with
ERS. The third section will discuss the implications of finding complementing
neural networks which has learnt different features of the same dataset. The
fourth section will discuss the application of combinations of neural network
with different feature emphasis. This paper will conclude with applications
and implications of using ERS as a output space entropy controlling tool.

##  4 Entropy Management via Entropy Regularization Series (ERS)

The Entropy Regularization Series (ERS) is a set of entropy regularization
terms to be added to contrastive learning optimization functions, as follows.
ERS is to be added on the contrastive learning stages of unsupervised
classification to encourage different decision boundary and entropy
distribution in the output space, thereby enforcing different feature
emphasis. Experiments show that training neural networks with different
confusion matrix is not possible without the ERS. The implementation of ERS
merely requires an additional tens of lines of code to the loss function.

$\displaystyle(-\lambda_{0},-\lambda_{1},+\lambda_{2},-\lambda_{3})$ (1)
$\displaystyle\bigdot($
$\displaystyle\sum_{c\in\mathcal{C}}\Phi_{\upeta}^{{}^{\prime}c}\log\Phi_{\upeta}^{{}^{\prime}c},\sum_{c\in\mathcal{C}}\Phi_{\upeta}^{{}^{\prime}c}\log\Phi_{\upeta}^{{}^{\prime}c},$
$\displaystyle\sum_{c\in\mathcal{C}}\sum_{k\in\mathcal{N}_{X}}\Phi_{\upeta}^{c}\Phi_{\upeta}^{k}\log\Phi_{\upeta}^{c}\Phi_{\upeta}^{k},$
$\displaystyle\sum_{c\in\mathcal{C}}\sum_{k\in\mathcal{N}_{X}}\Phi_{\upeta}^{{}^{\prime}c}\Phi_{\upeta}^{k}\log\Phi_{\upeta}^{{}^{\prime}c}\Phi_{\upeta}^{k})$
$\displaystyle,\Phi_{\upeta}^{{}^{\prime}c}=\frac{1}{|\mathcal{D}|}\sum_{X\in\mathcal{D}}\Phi_{\upeta}^{c}(X))$

Entropy terms were previously added in semantic clustering to encourage
uniform prediction amongst classes in the contrastive learning stage. Higher
order entropy terms in ERS plays the role of tweaking and modifying prediction
amongst classes. Our experiment shows that ERS in unsupervised classification
objective functions are indispensable for identification of different
features, ERS improve the feature diversity of the neural network set in
general.

Higher order entropy terms play the role of controller in controlling the
distances between clusters in the output space, hence it is possible to
produce neural networks with different confusion matrices. It is possible to
both maximize or minimize for between-cluster spaces and the smoothing of the
clustering process.

###  4.1 ERS Acting Upon Unsupervised Semantic Clustering

There are 3 portions to unsupervised classification in the
[wvangansbeke2020scan], SimCLR, Semantic Clustering(SCAN), and Selflabel(SLL).
SimCLR and SCAN being contrastive learning stages. Adding entropy terms in the
objective functions of contrastive learning stages can improve the diversity
of neural network trained. Regularization term in SimCLR is as follows.

$\displaystyle\min_{\theta}d(\Phi_{\theta}(X_{i}),\Phi_{\theta}(T[X_{i}]))$
(2)
$\displaystyle-\lambda_{0}\langle\Phi_{\theta}(X_{i}),\Phi_{\theta}(T[X_{i}])\rangle\log\langle\Phi_{\theta}(X_{i}),\Phi_{\theta}(T[X_{i}])\rangle$

Regularization term in SCAN portion of the pipeline is as follows.
$\lambda_{2}$ and $\lambda_{3}$ plays the role of a spring and damper for
between-clusters entropy.

$\displaystyle\Uplambda=$
$\displaystyle-\frac{1}{|\mathcal{D}|}\sum_{X\in\mathcal{D}}\sum_{k\in\mathcal{N}_{X}}\log\langle\Phi_{\upeta}(X),\Phi_{\upeta}(k)\rangle$
(3)
$\displaystyle+\lambda_{1}\sum_{c\in\mathcal{C}}\Phi_{\upeta}^{{}^{\prime}c}\log\Phi_{\upeta}^{{}^{\prime}c}$
$\displaystyle-\lambda_{2}\sum_{c\in\mathcal{C}}\sum_{k\in\mathcal{N}_{X}}\Phi_{\upeta}^{c}\Phi_{\upeta}^{k}\log\Phi_{\upeta}^{c}\Phi_{\upeta}^{k}$
$\displaystyle+\lambda_{3}\sum_{c\in\mathcal{C}}\sum_{k\in\mathcal{N}_{X}}\Phi_{\upeta}^{{}^{\prime}c}\Phi_{\upeta}^{k}\log\Phi_{\upeta}^{{}^{\prime}c}\Phi_{\upeta}^{k}$
$\displaystyle,\Phi_{\upeta}^{{}^{\prime}c}=\frac{1}{|\mathcal{D}|}\sum_{X\in\mathcal{D}}\Phi_{\upeta}^{c}(X)$

The scalar terms $(\lambda_{i})_{i\geq 0}=f(classes)$ , is a function of
number of classes. Below is a table of scalar terms and the classification
accuracy of different lambda combinations. Together, the new optimization
function presents a control function related to number of classes to modify
the entropy in the input space. It is hence possible to specify a specific
behavior desired of output neural network, and then simply by modifying
relative values of $(\lambda_{i})_{i\geq 1}$ train a converging network with
the desired property.

Table 1: Experiment Results Desc | $\lambda_{0}$ | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$ | Scan | SLL
---|---|---|---|---|---|---
SCAN | 0 | 5 | 0 | 0 | 0.44 | 0.507
1 | 0 | 5 | 4 | 0 | 0.439 | 0.503
2 | 2 | 5 | 4 | $8$ | 0.4335 | 0.5012
3 | 2 | 5 | 4 | $4$ | 0.4005 | 0.4408
4 | 2 | 5 | $4\sqrt{n}$ | $-8/n$ | 0.456 | 0.4767

Reflecting the sensitivity of spring constant and dampening ratio in a
physical spring system, neural networks trained with objective functions with
regularization terms that are scalar multiples of each other can also have
significantly different behaviors. It is also possible to improve the
performance of a single classifier in the contrastive learning stage.

####  4.1.1 Reasoning About Entropy State

By optimizing for entropy throughout, we are giving the constraint of number
of classes throughout the output space, forcing different values and
structures of classes in the trained neural network. The regularization of
entropy throughout the pipeline, encourages us to introduce an abridged
notation to reason about the state of entropy that the training has result in.
It will allow for the convenience of solving for deducing about the stability
of training process and the properties of the output neural network. The
3-step unsupervised classification process is essentially a cascading function
of the entropy in the input set.

Table 2: Entropy Regularization State Notation $(\lambda_{i})_{i\geq 1}$ | Value | Notation
---|---|---
$\lambda_{0}$ | 0 | $g(x)=x$
$\lambda_{0}$ | $>0$ | $g(x)=\hat{x}$

Let $g(x)$ be the pretext regularization function. Similarly, we define the
following for SCAN.

$\displaystyle\Uplambda=$
$\displaystyle-\frac{1}{|\mathcal{D}|}\sum_{X\in\mathcal{D}}\sum_{k\in\mathcal{N}_{X}}\log\langle\Phi_{\upeta}(X),\Phi_{\upeta}(k)\rangle$
(4)
$\displaystyle+\lambda_{2}x-\lambda_{3}x^{\prime}+\lambda_{4}x^{\prime\prime}$

Let $\lambda_{2}x-\lambda_{3}x^{\prime}+\lambda_{4}x^{\prime\prime}=h(x)$ be
the SCAN regularization function. Then compounding the two regularization
functions, the state of entropy as a result of the training process can be
expressed as $h(g(x))$. Hence, the contrastive training process is maximizing
similarity together with a second order differential equation of entropy,
offering fine-grained knobs to the desired output entropy state, and the
decision boundary formation within clusters. The ERS serves as a control
function for entropy in the contrastive learning stage.

###  4.2 Second Order Entropy Control Functions

$(\lambda_{i})_{i\geq 1}$ is a function of number of classes, aka
$(\lambda_{i})_{i\geq 1}=f_{i\geq 1}(n)$. Expressing the entropy control
function as an inner product between a series of functions of number of
classes, we have the following.

$\displaystyle h(g(x))$ $\displaystyle=h(n,g(x))$ (5)
$\displaystyle=\lambda_{1}g(x)-\lambda_{2}g(x)^{\prime}+\lambda_{3}g(x)^{\prime\prime}$
$\displaystyle=f_{1}(n)g(x)-f_{2}(n)g(x)^{\prime}+f_{3}(n)g(x)^{\prime\prime}$
$\displaystyle=\langle f_{i\geq 1}(n),g(x)\rangle$

Self-labeling which attempts to correctly classify noisy neighbors near
clusters by minimizing cross-entropy loss, is the only step in the pipeline
which only optimizes for cross-entropy. ERS makes it possible to
systematically produce converging neural networks with different decision
boundaries simply by modifying $(f_{i}(n))_{n\geq 1}$. Experiments show that
neural networks trained with different objective functions are most confident
about a wide variety of different prototype images for each superclass. This
offers a special edge to the task of CIFAR100-20, as it allows for the mining
of more subclass prototypes within a super-class, thereby improving overall
accuracy of classification.

###  4.3 Grid-Searching for Optimization Functions to Produce the Best
Performing Neural Network in Contrastive Learning Stage

On the contrastive learning stages, ERS has consistently been able to mine for
neural networks that beat state-of-the-art by $0.01-0.02$ at the end of
contrastive learning stage on the CIFAR10 and CIFAR100-20 classification task.
However, the slight edge gained doesn’t persist through the self-labelling
stage. ERS is also able to produce neural networks which in turn train better
out-of-distribution classifiers on the same dataset.

##  5 A Sensitive Knob

The ERS is necessary to train neural networks to learn different features.
Experiment shows that simply by tweaking the model without any regularization
term, the confusion matrix remains largely similar. The terms forces different
structure on the neural network, which results in differences in confusion
matrix.

ERS is a sensitive knob for tweaking the entropy in output space. In this
section, we will discuss an example neural network series, trained with
$\lambda_{3}=\\{4,8,16,32\\}$. To demonstrate the fact that ERS is able to
produce neural networks which learn different set of feature at ease, we
introduce the notion of n guess accuracy.

####  5.0.1 N Guesses Accuracy

Given n neural networks, the probability that at least one neural network has
learnt the features of a class and matched the feature correctly to a class
can be approximated by n guess accuracy.

The accuracy of N guesses is calculated as follows. Given n neural networks,
if any of the n neural network predicts the label correctly, then it is
calculated as a correct prediction. There is no hierarchy or preference to the
set of prediction label produced by the set of n neural networks.

###  5.1 Self-complementing Series

The $\lambda_{3}$ term is a sensitive knob for controlling the fine-grained
cluster formation process. It is easy to train a set of neural networks simply
by multiplying the $\lambda_{3}$ with a geometric series. By tweaking
$\lambda_{3}$， we are almost certain to find a neural network which has learnt
a different set of correct features.

Table 3: 2 Guess Of a $\lambda_{3}$ Series 2 guess ACC | Agreement | $\lambda_{0}$ | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$
---|---|---|---|---|---
63.79 | 44.90 | 2 | 5 | 4 | 4
|  | 2 | 5 | 4 | 8
61.82 | 55.81 | 2 | 5 | 4 | 8
|  | 2 | 5 | 4 | 16
61.8 | 50.59 | 2 | 5 | 4 | 16
|  | 2 | 5 | 4 | 32
61.68 | 42.64 | 2 | 5 | 4 | 4
|  | 2 | 5 | 4 | 32

The neural network in the series agrees samples in which they are both
certain, and only disagrees on samples where there are confusion. With the
ERS, we are able to identify classes and samples where there are second
opinions. The ERS can serve as a filtering method for noisy samples and noisy
classes.

Table 4: 3/4 Guess Of a $\lambda_{3}$ Series 3/4 guess ACC | 2-agreement | $\lambda_{0}$ | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$
---|---|---|---|---|---
69.35 | 75.86 | 2 | 5 | 4 | 4
|  | 2 | 5 | 4 | 8
|  | 2 | 5 | 4 | 32
67.80 | 83.97 | 2 | 5 | 4 | 4
|  | 2 | 5 | 4 | 8
|  | 2 | 5 | 4 | 16
67.49 | 82.50 | 2 | 5 | 4 | 4
|  | 2 | 5 | 4 | 16
|  | 2 | 5 | 4 | 32
71.92 | NA | 2 | 5 | 4 | 4
|  | 2 | 5 | 4 | 8
|  | 2 | 5 | 4 | 16
|  | 2 | 5 | 4 | 32

Within the same series, it is possible to sieve for confident samples with
majority votes for at least three quarters of the samples. We can further
clamp down on samples where all of the networks do not agree on, aka samples
with very high confusion. Below is an image of the most confident prototypes
of the series of neural network. This series of neural network has learnt the
many sub-classes within the 20 super-classes of the CIFAR100 dataset. For
instance, in the class of super-class of large carnivores, this series has
learnt bear, tiger, leopard and lion respectively as their most confident
prototype for the same superclass.

ERS can mine for a wider set of representation of a class, when there are many
sub-classes within a super-class, the advantage will be prominent.

##  6 Different Classes of ERS Functions

###  6.1 Extended Stability

Although it is possible to simply train for instance, 4 neural networks and
achieve the aim of filtering for weakly-classified samples. It is also
possible to explore the entropy space further with even larger classes of ERS
functions. It has been proven via experimentation that the following lambda
bounds will produce optimization functions that can produce a converging
neural network. The number of possible ERS functions which can train
converging neural networks in the contrastive learning stage is very large.

Table 5: Big O Bounds for $f(n)$ $(\lambda_{i})_{i\geq 1}$ | Big O Bound
---|---
$\lambda_{0}$ | $O(1)$
$\lambda_{1}$ | $O(1)$
$\lambda_{2}$ | $O(n)$
$\lambda_{3}$ | $O(n\sqrt{(}n))$
Table 6: Stable Templates $\lambda_{0}$ | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$
---|---|---|---
0 | O(1) | 0 | 0
0 | O(1) | O(1) | 0
O(1) | O(1) | O(1) | 0
O(1) | O(1) | O(1) | O(1)
O(1) | O(1) | $O(\sqrt{n})$ | 0
O(1) | O(1) | 0 | $O(n)$
O(1) | O(1) | $O(\sqrt{n})$ | $O(\sqrt{n})$
O(1) | O(1) | $O(1)$ | $O(\frac{1}{n})$
O(1) | O(n) | $O(\frac{1}{n})$ | $O(n\sqrt{n})$

The decision boundaries of neural network trained is very sensitive to small
changes in $(\lambda_{i})_{i\geq 1}$. It is reasonable to expect that one can
find neural networks that can identify all other subclass prototypes in the
super-class by grid-searching the set of $\lambda$s. With the ERS, it is
possible to have multiple neural networks which is confident about more sub-
classes within the super-class to learn the sub-classes within a super-class.

It is observed that the accuracy of a single classifier in cifar20 can hardly
surpass 0.5 due to both the limit of number of classes and lack of labels.
Having multiple networks learning the 20 classes in CIFAR100-20, makes it
possible to have different neural networks learn the features of smaller
classes, which result in a better superclass classification accuracy.

###  6.2 Compounding Effect of Cross-Type ERS

On the CIFAR100 dataset, we have trained 25 Res18Nets for classification into
20 coarse classes. Each of the 25 Res18Nets has been trained with a different
lambda series. Results in the following table is obtained by combining 2 to 4
of the 25 neural networks.

Table 7: N-guess Performances N Guess | Best ACC | Mean ACC | Median ACC
---|---|---|---
1 nn | 50.7 | NA | NA
2 guess | 64.50 | 58.20 | 59.24
3 guess | 70.57 | 63.70 | 64.70
4 guess | 73.99 | 68.90 | 69.10

Each of the classifier is independently an at least $>0.40$ accuracy
classifier. And $>0.85$ of the classifiers show at least $0.1$ improvement in
classification guess. When we compound ERS of different classes, it is much
more likely to make the n guess of the neural networks more robust. Below is a
listing of some of the best combinations of neural networks. Results show that
some of the best combinations often involve neural networks with non-zero
$\lambda_{0},\lambda_{2},\lambda_{3}$, demonstrating the indispensability of
the entropy regularization terms in training neural networks to learn
different features.

Table 8: Some 2 Guess Combinations 2 guess ACC | $\lambda_{0}$ | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$
---|---|---|---|---
64.50 | 0 | 5 | 0 | 0
| 2 | 5 | 4 | 0
64.12 | 0 | 5 | 0 | 0
| 2 | 5 | $1/4n^{2}$ | $1/2n$
63.79 | 2 | 5 | 4 | 8
| 2 | 5 | 4 | 4
Table 9: 3 Guess Combinations 3 guess ACC | $\lambda_{0}$ | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$
---|---|---|---|---
70.79 | 0 | 5 | 0 | 0
| 2 | 5 | 4 | 0
| 2 | 5 | $1/4n$ | $0.5/\sqrt{n}$
70.66 | 0 | 5 | 0 | 0
| 2 | 5 | 4 | 0
| 2 | 5 | 4 | 4
70.60 | 0 | 5 | 0 | 0
| 2 | 5 | $1/4n$ | $0.5/\sqrt{n}$
| 2 | 5 | $1/4n^{2}$ | $1/2n$

In the CIFAR100-20 task, there are 5 sub classes in each super-class, there
are multiple ways of mapping sub-classes to a super class. ERS mines the input
dataset for multiple sub-classes within the super-class. By compounding 2 to 3
neural networks whereby each network learns a subclass within a super-class,
the n guess predictions will in expectation make good guesses which will
include each subclass.

Best 3 combinations often have confident prototypes drawn from a larger
variety of sub-classes.  While there may be confusion across super-classes,
each subclass is only represented once in each classifier confident
prototypes. Combinations in aggregation learn to be confident about a large
set of sub-classes. This will allow us to make a move towards classifying the
dataset into more sub-classes, and eventually approach 100 classes in the
CIFAR100 task.

####  6.2.1 Learning Sub-classes Within Super-class

With the ERS, we are able to mine for different prototypical images within a
super-class in the CIFAR100-20 task. Instead of being forced to view a super-
class as a set non-descriptive generic feature, ERS is able to mine for
specific subclass features within a super-class. And together, paint a super-
class as a collection of specific subclass features.

##  7 Majority Vote

By training several neural network, and have them voting on a classes of a
sample can produce state-of-the-art prediction accuracy. N guess accuracy is
the upper bound of majority vote accuracy. We are able to improve upon the
accuracy of a single prediction set by compounding the accuracy of several
classifiers. Our best combinations which involves 27 different neural networks
voting for the label of every single sample, reaches a new state-of-the-art
accuracy rate of 0.58.

Table 10: State-of-the-Art Results on CIFAR100-20 by Majority Voting Number of Classifiers | Accuracy by Majority Vote
---|---
State-of-the-Art | 50.7
3 | 55.9
4 | 56.1

This again demonstrates that the fuzziness introduced by ERS allows for
meaningful verification check across neural network trained from the same
process. We also observed that by dividing the classifiers into three tiers
according to its accuracy rate, and organize a voting amongst neural networks
which are drawn from each of the tiers more easily produce an accuracy rate
which is higher than randomly selected neural networks. Although

##  8 Conclusion

Neural networks trained with ERS have different feature extraction emphasis.
The knobs presented by $(\lambda_{i})_{i\geq 1}$ is fine-grained decision
boundaries knob to neural networks trained is very sensitive to small changes
in the series. This presents the possibility of using simple techniques on
$\lambda$s to automatically mine-train neural networks which learn different
features by exploiting the differences in a wide array of of correct features
learnt about the original dataset. The ability to generate multiple converging
neural network at ease will allow us to characterize the feature set of input
data in greater detail in the unsupervised context.