# KCNet: An Insect-Inspired Single-Hidden-Layer Neural Network with Randomized
Binary Weights for Prediction and Classification Tasks

Jinyung Hong
School of Computing, Informatics, and Decision Systems Engineering
Arizona State University
Tempe, AZ 85281, USA
<EMAIL_ADDRESS>
&Theodore P. Pavlic
School of Computing, Informatics, and Decision Systems Engineering
School of Sustainability
School of Complex Adaptive Systems
School of Life Sciences
Tempe, AZ 85281, USA
<EMAIL_ADDRESS>

###### Abstract

Fruit flies are established model systems for studying olfactory learning as
they will readily learn to associate odors with both electric shock or sugar
rewards. The mechanisms of the insect brain apparently responsible for odor
learning form a relatively shallow neuronal architecture. Olfactory inputs are
received by the antennal lobe (AL) of the brain, which produces an encoding of
each odor mixture across $\sim$50 sub-units known as glomeruli. Each of these
glomeruli then project its component of this feature vector to several of
$\sim$2000 so-called Kenyon Cells (KCs) in a region of the brain known as the
mushroom body (MB). Fly responses to odors are generated by small downstream
neuropils that decode the higher-order representation from the MB. Research
has shown that there is no recognizable pattern in the glomeruli–KC
connections (and thus the particular higher-order representations); they are
akin to fingerprints – even isogenic flies have different projections.
Leveraging insights from this architecture, we propose _KCNet_ , a single-
hidden-layer neural network that contains sparse, randomized, binary weights
between the input layer and the hidden layer and analytically learned weights
between the hidden layer and the output layer. Furthermore, we also propose a
dynamic optimization algorithm that enables the KCNet to increase performance
beyond its structural limits by searching a more efficient set of inputs. For
odorant-perception tasks that predict perceptual properties of an odorant, we
show that KCNet outperforms existing data-driven approaches, such as XGBoost.
For image-classification tasks, KCNet achieves reasonable performance on
benchmark datasets (MNIST, Fashion-MNIST, and EMNIST) without any data-
augmentation methods or convolutional layers and shows particularly fast
running time. Thus, neural networks inspired by the insect brain can be both
economical and perform well.

##  1 Introduction

Connectomics [48] aims to produce a blueprint of the connections between all
neurons. Such maps promise to help illuminate how the brain learns and how
memories are formed. Active connectome research focuses not only on the human
brain [49] but also on the brains of insects [22, 50]. The fruit fly
_Drosophila melanogaster_ has shown excellent associative learning ability to
link certain odorant stimuli (i.e., complex mixtures of odor compounds) to
rewards or penalties [12]; the fly tends to recognize a mixture of stimuli as
if they were a unitary object [19, 20]. Studies of the fruit fly nervous
system (as well as related studies in insects in general) have revealed that
olfactory signals from the antenna terminate in a so-called antennal lobe (AL)
of the brain, which is a bundle of $\sim$50 discrete subunits known as
glomeruli. These glomeruli act as feature extractors, turning each complex
odor mixture a 50-digit code. Projection neurons (PNs) connect each of these
glomeruli code components to multiple of the $\sim$2000 so-called Kenyon Cells
(KC), which are located in a remote area of the insect brain known as the
mushroom bodies (MB’s). Thus, the projection from the AL to the MB represents
a massive representational expansion. These KCs in the MB are then decoded by
smaller neuropils that lead to actuation of the fly behavior. From a deep
neural network perspective, this architecture has the form of a single-hidden-
layer network with an input layer (the AL), a hidden layer (the MB), and
several parallel output layers (the downstream neuropils that control fly
motion). However, research into the topology of the AL–MB connections has
revealed that the inputs to each KC are randomized, where even isogenic flies
appear to express different connections [13]. Furthermore, feedback inhibition
within the MB regulates excitability of the KCs to small regions, thereby
enforcing a sparse representation of each odor across the MBs that, when
artificially disrupted, greatly reduces the ability of flies to discriminate
between odors [39]. Thus, flies achieve neuronal function through the
involvement of only a small fraction of their latent-variable space in the MBs
[29], where each of these latent variables are effectively a randomized
combination of a few components of each feature [12].

Meanwhile, Artificial Neural Networks (ANNs) and deep learning have shown
excellent performance as tools applied in various fields such as computer
vision, speech recognition, and natural language processing, and nonlinear
predictive classifiers that can extract useful features from input data [45,
11]. However, training ANNs typically requires optimizing a non-convex
objective function with multiple local optima and saddle points [25], which is
difficult and costly to optimize. For this reason, neural networks with
relatively simple structures (i.e., single hidden layer) have gained renewed
attention [11]. The architecture of these models is easy to implement and
allows reasonable performance through simple optimization and fast learning
processes. A promising technique for the development of such networks is
_Randomization_ – construction of a neural network with a subset of randomly
generated and fixed weights so that the remaining weights can be learned in a
straightforward linear optimization problem. By projecting inputs randomly
into a high-dimensional space and then focusing training on finding weights
that map this space to the output, learning can be solved as a highly
tractable linear least square problem [45]. Furthermore, not only is the
nature of the optimization problem simpler, but the number of weights that
need to be trained is only a fraction of the total weights in the network.
Such networks have universal approximation properties [33, 37, 31], which has
allows them to be broadly applied to classification and regression problems.

In this paper, we propose a novel approach for generating randomized, single-
hidden-layer networks based on insights from the fly brain that allow for
increasing the efficiency of these simple architectures. In particular, we
introduce _KCNet_ , a single-hidden-layer neural network with _sparse_ and
randomized _binary_ weights that mimics how odor stimuli are encoded in the
insect brain’s high-dimensional KC representation. In addition, we also define
a dynamic optimization method based on the straight-through gradient estimator
[8], which allows the model to dynamically explore a more efficient set of
inputs. We first apply our model to odorant-perception tasks, which determines
how the perceptual characteristics of odorants can relate to the structural
properties of the molecules. We demonstrate that our model achieves better
performance compared to the existing data-driven approaches, such as XGBoost.
Furthermore, to demonstrate the proposed model’s versatility, we also address
image-classification tasks on image benchmark datasets, including MNIST,
Fashion-MNIST, and EMNIST. We show that without the need for data augmentation
or convolutional layers, KCNet can achieve reasonable performance.

##  2 Related Work

Our work is strongly inspired by insect neuroscience, particularly the
olfactory architecture of the fruit-fly nervous system. Olfactory sensory
neurons located on the antennae and maxillary palps of fruit flies are
responsible for detecting odors. The majority of these neurons only express
one of the $\sim$50 odor-receptor proteins [15, 52, 17, 24]. The olfactory
sensory neurons are projected to the AL, where they are merged so that the
neurons representing the same receptor gather in one of the $\sim$50 glomeruli
[17, 24]. Thus, this wiring diagram in which each odorant receptor is mapped
to one glomerulus shows how odors are represented in the AL. When an odorant
stimulus or a molecule binds to its cognate receptors, the sensory neurons
expressing these receptors fire, and an odor-specific pattern of glomerular
activity emerges [53]. PNs involved with each glomerulus pass olfactory
information to two higher brain centers: the lateral horn and the MB.
Specifically, the MB is responsible for the fly’s ability to learn to approach
or avoid odors, and an input neuropil of the MB, called the _calyx_ , contains
the axon terminals of the $\sim$50 types of PNs that connect with $\sim$2000
KCs (i.e., a $40\times$ representational expansion). This paper mainly focuses
on the transformation of odorant representation in the AL to the higher-order
representation across the KCs of the MB and how to implement artificial KCs in
our ANN model.

Our work is also influenced by Neural Network with Random Weights (NNRW), a
single-hidden-layer neural network with random weights from the input layer to
the hidden layer. The weights between the input and the hidden layers are
randomly generated in a suitable range and kept fixed during the training
process. Compared with conventional learning methods with global tuning, such
as deep learning with the back propagation, NNRW mitigates problems related to
slow learning convergence and local optimality traps due to highly non-convex
optimization. NNRW can achieve a much faster training speed with acceptable
accuracy. Furthermore, NNRW is easy to implement, and is theoretically capable
of universal approximation [33, 37, 31]. Although many studies of NNRWs have
been conducted under several names, such as Extreme Learning Machine (ELM)
[32], Random Vector-Functional Link (RVFL) networks [43], and Schmidt’s method
[46], the underlying ideas and training mechanism of the three methods are
very similar, and so we briefly introduce the formulation of ELM model as
follows.

Given a generic input $\textbf{x}\in\mathbb{R}^{d}$, the basic ELM model can
be represented by linearly combining $\mathcal{B}$ random nonlinear
transformations of the original input, as in

$f_{\mathcal{B}}(\textbf{x})=\sum_{i=1}^{\mathcal{B}}\beta_{i}h_{i}(\textbf{x};\textbf{w}_{i})$
(1)

where $i$th transformation $h_{i}$ is parameterized by a vector
$\textbf{w}_{i}$ of randomly drawn weights from a specific probabilistic
distribution or a certain rule. In contrast, the linear coefficients
$\beta_{i}$ are trained using training data. Thus, inputs are randomly
transformed to hidden latent variables, which the output layer is trained to
decode. In the _addictive_ case, a hidden node $h_{i}(\cdot)$ of the model
applies a predefined nonlinearity on a random affine combination of input
vector, as in:

$h_{i}(\textbf{x})=g(\textbf{a}_{i}^{T}\textbf{x}+b_{i})$ (2)

where random vector
$\textbf{w}_{i}=[\textbf{a}_{i}^{T},b_{i}]^{T}\in\mathbb{R}^{d+1}$ and $g$ is
a nonlinear function, such as a sigmoid. Given a set of $N$ data samples,
where each sample $i\in\\{1,\dots,N\\}$ consists of an input vector
$\textbf{x}_{i}$ and a one-hot-encoded target vector
$\textbf{y}_{i}\in\mathbb{R}^{c}$, a properly trained model will best
approximate the target matrix Y with the linear system
$\textbf{H}\boldsymbol{\beta}$, where which can be written compactly as
$\textbf{H}\boldsymbol{\beta}=\textbf{Y}$ where

$\displaystyle\textbf{H}=\begin{pmatrix}h_{1}(\textbf{x}_{1};\textbf{w}_{1})&\dots&h_{\mathcal{B}}(\textbf{x}_{1};\textbf{w}_{\mathcal{B}})\\\
\vdots&\ddots&\vdots\\\
h_{1}(\textbf{x}_{N};\textbf{w}_{1})&\dots&h_{\mathcal{B}}(\textbf{x}_{N};\textbf{w}_{\mathcal{B}})\end{pmatrix}_{N\times\mathcal{B}}\;\text{and}\quad\boldsymbol{\beta}=\begin{pmatrix}\boldsymbol{\beta}_{1}^{T}\\\
\vdots\\\ \boldsymbol{\beta}_{\mathcal{B}}^{T}\\\
\end{pmatrix}_{\mathcal{B}\times
c}\;\text{and}\quad\textbf{Y}=\begin{pmatrix}\textbf{y}_{1}^{T}\\\ \vdots\\\
\textbf{y}_{\mathcal{B}}^{T}\\\ \end{pmatrix}_{N\times c}.$ (3)

That is, H is the hidden layer output matrix of the network; the $i$th column
of H indicates the $i$th hidden node’s output vector with respect to
$\textbf{x}_{1},\dots,\textbf{x}_{N}$, and the $j$th row of H represents the
output vector of the hidden layer with respect to input $\textbf{x}_{j}$.
Because the learning problem is equivalent to a linear regression of the
output $(\textbf{Y})$ on the transformed versions of the inputs (embedded in
H), the output weights $\boldsymbol{\beta}$ can be estimated as
$\boldsymbol{\beta}^{*}=\textbf{H}^{\dagger}\textbf{Y}$ where
$\textbf{H}^{\dagger}$ is the Moore–Penrose generalized inverse [6, 7] of the
hidden layer output matrix H. As we discuss herein, our model is loosely
connected to a specific configuration of NNRWs and exploits the learning
process of training the output weights $\boldsymbol{\beta}$.

##  3 KCNet: An Insect-Inspired Single Hidden Layer Neural Network

###  3.1 Intuition

For bio-inspiration of our model, we focus our attention on: (i) how to form
random connections between the AL glomeruli (input layer) and the KCs (hidden
layer), and (ii) how to ensure hidden-layer sparsity to best differentiate one
odor stimulus from another. To define an arbitrary set of inputs for each KC
in the model, the connections between glomeruli and KCs can be represented by
sparse, binary, and random matrices [13]. The firing rates from $\sim$7
randomly selected glomeruli are passed and summed to each KC [13, 34]. For
example, Dasgupta et al. [18] proposed a computational model of the fly
olfactory system, including a sparse binary random projection matrix, to
generate a _tag_ for each odor composed of a set of neurons that are excited
when that odor is presented. We also refer to that definition and take 7
random indices of the input vector to generate the input for each KC.

The adequate method of representing the KCs’ output activity is critical
because the output of KCs is responsible for learning behavioral responses to
different odors [42]. In their fly-inspired algorithm, Dasgupta et al. [18]
use a Winner-Take-All (WTA) approach that models inhibitory feedback ensuring
that only the highest-firing 5% of KCs are used on the output. Their WTA
circuit selects the indices of the top $k$ values of KC activities; however,
this method is not ideal for the optimization approach that we introduce
later. Therefore, we base our algorithm on a computational model of the fly
brain from Endo et al. [23] that generates sparsity from global inhibitory
feedback between KCs [27, 39, 34]. In their model, KCs produce an intermediate
output that is then subject to global inhibition from the average glomerular
input to all KCs. The final KC is then generated by thresholding the inhibited
output through a ramp function, which is functionally equivalent the Rectified
Linear Unit (ReLU) activation function in artificial neural networks [26].
Thus, the output of the $j$th KC, $\mathcal{KC}_{out_{j}}$, was described as:

$\mathcal{KC}_{out_{j}}=\phi\left(\mathcal{KC}_{in_{j}}-C\frac{1}{N_{\mathcal{KC}}}\sum_{j}^{N_{\mathcal{KC}}}\mathcal{KC}_{in_{j}}\right)$
(4)

where $\mathcal{KC}_{in_{j}}$ indicates the weighted sum of input from 7
random indices of the input vector111See precise weighting details from Endo
et al. [23]., $\phi$ is the ReLU, $C$ is the strength of global inhibition,
and $N_{\mathcal{KC}}$ is the total number of KCs. Herein, we use $C=1.0$ and
$N_{\mathcal{KC}}=2000$ matching the values best calibrated to real KC
responses [23, 4].

###  3.2 Model Architecture

We take input from the feature space $\mathcal{X}\subseteq\mathbb{R}^{d}$ and
derive our model for a single output for simplicity. Our proposed model with
$\mathcal{B}$ ($\sim N_{\mathcal{KC}}$) hidden nodes is a parametric function
$f_{\mathcal{B}}:\mathbb{R}^{d}\to\mathbb{R}$ taking an input
$\textbf{x}=[x_{1},x_{2},\ldots,x_{d}]^{T}$. To define our randomized encoder
in the hidden nodes, we select $N_{in}\in[1,d)$ indices of the input at random
without replacement for each hidden node222$N_{in}<d$ to prevent hidden-unit
outputs becoming trivially 0 by Eq. (5). In our setting, $N_{in}=7$.. In other
words, the output $\bar{x}_{j}$ of each hidden node
$j\in\\{1,...,\mathcal{B}\\}$ is a weighted combination of all $d$ inputs
where only $N_{in}<d$ inputs have $w_{j,i}=1$ and all other inputs have
$w_{j,i}=0$. So the inputs are effectively gated by the weights on each hidden
node. Once the $j$th hidden node has produced weighted sum $\bar{x}_{j}$, the
post-processing step to produce intermediate output is performed in the hidden
layer. Mimicking Eq. (4) with $C=1$, the $j$th intermediate output
$\hat{x}_{j}$ is:

$\hat{x}_{j}=\bar{x}_{j}-\mu,\;\;\mu=\frac{1}{\mathcal{B}}\sum_{i=1}^{\mathcal{B}}\bar{x}_{i}$
(5)

where $\mathcal{B}$ is the number of hidden units. In the model’s decoder
definition, the proposed network is defined as follows in the same way that a
linear model is applied as a decoder in NNRW:

$f_{\mathcal{B}}(\textbf{x)}=\sum_{i=1}^{\mathcal{B}}\beta_{i}h_{i}=\sum_{i=1}^{\mathcal{B}}\beta_{i}g(\hat{x}_{i})$
(6)

where $g$ is the ReLU function [26] that allows the model to produce sparse
hidden output, which is more biologically plausible. Therefore, our model can
be formulated in the same form as in NNRW with the hidden layer output matrix
H and target matrix Y from Eq. (3). However, in contrast with NNRW, we
estimate $\boldsymbol{\beta}$ with a regularized Moore–Penrose inverse:

$\boldsymbol{\beta}^{*}=(\textbf{H}^{T}\textbf{H}+\lambda\textbf{I})^{-1}\textbf{H}^{T}\textbf{Y}$
(7)

where $\lambda$ is a specific regularization term; using a ridge
regularization helps to improve generalizability and performance of the model.
The overall architecture of the KCNet is depicted in Fig. 1.

(a) Overall architecture of the KCNet. Between the input layer and the hidden
layer, solid (dashed) lines indicate weight of 1 (0).

(b) Process of producing hidden layer output. Step 1 is random projection,
Step 2 in Eq. (5), and Step 3 in Eq. (6).

Figure 1: Visualization of model architecture of the KCNet

Our model has several noticeable characteristics that can be translated in
terms of computer science and machine learning, and the details can be seen in
Appendix A.2.

###  3.3 Dynamic Optimization Algorithm (DOA) For Feature Selection

Although KCNet can perform well by increasing the number of the nodes in the
hidden layer (i.e., the hidden layer’s width), we define the Dynamic
Optimization Algorithm (DOA), an iterative method that allows the KCNet to
explore a more efficient set of random inputs to obtain better performance
than conventional ones. For a fully trained single-hidden-layer neural
network, all the hidden weights and the output weights are optimized with
respect to a dataset and a loss function using gradient descent, whereas the
hidden weights of the KCNet are initially fixed to either 0 or 1, and training
adapts only the output weights during its learning process. What if we can
find efficient combinations of inputs that allow the model to achieve better
performance? How can we find which connection is effective while turning on or
off the connection between the input and hidden layers? Our algorithm never
updates the weights directly to address the issue. Since it associates two
options, 0 or 1, for each connection, we instead optimize the _preference_ to
select the weight between 0 and 1 to use.

##### Initialization

Let $w_{ji}$ be the connection between the $i$th input node and the $j$th
hidden node, and let $s_{ji}\in[-1,1]$ be the _preference score_ for $w_{ji}$,
indicating the degree to which it prefers to choose between 0 and 1. We
initially draw $s_{ji}\sim\mathbb{U}(lb,ub)$ randomly from a uniform
distribution between $lb=-1$ and $ub=1$ after generating the randomized binary
weight $w_{ji}\in\\{0,1\\}$ on each input. Then, we can readily convert the
weight $w_{ji}$ into the preference score $s_{ji}$, and vice versa, by
defining selection functions $\rho_{w\rightarrow s}$ and $\rho_{s\rightarrow
w}$ as follows:

$\displaystyle s_{ji}=\rho_{w\rightarrow
s}(w_{ji})\sim\begin{cases}\mathbb{U}(0,ub)&w_{ji}=1,\\\
\mathbb{U}(lb,0)&\text{otherwise}\end{cases}\quad\;\text{and}\;\quad
w_{ji}=\rho_{s\rightarrow w}(s_{ji})=\begin{cases}1&s_{ji}>0,\\\
0&\text{otherwise}\end{cases}$ (8)

With the above setting, we can start by iterating the following forward pass
and the backward pass to update the preference score $s_{ij}$ instead of
$w_{ij}$ using the gradient descent method. However, because neither selection
functions $\rho$ are differentiable, we cannot directly compute the gradients
for $s_{ji}$. We explain how to solve this in the part of the backward pass.

##### Forward pass: training output weight

We split the dataset into two sets randomly; the training set for learning the
output weight $\beta$ and the validation set for searching a random collection
of input. Then, we build a model using the weight $w_{ji}$ and train $\beta$
using a least-squares method.

##### Backward pass: updating preference score

In the backward pass, all the preference scores are updated with gradient
descent method based on the straight-through gradient estimation [8] using the
validation set by propagating to $s_{ji}$ the gradients that $w_{ji}$
receives. Specifically, the selection function $\rho_{s\rightarrow w}$ is a
kind of threshold function in Eq. (8), and we ignore the derivative of the
function and pass on the gradients that $w_{ji}$ receives as if the function
was an identity function. Therefore, we can bypass the selection function in
the backward pass and compute the loss gradient with respect to $s_{ji}$:

$\nabla_{s_{ji}}=\partial\mathcal{L}/\partial
s_{ji}\approx\partial\mathcal{L}/\partial w_{ji}$ (9)

where $\mathcal{L}$ is the objective function. Therefore, it is then back-
propagated to obtain gradients on the parameters that influence $s_{ji}$ (see
the detail of the full derivative in Appendix A.1). Similar to usual back-
propagation method, the preference score $s_{ji}$ is updated by gradient
descent with learning rate $\alpha>0$:

$s_{ji}\leftarrow s_{ji}-\alpha\nabla_{s_{ji}}$ (10)

and then the updated preference score is saturated to be between $-1$ and $1$.
Based on the new score, the weight $w_{ji}$ is also updated by applying the
selection function $\rho_{s\rightarrow w}$ to it, and the forward pass to
train $\beta$ of the model performs again. We call the pair of forward and
backward passes an _epoch_ , and by continually iterating epochs, the KCNet
can automatically find a more efficient combination of inputs that distinguish
the representation belonging to one category from others. To terminate the
iterative process, we established two stopping criteria: i) if a process has
run a certain number of times, and ii) if the performance of the current model
exceeds the standard we set. Our experiments demonstrate that this simple
algorithm allows the model to find convincing configurations of random inputs
resulting in good results across different datasets (see details in Algorithm
1 in Appendix).

##### Ensemble DOA to assemble a weight matrix

Although the above DOA can enhance the conventional KCNet, the computational
burden drastically increases if the size of the dataset or the number of
hidden nodes in the model is enormous due to, for example, the computation of
an inverse matrix or an inner product between input matrix and the derivative
of a loss matrix with respect to the input in the backward pass. Furthermore,
we noticed that the larger the size of the weight matrix, the more perturbed
the convergence of the performance of the DOA-applied KCNet and the more
sensitive the learning rate (see Fig. 7 in Appendix). Thus, instead of
directly applying the DOA to the KCNet with a large weight matrix, we
introduce another use case of DOA, _Ensemble DOA_ , to assemble a large weight
matrix by concatenating several small weight matrices after each small weight
matrix was adapted individually by the DOA. In our experiments, we empirically
determined that it was most effective to divide a large weight matrix into
more than 10 small matrices. In this condition, KCNet with the Ensemble DOA
shows a much faster training speed than the KCNet with the DOA and outperforms
the original KCNet and the KCNet directly applied by the DOA.

##  4 Experiments

###  4.1 Odor Perception

The odorant perception task is the prediction of an identity of an odor from
the physical, chemical, and structural properties of its odor molecules.
Understanding odorant perception continues to be an active area of research
due to the complexity of the task. First of all, the dimensionality of the
feature space for odorant perception and the relationship between the
structure and odor are still unknown [10, 40, 47]. Furthermore, odor molecules
cannot be predicted to have the same odor properties by looking at
similarities in structural properties [9, 21]. Rather, odor molecules with
completely different structural features may have similar scents [2]. And
depending on how concentrated the odor stimulus is in the environment, it can
be perceived in many different smells [2].

Chacko et al. [14] focused on a specialized form of the odorant perception
task where odors could be classified by the presence or absence of two
particular subjective properties referred to as _Sweet_ and _Musky_. They used
a psychophysical dataset including odor perception ratings of 55 human
subjects. Based on this dataset, Chacko et al. used machine learning to
predict these perceptual characteristics from an odor molecule based on its
structural properties. We applied our models to the same dataset, and we found
that our KCNet models outperform the other data-driven approaches, such as
XGBoost, that were used by Chacko et al. [14].

##### Data pre-processing

Here, we briefly summarize the data pre-processing step used by Chacko et al.
[14] to define the odor-perception classification problem. For input labels,
Keller and Vosshall [35] proposed a dataset that consists of odorant attribute
data for 480 structurally distinct compounds at two concentrations. The odor
properties included were 20 semantic descriptors, such as _Sweet, Musky,
Flower_ , and so on, associating with odor quality. 55 subjects evaluated each
of these perceptual attributes in the range of 0 to 100. The degree of the
evaluation indicates how well each semantic descriptor describes the
perceptual characteristics of the odor. We assigned ground-truth labels for
their odor quality to each odorant compound based on the frequency of using
semantic descriptors assessed by the 55 subjects. To consider the complexity
of odor perception and the possibility that an odor stimulus can be recognized
to have multiple odor characteristics, we selected the top three semantic
descriptors for each odorant compound. Thus, we defined each odor stimulus to
be represented by three Odor Characters (OCs).

For input features, we utilized 196 two-dimensional RDKit molecular data as
input features [28]. It consists of topological descriptors, connectivity
descriptors, constitutional descriptors, molecular property, and MOE-type
descriptors and provides quantitative information about the physical,
chemical, and topological features measured in the 2D graph representations of
the structures. After some zero values of the descriptors were removed and the
dilution was converted into the one-hot encoding as a categorical feature, we
have 960 data samples (480 compounds $\times$ 2 dilutions) with 159 input
features. After data pre-processing, we noticed that other OCs showed a high
imbalance in their distribution, whereas _Sweet_ and _Musky_ OCs had enough
samples for the positive class (58.13%, 41.9%, respectively). Therefore, we
formulated it as a binary classification task by creating two datasets: one
for the _Sweet_ OC and another for the _Musky_ OC.

##### Methods

We randomly split the processed dataset into a training and testing set with a
9:1 ratio after applying a standard normalization. Only the training set was
utilized for model parameter tuning using a 5-fold cross-validation.

Figure 2: Test F1 score with 95% confidence interval of the KCNet for odor
perception depending on its hidden size.

During cross-validation, the training set was split into five folds at every
iteration, where one fold was kept as a validation set to test the performance
with the trained parameters. For the classification of _Musky_ OC, we applied
an oversampling method to the training data, excluding the validation set, to
reduce the imbalance in the data distribution. The weighted F1 score was used
as an evaluation metric because of the imbalanced characteristic of the
datasets. We configured the hyperparameter search space consistent with the
models of Chacko et al. [14] and a specific range for the regularization term
$\lambda$ for our model. We utilized the Optuna framework [1] by repeating the
cross-validation 100 times. After finding the best-performing number of hidden
nodes of the KCNet, we set it as the number of hidden nodes for both the DOA-
applied model and the Ensemble-DOA-applied model, and we used the same value
of the regularization term $\lambda$ as was used for KCNet. Details of the
methods, hyperparameter search spaces, and implementations are given in
Appendix A.4.

##### Results

Table 1 shows the performance comparison of test-weighted F1 score with 95%
confidence interval averaged by five experiments among models. The performance
variances of the KCNet and its variants are natural because their weight
matrices are stochastically determined, while the performances of other models
are stable after their hyperparameters are fixed. Our model and its variants
usually outperforms other models, and the best performing algorithm was the
KCNet with Ensemble DOA for _Sweet_ OC and the original KCNet for _Musky_ OC,
with test-weighted F1-scores of 0.8193 and 0.7434, respectively. Applying the
DOA to the KCNet deteriorated the performance of the KCNet for _Musky_ OC,
which may indicate that the sample size for this class is not large enough,
and the oversampling techniques might disturb the performance of the DOA. As
shown in Fig. 2, after sufficiently stretching the hidden nodes, the
performance of KCNet for _Sweet_ OC seems stable, but the performance of KCNet
for _Musky_ OC is more variable. However, the KCNet with the Ensemble DOA
alleviated the situation and achieved better performance than the KCNet with
the DOA. See the details of other plots and results in Appendix A.4.

Table 1: _Test-weighted F1 score of all models for odor perception._ “w/ DOA” is the DOA-applied KCNet and “w/ Ensemble DOA” is the Ensemble-DOA-applied KCNet. $\text{Mean}_{\pm\text{SD}}(\text{\\# of hidden units})$ for “KCNet” and “w/ DOA” and $\text{Mean}_{\pm\text{SD}}(\text{\\# of submodels}\times\text{\\# of hidden units for each submodel})$ for “w/ Ensemble DOA”. The best performance is displayed in bold. | Classification Tasks
---|---
Model | Sweet/Non-sweet | Musky/Non-musky
Adaboost | 0.7444 | 0.6436
Gradient Boosting Machine | 0.7538 | 0.7186
XGBoost | 0.7316 | 0.6627
Random Forest | 0.7427 | 0.6826
K-Nearest Neighbors | 0.7839 | 0.6538
Support Vector Machine | 0.7980 | 0.6629
KCNet | $0.8134_{\pm 0.017}\;(\text{1,300})$ | $\textbf{0.7434}_{\pm\textbf{0.052}}\;(700)$
w/ DOA | $0.8156_{\pm 0.067}\;(\text{1,300})$ | $0.7004_{\pm 0.027}\;(700)$
w/ Ensemble DOA | $\textbf{0.8193}_{\pm\textbf{0.022}}\;(13\times 100)$ | $0.7247_{\pm 0.055}\;(10\times 70)$

###  4.2 Image Classification

To show the general applicability of our model, we applied it to the image-
classification task using well-known image benchmark datasets, MNIST [36],
Fashion-MNIST [54], and EMNIST [16].

##### Datasets

MNIST is comprised of 10-class $28\times 28$ pixel grayscale images of
handwritten digits. It contains 60,000 images for training and 10,000 images
for testing. Fashion-MNIST is a dataset including $28\times 28$ grayscale
images of 70,000 fashion products from 10 categories, with 7,000 images per
category. The training (test) set has 60,000 (10,000) images. The EMNIST
dataset is a set of handwritten characters and digits derived from the NIST
Special Database 19 and converted to a $28\times 28$ image format. Six
datasets that comprise the EMNIST dataset but we focus one subset, EMNIST-
Balanced, which has 47 classes and 131,600 images so that 112,800 images for
training and 18,800 images for testing can be used.

##### Methods

We implemented the KCNet with PyTorch [44] to boost the training speed
supported by GPUs; PyTorch supports the above three datasets so that they can
be easily used. Therefore, we conducted our experiments after only applying a
standard normalization to them without the use of any data-augmentation
method. We also implement the ELM model and a Fully-trained Single-Hidden-
layer Neural network (FSHN) and test them to compare the performance with our
model. The ELM was introduced as a baseline classifier by Cohen et al. [16],
and its characteristics are very similar to our model. We formulated the ELM
model in a method that is often used in practice [45, 32, 11] as follows: i)
its hidden weights $\textbf{w}_{i}=[\textbf{a}_{i}^{T},b_{i}]^{T}$ were
defined by drawing uniform random variables
$\textbf{a}_{i}\sim\mathbb{U}(-1,1)$ and $b_{i}\sim\mathbb{U}(0,1)$; ii) the
sigmoid function is used as a nonlinearity function $g$; and iii) the output
weight $\beta$ was estimated using the Moore–Penrose inverse. FSHN was
implemented using the linear models followed by the ReLU function and
optimized using the stochastic gradient descent, supported by PyTorch. The
number of hidden nodes in the FSHN was set to have the number of adaptable
parameters similar to KCNet. For hyperparameter setting in the DOA-applied
KCNet and the Ensemble-DOA-applied KCNet, the original training set was
randomly split into a training set and a validation set with a 5:1 ratio every
epoch. See Appendix A.4 for details of methods, hyperparameter search spaces,
and implementations.

##### Results

Table 2 shows the results of test accuracy with 95% confidence interval on the
image benchmark datasets over five experiments.

Table 2: _Test accuracy of all models for image classification._
$\text{Mean}_{\pm\text{SD}}(\text{\\# of hidden units})$ for “ELM”, “KCNet”
and “w/ DOA” and $\text{Mean}_{\pm\text{SD}}(\text{\\# of learnable
parameters})$ for “FSHN”. $\text{Mean}_{\pm\text{SD}}(\text{\\# of
submodels}\times\text{\\# of hidden units for each submodel})$ for “w/
Ensemble DOA”. The best performance is displayed in bold.

| Image Dataset
---|---
Model | MNIST | Fashion-MNIST | EMNIST-Balanced
ELM [32] | $0.9658_{\pm 0.002}\;(\text{6,500})$ | $0.8813_{\pm 0.006}\;(\text{7,000})$ | $0.7747_{\pm 0.002}\;(\text{4,500})$
FSHN | $0.975_{\pm 0.002}\;(\text{65,200})$ | $0.8671_{\pm 0.002}\;(\text{70,765})$ | $\textbf{0.8179}_{\pm\textbf{0.002}}\;(\text{211,375})$
KCNet | $0.9735_{\pm 0.003}\;(\text{6,500})$ | $0.8849_{\pm 0.002}\;(\text{7,000})$ | $0.7762_{\pm 0.002}\;(\text{4,500})$
w/ DOA | $0.9731_{\pm 0.002}\;(\text{6,500})$ | $0.8845_{\pm 0.002}\;(\text{7,000})$ | $0.7761_{\pm 0.002}\;(\text{4,500})$
w/ Ensemble DOA | $\textbf{0.9776}_{\pm\textbf{0.001}}\;(10\times 650)$ | $\textbf{0.886}_{\pm\textbf{0.001}}\;(10\times 700)$ | $0.8058_{\pm 0.002}\;(10\times 450)$

Our model outperforms ELM, and Fig. 3(a) (and Fig. 8 in Appendix) confirms
that the KCNet performs better than the ELM model across all the image
benchmark datasets.

(a) Test accuracy of KCNet and ELM for MNIST depending on their hidden sizes

(b) The comparison of running time, including model configuration time,
training time, and evaluation time (sec) for KCNet, ELM, and FSHN

Figure 3: For image classification task using MNIST, the performance
comparison between KCNet and ELM and the running time comparison between KCNet
and FSHN

The best-performing model is the FSHN for EMNIST-Balanced, and the Ensemble-
DOA-applied KCNet for MNIST and Fashion-MNIST. The direct application of DOA
to KCNet seems rather performance-deprecating, which shows that the larger the
weight matrix, the less fully exploitable the DOA. However, the Ensemble DOA
is very effective at enhancing the KCNet because the DOA can be fully utilized
to explore an effective set of inputs for relatively small weight matrices.
Even though the performance of KCNet was worse than the FSHN, the running time
of our model was much faster than FSHN (and ELM) across all the benchmark
datasets as shown in Fig. 3(b). Therefore, the KCNet allows for compromising
performance slightly in order to increase speed. For more details, see
Appendix A.4.

##  5 Conclusions and Broader Impact

We proposed a fast, insect-brain-inspired single-hidden-layer neural network
and a dynamic optimization algorithm that allows the model to explore
meaningful connections in hidden weights and to utilize an ensemble method of
assembling weight matrices. Although we showed potential of our model through
the above experimental results, there are several limitations yet to be
addressed, as detailed in Appendix A.5. For future work, by using the
discriminating power of our model, we will extend application to other tasks,
such as one-shot learning or contrastive learning. Furthermore, we will
construct a more brain-like model by combining our model with biologically
plausible learning algorithms, such as target propagation [41].

##### Broader Impact

This study integrates ideas from computational insect neuroscience with
machine learning to show how neural networks can be made smaller and simpler
to train with little compromise in performance. The net societal impacts of
this research are primarily positive. By helping to reduce the resource
footprint and increase the speed of neural networks used for common tasks,
this research increases the sustainability of artificial neural networks. That
said, reducing the barriers to the application of AI can lead to faster growth
in the use of AI, which can be positive or negative. Beyond the technological
impacts, this research has positive scientific value as it further illuminates
the adaptive function underlying architectures in the insect brain,
potentially aiding in understanding the evolution of these architectures.
Despite the many qualitative large-scale differences between arthropod and
vertebrate brains, a number of similar circuit motifs and smaller-scale brain
structures have been found. Thus, better understanding how insects learn to
discriminate among complex signals can eventually also aid in understanding
the mechanisms and evolution of similar functions in the human brain.

## Acknowledgments and Disclosure of Funding

This work was supported in part by NSF grant SES-1735579.

## References

  * Akiba et al. [2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 2623–2631, 2019.
  * Amoore [1970] John E. Amoore.  _Molecular Basis of Odor_.  Charles C. Thomas, Springfield, Illinois, 1970.
  * Andoni and Indyk [2006] Alexandr Andoni and Piotr Indyk.  Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions.  In _2006 47th annual IEEE symposium on foundations of computer science (FOCS’06)_ , pages 459–468. IEEE, 2006.
  * Aso et al. [2009] Yoshinori Aso, Kornelia Grübel, Sebastian Busch, Anja B Friedrich, Igor Siwanowicz, and Hiromu Tanimoto.  The mushroom body of adult drosophila characterized by gal4 drivers.  _Journal of neurogenetics_ , 23(1-2):156–172, 2009.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Banerjee [1973] KS Banerjee.  Generalized inverse of matrices and its applications, 1973.
  * Ben-Israel and Greville [2003] Adi Ben-Israel and Thomas NE Greville.  _Generalized inverses: theory and applications_ , volume 15.  Springer Science & Business Media, 2003.
  * Bengio et al. [2013] Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Bentley [2006] Ronald Bentley.  The nose as a stereochemist. enantiomers and odor.  _Chemical reviews_ , 106(9):4099–4112, 2006.
  * Bushdid et al. [2014] Caroline Bushdid, Marcelo O Magnasco, Leslie B Vosshall, and Andreas Keller.  Humans can discriminate more than 1 trillion olfactory stimuli.  _Science_ , 343(6177):1370–1372, 2014.
  * Cao et al. [2018] Weipeng Cao, Xizhao Wang, Zhong Ming, and Jinzhu Gao.  A review on neural networks with random weights.  _Neurocomputing_ , 275:278–287, 2018.
  * Caron [2013] Sophie JC Caron.  Brains don’t play dice—or do they?  _Science_ , 342(6158):574–574, 2013.
  * Caron et al. [2013] Sophie JC Caron, Vanessa Ruta, LF Abbott, and Richard Axel.  Random convergence of olfactory inputs in the drosophila mushroom body.  _Nature_ , 497(7447):113–117, 2013.
  * Chacko et al. [2020] Rinu Chacko, Deepak Jain, Manasi Patwardhan, Abhishek Puri, Shirish Karande, and Beena Rai.  Data based predictive models for odor perception.  _Scientific reports_ , 10(1):1–13, 2020.
  * Clyne et al. [1999] Peter J Clyne, Coral G Warr, Marc R Freeman, Derek Lessing, Junhyong Kim, and John R Carlson.  A novel family of divergent seven-transmembrane proteins: candidate odorant receptors in drosophila.  _Neuron_ , 22(2):327–338, 1999.
  * Cohen et al. [2017] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik.  Emnist: Extending mnist to handwritten letters.  In _2017 International Joint Conference on Neural Networks (IJCNN)_ , pages 2921–2926. IEEE, 2017.
  * Couto et al. [2005] Africa Couto, Mattias Alenius, and Barry J Dickson.  Molecular, anatomical, and functional organization of the drosophila olfactory system.  _Current Biology_ , 15(17):1535–1547, 2005.
  * Dasgupta et al. [2017] Sanjoy Dasgupta, Charles F Stevens, and Saket Navlakha.  A neural algorithm for a fundamental computing problem.  _Science_ , 358(6364):793–796, 2017.
  * Deisig et al. [2001] Nina Deisig, Harald Lachnit, Martin Giurfa, and Frank Hellstern.  Configural olfactory learning in honeybees: negative and positive patterning discrimination.  _Learning & Memory_, 8(2):70–78, 2001.
  * Devaud et al. [2015] Jean-Marc Devaud, Thomas Papouin, Julie Carcaud, Jean-Christophe Sandoz, Bernd Grünewald, and Martin Giurfa.  Neural substrate for higher-order learning in an insect: mushroom bodies are necessary for configural discriminations.  _Proceedings of the National Academy of Sciences_ , 112(43):E5854–E5862, 2015.
  * Egawa et al. [2006] Toru Egawa, Akiyo Kameyama, and Hiroshi Takeuchi.  Structural determination of vanillin, isovanillin and ethylvanillin by means of gas electron diffraction and theoretical calculations.  _Journal of Molecular Structure_ , 794(1-3):92–102, 2006.
  * Eichler et al. [2017] Katharina Eichler, Feng Li, Ashok Litwin-Kumar, Youngser Park, Ingrid Andrade, Casey M Schneider-Mizell, Timo Saumweber, Annina Huser, Claire Eschbach, Bertram Gerber, et al.  The complete connectome of a learning and memory centre in an insect brain.  _Nature_ , 548(7666):175–182, 2017.
  * Endo et al. [2020] Keita Endo, Yoshiko Tsuchimoto, and Hokto Kazama.  Synthesis of conserved odor object representations in a random, divergent-convergent network.  _Neuron_ , 108(2):367–381, 2020.
  * Fishilevich and Vosshall [2005] Elane Fishilevich and Leslie B Vosshall.  Genetic and functional subdivision of the drosophila antennal lobe.  _Current Biology_ , 15(17):1548–1553, 2005.
  * Glorot and Bengio [2010] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * Glorot et al. [2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 315–323. JMLR Workshop and Conference Proceedings, 2011.
  * Gruntman and Turner [2013] Eyal Gruntman and Glenn C Turner.  Integration of the olfactory code across dendritic claws of single mushroom body neurons.  _Nature neuroscience_ , 16(12):1821–1829, 2013\.
  * Hall and Kier [1991] Lowell H Hall and Lemont B Kier.  The molecular connectivity chi indexes and kappa shape indexes in structure-property modeling.  _Reviews in computational chemistry_ , pages 367–422, 1991.
  * Honegger et al. [2011] Kyle S Honegger, Robert AA Campbell, and Glenn C Turner.  Cellular-resolution population imaging reveals robust sparse coding in the drosophila mushroom body.  _Journal of neuroscience_ , 31(33):11772–11785, 2011.
  * Huang and Chen [2007] Guang-Bin Huang and Lei Chen.  Convex incremental extreme learning machine.  _Neurocomputing_ , 70(16-18):3056–3062, 2007\.
  * Huang et al. [2006a] Guang-Bin Huang, Lei Chen, Chee Kheong Siew, et al.  Universal approximation using incremental constructive feedforward networks with random hidden nodes.  _IEEE Trans. Neural Networks_ , 17(4):879–892, 2006a.
  * Huang et al. [2006b] Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew.  Extreme learning machine: theory and applications.  _Neurocomputing_ , 70(1-3):489–501, 2006b.
  * Igelnik and Pao [1995] Boris Igelnik and Yoh-Han Pao.  Stochastic choice of basis functions in adaptive function approximation and the functional-link net.  _IEEE transactions on Neural Networks_ , 6(6):1320–1329, 1995.
  * Inada et al. [2017] Kengo Inada, Yoshiko Tsuchimoto, and Hokto Kazama.  Origins of cell-type-specific olfactory processing in the drosophila mushroom body circuit.  _Neuron_ , 95(2):357–367, 2017.
  * Keller and Vosshall [2016] Andreas Keller and Leslie B Vosshall.  Olfactory perception of chemically diverse molecules.  _BMC neuroscience_ , 17(1):1–17, 2016.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Li et al. [1997] Jin-Yan Li, Wing Sun Chow, Boris Igelnik, and Yoh-Han Pao.  Comments on" stochastic choice of basis functions in adaptive function approximation and the functional-link net"[with reply].  _IEEE transactions on neural networks_ , 8(2):452–454, 1997.
  * Liang et al. [2006] Nan-Ying Liang, Guang-Bin Huang, Paramasivan Saratchandran, and Narasimhan Sundararajan.  A fast and accurate online sequential learning algorithm for feedforward networks.  _IEEE Transactions on neural networks_ , 17(6):1411–1423, 2006.
  * Lin et al. [2014] Andrew C Lin, Alexei M Bygrave, Alix De Calignon, Tzumin Lee, and Gero Miesenböck.  Sparse, decorrelated odor coding in the mushroom body enhances learned odor discrimination.  _Nature neuroscience_ , 17(4):559–568, 2014.
  * Mamlouk and Martinetz [2004] Amir Madany Mamlouk and Thomas Martinetz.  On the dimensions of the olfactory perception space.  _Neurocomputing_ , 58:1019–1025, 2004.
  * Meulemans et al. [2020] Alexander Meulemans, Francesco S Carzaniga, Johan AK Suykens, João Sacramento, and Benjamin F Grewe.  A theoretical framework for target propagation.  _arXiv preprint arXiv:2006.14331_ , 2020.
  * Owald and Waddell [2015] David Owald and Scott Waddell.  Olfactory learning skews mushroom body output pathways to steer behavioral choice in drosophila.  _Current opinion in neurobiology_ , 35:178–184, 2015.
  * Pao and Takefuji [1992] Y-H Pao and Yoshiyasu Takefuji.  Functional-link net computing: theory, system architecture, and functionalities.  _Computer_ , 25(5):76–79, 1992.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In _NIPS-W_ , 2017.
  * Scardapane and Wang [2017] Simone Scardapane and Dianhui Wang.  Randomness in neural networks: an overview.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 7(2):e1200, 2017.
  * Schmidt et al. [1992] Wouter F Schmidt, Martin A Kraaijveld, Robert PW Duin, et al.  Feed forward neural networks with random weights.  In _International Conference on Pattern Recognition_ , pages 1–1. IEEE COMPUTER SOCIETY PRESS, 1992.
  * Sell [2006] Charles S Sell.  On the unpredictability of odor.  _Angewandte Chemie International Edition_ , 45(38):6254–6261, 2006.
  * Seung [2012] Sebastian Seung.  _Connectome: How the brain’s wiring makes us who we are_.  Houghton Mifflin Harcourt, 2012.  ISBN 978-0547508184.
  * Sporns et al. [2005] Olaf Sporns, Giulio Tononi, and Rolf Kötter.  The human connectome: a structural description of the human brain.  _PLoS Comput Biol_ , 1(4):e42, 2005.
  * Takemura et al. [2017] Shin-ya Takemura, Yoshinori Aso, Toshihide Hige, Allan Wong, Zhiyuan Lu, C Shan Xu, Patricia K Rivlin, Harald Hess, Ting Zhao, Toufiq Parag, et al.  A connectome of a learning and memory center in the adult drosophila brain.  _Elife_ , 6:e26975, 2017.
  * van Schaik and Tapson [2015] André van Schaik and Jonathan Tapson.  Online and adaptive pseudoinverse solutions for elm weights.  _Neurocomputing_ , 149:233–238, 2015.
  * Vosshall et al. [1999] Leslie B Vosshall, Hubert Amrein, Pavel S Morozov, Andrey Rzhetsky, and Richard Axel.  A spatial map of olfactory receptor expression in the drosophila antenna.  _Cell_ , 96(5):725–736, 1999.
  * Wang et al. [2003] Jing W Wang, Allan M Wong, Jorge Flores, Leslie B Vosshall, and Richard Axel.  Two-photon calcium imaging reveals an odor-evoked map of activity in the fly brain.  _Cell_ , 112(2):271–282, 2003.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] We specified the
limitations of our work in Appendix A.5.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes] We
described it in Broader Impact.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [Yes] We
described all the assumptions of our work in the Section 3 and Appendix A.2.

    2. (b)

Did you include complete proofs of all theoretical results? [Yes] All
biological intuitions for our model were described in the Section 3.1, and the
full derivative used in our optimization method was specified in Appendix A.1.

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
All source codes for our model and optimization method are available at the
link in Appendix A.4.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] All the details of our experiments including
methods, hyperparameter search space and implementations were described in
Appendix A.4.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] All experimental results were conducted
averaged by five experiments with 95% confidence interval.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The hardware
specifications of servers that we used to experiment were described in
Appendix A.4.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] All assets
or datasets that we used were published openly and we left all the reference
for each of them in the Section 4.

    2. (b)

Did you mention the license of the assets? [N/A]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Appendix

###  A.1 The Derivative of $\nabla_{s_{ji}}$

All of our experiments make use of classification tasks, with model outputs
generated by applying the softmax function. Consequently, the objective
function $\mathcal{L}$ that we use is the categorical cross-entropy loss
function. The loss function for the odor-perception task is a binary cross-
entropy loss, which is a special form of the categorical cross-entropy loss.

Given an input vector $\textbf{x}\in\mathbb{R}^{d}$ and one-hot-encoded target
vector $\textbf{y}\in\mathbb{R}^{c}$, consider the case where the output
weight matrix $\boldsymbol{\beta}^{*}$ of the KCNet was selected to best fit
the linear system $\boldsymbol{\beta}^{T}\textbf{h}=\textbf{y}$ where h is the
hidden layer output vector of the network, as in:

$\displaystyle\textbf{h}=\begin{pmatrix}h_{1}(\hat{\textbf{x}})\\\ \vdots\\\
h_{\mathcal{B}}(\hat{\textbf{x}})\end{pmatrix}_{\mathcal{B}\times
1}\;\text{and}\quad\boldsymbol{\beta}^{*}=\begin{pmatrix}\boldsymbol{\beta}_{1}^{T}\\\
\vdots\\\ \boldsymbol{\beta}_{\mathcal{B}}^{T}\\\
\end{pmatrix}_{\mathcal{B}\times c},$ (11)

and $\hat{\textbf{x}}$ is the processed input in Eq. (5). Based on this, the
prediction $\hat{\textbf{y}}$ of the KCNet is:

$\hat{\textbf{y}}=\textsc{softmax}(\textbf{z})=\textsc{softmax}(\boldsymbol{\beta}^{*^{T}}\textbf{h}).$
(12)

In the backward pass, the batch gradient descent using validation set is
performed. Given the validation set, the gradient $\nabla_{s_{ji}}$ can be
computed as follows:

$\begin{split}\nabla s_{ji}&\approx\frac{\partial\mathcal{L}}{\partial
w_{ji}}\\\
&=\frac{\partial\mathcal{L}}{\partial\textbf{z}}\frac{\partial\textbf{z}}{\partial
h_{j}}\frac{\partial
h_{j}}{\partial\hat{x}_{j}}\frac{\partial\hat{x}_{j}}{\partial\bar{x}_{j}}\frac{\partial\bar{x}_{j}}{\partial
w_{ji}}\end{split}$ (13)

where

$\displaystyle\frac{\partial\mathcal{L}}{\partial\textbf{z}}=(\hat{\textbf{y}}-\textbf{y})\in\mathbb{R}^{c},\qquad\frac{\partial\textbf{z}}{\partial
h_{j}}=\begin{pmatrix}\beta_{1j}\\\ \vdots\\\ \beta_{cj}\end{pmatrix}_{c\times
1},\qquad\frac{\partial
h_{j}}{\partial\hat{x}_{j}}=\begin{cases}1&h_{j}>0\;\text{in the forward
pass},\\\ 0&\text{otherwise}\end{cases}$ (14)
$\displaystyle\frac{\partial\hat{x}_{j}}{\partial\bar{x}_{j}}=1-\frac{1}{\mathcal{B}},\qquad\text{and}\qquad\frac{\partial\bar{x}_{j}}{\partial
w_{ji}}=\begin{cases}x_{i}&w_{ji}=1,\\\ 0&\text{otherwise}\end{cases}.$ (15)

Because $\partial\hat{x}_{j}/\partial\bar{x}_{j}$ is a constant and can be
integrated with the learning rate $\alpha$, we can write $\nabla_{s_{ji}}$
concisely as follows:

$\nabla_{s_{ji}}=\begin{cases}x_{i}\sum_{k=1}^{c}(\hat{y}_{k}-y_{k})\beta_{kj}&h_{j}>0\;\text{in
the forward pass and }\;w_{ji}=1,\\\ 0&\text{otherwise}\end{cases}.$ (16)

###  A.2 Detailed characteristics of the KCNet

Although the KCNet is devised from the underlying perspectives of biology and
neuroscience, several apparent characteristics of the model can also be
interpreted from the perspective of computer science and machine learning.

First of all, determining the weight between the input and the hidden layers
in the KCNet can be seen as a special case of random weight generation in a
NNRW, and the KCNet also uses a bounded nonlinear piecewise continuous
activation function, which is the ReLU in our setting. Therefore, KCNet
inherits the _universal approximation_ property of NNRWs, which means that a
single-hidden-layer network with randomly generated hidden nodes and with a
broad type of activation functions can universally approximate any continuous
target functions in any compact subset $\mathcal{X}\subseteq\mathbb{R}^{d}$
[32, 31, 30]. So, theoretically, the more the number of hidden nodes in the
KCNet (i.e., the wider the hidden layer), the better its performance (see the
details of KCNet performance results in Section 4).

The second property of the proposed network is that from the point of view in
computer science, the hidden-layer output can be viewed as the output of a
hash function. Dasgupta et al. [18] took this perspective to interpret the
Kenyon Cells (KCs) as locality-sensitive tags for odors; the more similar a
pair of odors, the more similar their assigned tags. Based on this idea, they
developed a brain-inspired class of Locality-Sensitive Hash (LSH) functions
[3] for efficiently searching approximate nearest neighbors of high-
dimensional points. Because our formulation is partly derived from those
insights, our model also can generate tags that preserve the neighborhood
structure of the input points using distance-preserving embedding of points,
as a family of LSH. Whereas we have focused on the trade-off between
conciseness and expressiveness in the width of the KC hidden layer in
artificial neural network applications, Dasgupta et al. [18] provided
computational theory to better understand the performance of architectures
matching the scale of the fly brain. Moreover, in their application, a
Hebbian-like update rule was used for unsupervised learning of whether an odor
had been experienced before (in contrast to the supervised learning we
employed). However, their analysis combined with ours implies that KCNet may
be employed in unsupervised learning applications, and there will likely be a
similar trade-off between the width of the hidden layer and the overall
performance of the trained networks.

Finally, to mimic the KCs receiving global inhibition from each other, we
subtract the mean of all hidden nodes outputs from each intermediate output in
Eq. (5). That computation can be viewed as a form of the _layer normalization_
in machine learning [5], which helps to maintain the contributions of each
feature so that the network can be unbiased, reduce the internal covariance
shift, and accelerate the optimization process. Thus, our model helps to
import insights from machine learning into the interpretation of the adaptive
function of global inhibition in neuronal structures like the mushroom body.

###  A.3 Complete algorithm of the Dynamic Optimization Algorithm (DOA)

1: Input:
$\mathcal{D}=\\{\textbf{x}_{i}\in\mathbb{R}^{d},\textbf{y}_{i}\in\mathbb{R}^{c}\\},i=1,\dots,N$
(Dataset, excluding testing set, with the size $N$ samples),

2: Input: $\mathcal{B}\in\mathbb{R}$ (The number of hidden units),

3: Input: $\alpha\in\mathbb{R}$ (Learning rate)

4: Output: $\textbf{W}\in\mathbb{R}^{\mathcal{B}\times d}$ (The weight matrix
between the input and the hidden layers),

5: Output: $\boldsymbol{\beta}\in\mathbb{R}^{c\times\mathcal{B}}$ (The weight
matrix between the hidden and the output layers)

{/*Initialization*/}

6: Create the initial weight matrix
$\textbf{W}\in\mathcal{R}^{\mathcal{B}\times d}$ and the score matrix
$\textbf{S}\in\mathcal{R}^{\mathcal{B}\times d}$

7: repeat

8: Randomly split $\mathcal{D}$ into training set $\mathcal{D}_{train}$ and
validation set $\mathcal{D}_{val}$

{/*In forward pass*/}

9: Build a model $\mathcal{M}$ using W and train $\boldsymbol{\beta}$ using
the training set $\mathcal{D}_{train}$

10: $\mathcal{P}\leftarrow$ Evaluate the model $\mathcal{M}$ using validation
set $\mathcal{D}_{val}$

11: if  _stopping criteria_ are satisfied then

12: break

13: end if

{/*In backward pass*/}

14: Compute $\nabla_{\textbf{S}}$ using the validation set $\mathcal{D}_{val}$

15: $\textbf{S}\leftarrow\textbf{S}-\alpha\nabla_{\textbf{S}}$

16: $\textbf{S}\leftarrow\text{clip}(\textbf{S},-1,1)$

17: $\textbf{W}\leftarrow\rho_{\textbf{S}\rightarrow\textbf{W}}(\textbf{S})$

18: until

Algorithm 1 Dynamic Optimization Algorithm (DOA) for Feature Selection

###  A.4 Details of Experiments

In this section, we describe the details of our experimental results. The
following details apply in common to the two tasks we addressed, odor
perception and image classification. Details specific to each task are
described in subsections of each task. All the following results were averaged
over five experimental replications.

##### Hardware specification of the server

The hardware specification of the server that we used to experiment is as
follows:

  * •

CPU: Intel® CoreTM i7-6950X CPU @ 3.00GHz (up to 3.50 GHz)

  * •

RAM: 128 GB (DDR4 2400MHz)

  * •

GPU: NVIDIA GeForce Titan Xp GP102 (Pascal architecture, 3840 CUDA Cores @ 1.6
GHz, 384 bit bus width, 12 GB GDDR G5X memory)

##### Source code

All source codes are available at
https://anonymous.4open.science/r/KCNet-E858. They have been provided as
electronic supplementary material as well.

##### Hyperparameter search space

Table 3 shows the hyperparameter search space of all models for all tasks. To
optimize over this hyperparameter space, we used the Optuna [1] framework with
100 iterations. It is possible to dynamically construct the parameter search
space because the framework allows users to formulate hyperparameter
optimization as the maximization/minimization process of an objective function
that takes a set of hyperparameters as input and returns a validation score.
Furthermore, it provides efficient sampling methods, such as relational
sampling that exploits the correlations among the parameters.

The hyperparameter search space for other non-KCNet machine learning
algorithms used as benchmarks were taken to match Chacko et al. [14] for odor
perception tasks.

To set the number of submodels for the Ensemble-DOA-applied KCNet, we
empirically determined that it was the most efficient to divide a large-sized
weight matrix of the KCNet into at least 10 small matrices for the tasks that
we addressed. Furthermore, we set a value such that the product of the number
of submodels and the number of hidden nodes for each submodel in the Ensemble-
DOA-applied KCNet equals the number of hidden nodes in the KCNet or DOA-
applied KCNet. By doing so, they have the exact dimension of the output weight
matrix.

Table 3: Hyperparameter Search Space Model | Parameters | Search Space
---|---|---
Adaboost | learning_rate | [0.1, 1.0]
n_estimators | [10, 100]
algorithm | {’SAMME’, ’SAMME.R’}
Gradient Boosting Machine (GBM) | learning_rate | [0.05, 1.0]
n_estimators | [10, 100]
max_depth | [2, 5]
min_samples_split | [10, 200]
min_samples_leaf | [30, 70]
max_features | [8, 159]
subsample | [0.6, 1.0]
XGBoost | max_depth | [2, 5]
min_child_weight | [1, 6]
gamma | [0.1, 10]
subsample | [0.6, 1]
colsample_bytree | [0.5, 1]
reg_alpha | [0.1, 20]
reg_lambda | [0.001, 100]
learning_rate | [0.01, 1.0]
n_estimators | [10, 200]
Random Forest | criterion | {’gini’, ’entropy’}
n_estimators | [50, 100]
max_depth | [2, 5]
min_samples_split | [2, 70]
min_samples_leaf | [1, 50]
K-Nearest Neighbors (KNN) | weights | {’uniform’, ’distance’}
p | {1, 2, 3}
n_neighbors | [2, 30]
Support Vector Machine (SVM) | C | [2e-3, 2e7]
KCNet | lambda | [2e-2, 2e3]
KCNet w/ DOA | n_epochs | {5, 10, 50, 100}
learning_rate | [5e-1, 5e-7]
stop_metric | [0.80, 0.97]
KCNet w/ Ensemble DOA | n_epochs | {5, 10, 50, 100}
learning_rate | [5e-1, 5e-7]
stop_metric | [0.80, 0.97]

####  A.4.1 Odor Perception

##### Methods and Implementations

Figure 4: Overall workflow of model development for odor perception tasks

Figure 4 depicts the overall workflow for odor-perception tasks. We used the
compound identification (CID) number of 480 odorants included in the original
dataset to generate the SMILES notation of the molecules, which was utilized
to compute the RDKit descriptors. We obtained the SMILES notations of the
odorant compounds using the PubChemPy package of Python programming language.
RDKit descriptors were calculated using ChemDes, a free web-based platform for
the calculation of molecular descriptors. All the preprocessing steps and
model training was performed using Python libraries: NumPy, pandas, imblearn,
and scikit-learn. All models that we tested are provided by scikit-learn, and
we also implemented our proposed model compatible with scikit-learn. The
ADASYN class of the imblearn package was used to scale and oversample the
training data during the five-fold cross-validation. In addition, we set the
random_state to 1 for reproducing our results readily. Figures are generated
using Matplotlib and Seaborn libraries supported by Python programming
language.

##### Results

Figure 5 shows the results about the KCNet, the DOA-applied KCNet, and the
Ensemble-DOA-applied KCNet for Sweet/Non-sweet OC. Figures 5(a) and 5(b)
represent the initial preference score and weight matrix, respectively, before
applying the DOA to the KCNet. Figure 5(c) depicts the change in the
validation-weighted F1 score during the DOA, and this resulted in the final
weight matrix of the KCNet shown in Fig. 5(d). Compared to the initial weight
matrix in Fig. 5(b), the final matrix was more densified to enable the
exploitation of the combination of inputs. We applied the Ensemble DOA to the
KCNet by dividing the weight matrix with 1,300 hidden nodes into 13 submodels
with 100 hidden nodes for each submodel. Figure 5(e) demonstrates the change
of the validation-weighted F1 score of 13 submodels during the Ensemble DOA
and the final weight matrix assembled by 13 submodels was shown in Fig. 5(f).
Compared to the weight matrix of the DOA-applied KCNet in Fig. 5(d), it
appears to have a slightly more regular pattern, which seems to be due to the
concatenation of multiple small matrices. The best hyperparameter settings for
the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for
_Sweet_ OC as follows:

  * •

The KCNet : lambda = 100

  * •

The DOA-applied KCNet : n_epoch = 100, learning_rate = 0.5, stop_metric = 0.87

  * •

The Ensemble-DOA-applied KCNet : n_epoch = 100, learning_rate = 0.5,
stop_metric = 0.90

Similar to the above description, Fig. 6 demonstrates the results about the
KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for
Musky/Non-musky OC. The best hyperparameter settings for the KCNet, the DOA-
applied KCNet and the Ensemble-DOA-applied KCNet for _Musky_ OC as follows:

  * •

The KCNet : lambda = 115

  * •

The DOA-applied KCNet : n_epoch = 100, learning_rate = 0.5, stop_metric = 0.80

  * •

The Ensemble-DOA-applied KCNet : n_epoch = 100, learning_rate = 0.1,
stop_metric = 0.82

(a) Initial preference score matrix of the KCNet with 1,300 hidden nodes
(x-axis: dim. of input features, y-axis: dim. of hidden units)

(b) Initial weight matrix of the KCNet with 1,300 hidden nodes (x-axis: dim.
of input features, y-axis: dim. of hidden units)

(c) Validation weighted F1 score during the DOA with 100 epochs. (x-axis: # of
epochs, y-axis: validation weighted F1 score)

(d) Final weight matrix of the KCNet with 1300 hidden nodes after the DOA
(x-axis: dim. of input features, y-axis: dim. of hidden units)

(e) Validation weighted F1 score of 13 submodels during the Ensemble DOA with
100 epochs x-axis: # of epochs, y-axis: validation weighted F1 score)

(f) Final weight matrix of the Ensemble-DOA-KCNet concatenated by 13 submodels
with 100 hidden nodes for each submodel (x-axis: dim. of input features,
y-axis: dim. of hidden units)

Figure 5: Results of the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-
applied KCNet for Sweet/Non-sweet

(a) Initial preference score matrix of the KCNet with 700 hidden nodes
(x-asis: dim. of input features, y-axis: dim. of hidden units)

(b) Initial weight matrix of the KCNet with 700 hidden nodes (x-asis: dim. of
input features, y-axis: dim. of hidden units)

(c) Validation weighted F1 score during the DOA with 100 epochs (x-axis: # of
epochs, y-axis: validation weighted F1 score)

(d) Final weight matrix of the KCNet with 700 hidden nodes after the DOA
(x-asis: dim. of input features, y-axis: dim. of hidden units)

(e) Validation weighted F1 score of 10 submodels during Ensemble DOA with 100
epochs (x-axis: # of epochs, y-axis: validation weighted F1 score)

(f) Final weight matrix of the Ensemble-DOA-applied KCNet concatenated by 10
submodels with 70 hidden nodes for each submodel (x-asis: dim. of input
features, y-axis: dim. of hidden units)

Figure 6: Results of the KCNet the DOA-applied KCNet, and the Ensemble-DOA-
applied KCNet for Musky/Non-musky

####  A.4.2 Image Classification

##### Methods and implementations

Here, we explain the details of implementing the Fully-trained Single-Hidden-
layer Neural Network (FSHN). Compared to the KCNet, all the parameters
including hidden weights in the FSHN are adaptable. Thus, for the sake of
accurate comparison of performance, it is important to set the number of
learnable parameters of the FSHN as similar to those of the KCNet. The number
of adaptable parameters for the KCNet is determined by multiplying the number
of hidden nodes with the number of output nodes. So, the numbers of learnable
parameters of the KCNet for MNIST, Fashion-MNIST, and EMNIST-Balanced are
65,000 (6,500 $\times$ 10), 70,000 (7,000 $\times$ 10), and 211,500 (4,500
$\times$ 47), respectively, whereas the numbers of adaptable parameters of the
FSHN are 65200, 70765, and 211,375, respectively, as shown in Table 2. Because
the FSHN exploits an usual linear model supported by PyTorch and it includes
biases as parameters, we set them to match the number of parameters of the
KCNet as closely as possible. The numbers of hidden nodes in the FSHN that we
set can be seen at source codes that we provided.

For the hyperparameter setting for the FSHN, by referring to the basic
tutorial of constructing a classifier using MNIST that PyTorch provides, we
define the FSHN as follows: training_batch_size = 64, test_batch_size = 1000,
epoch = 5, learning_rate = 0.1. This setting was shared for other image
classification tasks. The reason why the number of epochs was set to 5 for the
FSHN was the number that allows the FSHN to achieve better performance for
MNIST than the KCNet.

##### Analysis of the DOA

In this section, we explain the limitation of the DOA and how the limit
resulted in the invention of the Ensemble DOA. To analyze the effectiveness of
applying the DOA to the KCNet, we tested our model and optimization method
using MNIST dataset. It was not appropriate to analyze the performance of the
DOA using the odor-perception datasets because, as we mentioned, the number of
data samples for odor perception was not enough (only 960 data samples), and
the dataset had the inherited characteristic of imbalance. In contrast, the
MNIST dataset is large enough and has a balanced distribution of all classes.

Figure 7 depicts the efficiency of applying the DOA to the KCNet with a small
weight matrix and the increased sensitivity of the KCNet to learning rate as
the number of hidden units increases. Figures 7(a) and 7(b) show a noticeable
change of the weight matrix of the KCNet before and after the DOA. This
indicates the intuitive effectiveness of applying the DOA to the KCNet. The
method allows the model to explore a useful combination of inputs, and the
majority of meaningful features in MNIST images were chosen and exploited to
allow the model to have a powerful discrimination ability. However, this
phenomenon can likely be shown when applying the DOA to the KCNet with a
relatively small weight matrix.

From Figs. 7(c) to 7(f), the convergence of validation accuracy of the KCNet
with the same value of learning rate, which is 0.5, during the DOA was shown
as the number of its hidden nodes increases. As shown in Fig. 7(c), the
validation of accuracy of the KCNet with 100 hidden nodes was converged well,
and this resulted in an notable transformation of the weight matrix shown in
Figs. 7(a) and 7(b). This corresponded to the substantial increase of test
accuracy in Fig. 7(g). However, the larger the weight matrix of the KCNet, the
more perturbed the convergence of validation accuracy and the more sensitive
the KCNet to the learning rate. This gave rise to a marginal improvement of
the DOA-applied KCNet as its weight matrix is too large to properly deal with
the complexity of exploring a set of inputs in Fig. 7(g). Furthermore, the
larger the weight matrix, the more burdensome the computation of the backward
pass of the DOA.

Thus, to reduce the computational burden of the backward pass and to exploit
the full power of the DOA, we devised the Ensemble DOA, the ensemble method of
assembling a weight matrix by concatenating several smaller matrices after
applying the DOA to each small matrix. Figure 7(h) demonstrates the running
time, including the model configuration time, training time, and evaluation
time, of the KCNet, the DOA-applied KCNet. The running time refers to the time
spent running on a single CPU and RAM because we did not want to include any
redundant time such as the time to exchange the data between CPU and GPU. As
shown in Fig. 7(h), the running time of the Ensemble-DOA-applied KCNet is much
faster than the DOA-applied KCNet, and the application of the Ensemble DOA
allows the KCNet to achieve better performance than the DOA-applied KCNet. The
results can be shown in Tables 1 and 2 for all tasks.

(a) Initial weight matrix with 100 hidden nodes before the DOA

(b) Final weight matrix with 100 hidden nodes after the DOA with 50 epochs

(c) Validation accuracy with 100 hidden nodes during the DOA with 50 epochs

(d) Validation accuracy with 500 hidden nodes during the DOA with 50 epochs

(e) Validation accuracy with 1,000 hidden nodes during the DOA with 50 epochs

(f) Validation accuracy with 2,000 hidden nodes during the DOA with 50 epochs

(g) The change of test accuracy before and after applying the DOA

(h) The running time (sec) for the KCNet variants

Figure 7: For image classification using MNIST, the change of weight matrix of
the KCNet before and after the DOA and the relationship between validation
accuracy depending on its hidden size and learning rate = 0.05

##### Results

(a) Test accuracy of KCNet and ELM for Fashion-MNIST depending on their hidden
sizes

(b) Test accuracy of KCNet and ELM for EMNIST-Balanced depending on their
hidden sizes

Figure 8: Test accuracy of KCNet and ELM for Fashion-MNIST and EMNIST-Balanced
depending on their hidden sizes

Figure 3(b) in the main text shows the running time, including model
configuration, training, and evaluation time of the KCNet, ELM, and FSHN using
a single CPU and RAM only (i.e., not making use of a GPU). This is because we
wanted to exclude any redundant time such as the data transmission time
between GPU and GPU.

Figure 8 (and 3(a) in the main text) shows that the KCNet mostly outperforms
the ELM on all image benchmark datasets, depending on their size of hidden
nodes. We set the following range of hidden nodes to get the result within the
limitation of RAM of GPU we used; $\\{500,1000,\dots,6500\\}$ for MNIST,
$\\{500,1000,\dots,7500\\}$ for Fashion-MNIST, and $\\{500,1000,\dots,4500\\}$
for EMNIST-Balanced.

Figures 9 through 11 depict all the additional results about the DOA-applied
KCNet and the Ensemble-DOA-applied KCNet for MNIST, Fashion-MNIST, and EMNIST-
Balanced, respectively. We observed that it had a tendency that the DOA
couldn’t improve the KCNet if its weight matrix was relatively large. As shown
in Fig. 9(c) indicating the change of validation accuracy while applying the
DOA to the KCNet, the validation accuracy gradually decreased as the DOA
progressed. The difference between the weight matrix before (Fig. 9(b)) and
after the DOA (Fig. 9(d)) was very marginal, which might be the cause of
degrading the performance of the DOA-applied KCNet compared to the original
KCNet shown in Table 2. However, the validation accuracy of 10 submodels
during the Ensemble DOA in Fig. 9(e) was converged well, and Fig. 9(f)
represents a noticeable improvement of the weight matrix. Therefore, we were
able to validate the hyperparameter setting for the Ensemble-DOA-applied KCNet
based on the increasing trend of its submodels’ validation accuracy, which was
applied for other image benchmark datasets. The best hyperparameter settings
for the KCNet, the DOA-applied KCNet, and the Ensemble-DOA-applied KCNet for
MNIST as follows:

  * •

KCNet: lambda = 13

  * •

DOA-applied KCNet: n_epoch = 5, learning_rate = 1e-4, stop_metric = 0.96

  * •

Ensemble-DOA-applied KCNet : n_epoch = 100, learning_rate = 0.01, stop_metric
= 0.87

The additional experimental results for Fashion-MNIST (Fig. 10) and EMNIST-
Balanced (Fig. 11) also tend to be similar to those for MNIST. We specify the
best hyperparameter settings for the KCNet, the DOA-applied KCNet, and the
Ensemble-DOA-applied KCNet for Fashion-MNIST as follows:

  * •

KCNet: lambda = 5

  * •

DOA-applied KCNet: n_epoch = 5, learning_rate = 1e-5, stop_metric = 0.90

  * •

Ensemble-DOA-applied KCNet: n_epoch = 20, learning_rate = 5e-5, stop_metric =
0.84

The best hyperparameter settings for the KCNet, the DOA-applied KCNet, and the
Ensemble-DOA-applied KCNet for EMNIST-Balanced are as follows:

  * •

KCNet: lambda = 10

  * •

DOA-applied KCNet: n_epoch = 5, learning_rate = 1e-4, stop_metric = 0.90

  * •

Ensemble-DOA-applied KCNet: n_epoch = 30, learning_rate = 0.01, stop_metric =
0.85

(a) Initial preference score matrix of the KCNet with 6,500 hidden nodes
before the DOA (x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(b) Initial weight matrix of the KCNet with 6,500 hidden nodes before the DOA
(x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(c) Validation accuracy during the DOA with 5 epochs (x-axis: # of epochs,
y-axis: validation accuracy)

(d) Final weight matrix of the DOA-applied KCNet with 6,500 hidden nodes
(x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(e) Validation accuracy of 10 submodels during the Ensemble DOA with 100
epochs and stop_metric = 0.87 (x-axis: # of epochs, y-axis: validation
accuracy)

(f) Final weight matrix of the Ensemble-DOA-applied-KCNet with 6,500 hidden
nodes (x-axis: dim. of input features, y-axis: dim. of hidden nodes)

Figure 9: For MNIST, results about the DOA-applied KCNet and the Ensemble-DOA-
applied KCNet

(a) Initial preference score matrix of the KCNet with 7,000 hidden nodes
before the DOA (x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(b) Initial weight matrix of the KCNet with 7,000 hidden nodes before the DOA
(x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(c) Validation accuracy during DOA with 5 epochs (x-axis: # of epochs, y-axis:
validation accuracy)

(d) Final weight matrix of the DOA-applied-KCNet with 7,000 hidden nodes
(x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(e) Validation accuracy of 10 submodels during Ensemble DOA with 20 epochs and
stop_metric = 0.84 (x-axis: # of epochs, y-axis: validation accuracy)

(f) Final weight matrix of the Ensemble-DOA-applied-KCNet with 7,000 hidden
nodes (x-axis: dim. of input features, y-axis: dim. of hidden nodes)

Figure 10: For Fashion-MNIST, results about the DOA-applied KCNet and the
Ensemble-DOA-applied KCNet

(a) Initial preference score matrix of the KCNet with 4,500 hidden nodes
before the DOA (x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(b) Initial weight matrix of the KCNet with 4,500 hidden nodes before the DOA
(x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(c) Validation accuracy during DOA with 5 epochs (x-axis: # of epochs, y-axis:
validation accuracy)

(d) Final weight matrix of the DOA-applied-KCNet with 4,500 hidden nodes
(x-axis: dim. of input features, y-axis: dim. of hidden nodes)

(e) Validation accuracy of 10 submodels during Ensemble DOA with 30 epochs and
stop_metric = 0.85 (x-axis: # of epochs, y-axis: validation accuracy)

(f) Final weight matrix of the Ensemble-DOA-applied KCNet with 4,500 hidden
nodes (x-axis: dim. of input features, y-axis: dim. of hidden nodes)

Figure 11: For EMNIST-Balanced, results about the DOA-applied KCNet and the
Ensemble-DOA-applied KCNet

###  A.5 Limitations

We showed the potential for our model through the experimental results.
However, there are several limitations that we should handle in the future.

In our experiments, we trained and tested our model in supervised batch
learning, which has the limitation of scaling the size of the network. This
problem has been dealt with in NNRWs, and there are several works about
adapting the output weight using online learning [38, 51], which is more
practical for large datasets. Online learning also is more congruent with the
biological systems that inspired KCNet.

Vulnerability to noisy inputs is the second limit of our model. The main
motivation of the KCNet is the transformation of odor stimuli between
glomeruli in the antennal lobe and KCs in the mushroom bodies. So our current
approach implicitly assumes that, as in the pre-processing of odors in the
actual insect’s brain, all input features are already pre-processed
sufficiently. The image benchmark datasets that we tested satisfied this
assumption by applying standard normalization to them, but these conditions
are often not applicable to datasets of real-world problems. For example, we
tested the KCNet on CIFAR10, an image benchmark dataset with more input
features due to the large input and three RGB channels. However, KCNet
achieved about 51% accuracy because the feature dimensions are relatively
large, and the purpose of the KCNet is not to distinguish between critical and
background features in the images. It instead is a model that generates unique
latent representations with combinations of input values. So, directly feeding
the real-world images as input into the KCNet may not be a good strategy to
take advantage of KCNet’s strengths. Just as the central complex (CX) in the
insect’s brain is responsible for integrating and elaborating visual
information, we anticipate combining an additional feature extractor as an
artificial CX, such as pre-trained models, with the KCNet would solve this
problem. The KCNet can be a good ad-hoc model for other models, and it will
show excellent performance in real-world image tasks if it receives sanitized
features processed by a pre-trained model or a simple convolutional network.