# The Multiple Subnetwork Hypothesis
Enabling Multidomain Learning by Isolating Task-Specific Subnetworks in
Feedforward Neural Networks

Jacob Renn, Ian Sotnek, Benjamin Harvey
AI Squared, Inc.
United States
{jacob.renn, ian.sotnek<EMAIL_ADDRESS>
&Brian Caffo
Department of Biostatistics
Johns Hopkins University
Baltimore, MD, USA
<EMAIL_ADDRESS>

###### Abstract

Neural networks have seen an explosion of usage and research in the past
decade, particularly within the domains of computer vision and natural
language processing. However, only recently have advancements in neural
networks yielded performance improvements beyond narrow applications and
translated to expanded multitask models capable of generalizing across
multiple data types and modalities. Simultaneously, it has been shown that
neural networks are overparameterized to a high degree, and pruning techniques
have proved capable of significantly reducing the number of active weights
within the network while largely preserving performance. In this work, we
identify a methodology and network representational structure which allows a
pruned network to employ previously unused weights to learn subsequent tasks.
We employ these methodologies on well-known benchmarking datasets for testing
purposes and show that networks trained using our approaches are able to learn
multiple tasks, which may be related or unrelated, in parallel or in sequence
without sacrificing performance on any task or exhibiting catastrophic
forgetting.

##  1 Introduction

Recent advancements in deep learning have seen artificial neural networks
(ANNs) match or exceed human-level performance in narrow tasks [1, 2, 3, 4].
When exposing artificial intelligence (AI) to environments that are flexible
and novel, however, narrow AI loses the ability to flexibly identify and react
to patterns. AI remains unable to match biological intelligence in flexibly
negotiating dynamically changing environments. This is because biological
learning enables the continuous acquisition of new knowledge and development
of new capabilities across multiple disparate inputs, all without the newly
learned information destroying internal state representations necessary to
perform previously learned skills or utilize previously acquired knowledge.

The inability of a traditional ANN to learn multiple tasks across
heterogeneous data sources without exhibiting catastrophic forgetting is a
significant obstacle on the path to generalizable AI. Solving this challenge
is a step towards enabling AI to emulate biological continuous multidomain
learning; rather than relying on separate ANNs to learn and perform multiple
tasks, a single neural network capable of learning multiple tasks can
replicate the flexibility of human learning, simplifying the deployment of the
capabilities of an AI system in dynamic, chaotic environments [5, 6].

As demonstrated in this manuscript, the training procedure described herein
outputs a single neural network containing multiple subnetworks, each of which
is capable of learning a single task. Practically speaking, this methodology
yields a single network with the capabilities of multiple disparate networks,
while also doing so more efficiently by only utilizing a small portion of the
original network’s weights for each task.

##  2 Background

In this work, we are concerned with overcoming catastrophic forgetting in the
development of ANNs capable of learning multiple tasks across disparate
domains. To accomplish this, we pose and investigate the Multiple Subnetwork
Hypothesis and build upon it to create a multitask ANN architecture capable of
learning multiple tasks across various different data types and with different
input and output shapes.

###  2.1 The Multiple Subnetwork Hypothesis

We propose the Multitple Subnetwork Hypothesis – that a dense, randomly-
initialized feedforward ANN contains within its architecture multiple disjoint
subnetworks which can be utilized together to learn and make accurate
predictions on multiple tasks, regardless of the degree of similarity between
tasks or input types. Additionally, we have developed the network distillation
procedure Reduction in Sub-Network Neuroplasticity (RSN2), which optimizes and
sparsifies dense ANNs, to test our hypothesis. Our approach enables state of
the art performance on multiple tasks within the same neural network without
parameter overlap, and therefore without exposing subnetworks to the
conditions giving rise to interference during multitask learning.

Figure 1: Illustration of the Multiple Subnetwork Hypothesis. Multiple
subnetworks able to represent a fully-connected neural network for a given
task are identified in a single dense, randomly-initialized feedforward neural
network. These multiple specialist subnetworks can then be arrayed as a sparse
tensor to isolate task representations, shielding them from catastrophic
interference and enabling multi-domain task performance.

###  2.2 Sparse Subnetwork Identification and Isolation

It is well known and documented that ANNs are often extremely
overparameterized, and this overparameterization results in redundancies and
inefficiencies which can be eliminated to conserve computational resources
without negatively affecting model accuracy [7]. The pruning of an ANN to
reduce or eliminate extraneous parameters yields a sparse representation of
the original fully-connected network, and the challenge is therefore to
identify which subnetwork of parameters within a dense ANN will best represent
the entire original ANN [8, 9, 10].

One means of reducing parameter count and computational requirements without
sacrificing the accuracy of the sparse network is through the identification
of sparse subnetworks within dense neural networks. In a randomly-initialized
feed-forward network, there may be a subnetwork which can sufficiently
represent the original network, called in other works a “winning lottery
ticket” [11]. This methodology serves as the primary inspiration of our
Multiple Subnetwork Hypothesis, as we explore in this study whether the
parameters excluded from a subnetwork for one task may be utilized in the
formation of subnetworks capable of learning and performing additional tasks.

###  2.3 Interference-Resistant Multitask Learning

Continuous learning in both biological and artificial neural systems is
constrained by the stability-plasticity dilemma. Learning must be sufficiently
plastic to allow the acquisition of new knowledge, but also sufficiently
stable to prevent the forgetting of previous knowledge [12]. Interference
stemming from lack of stability during sequential learning in ANNs, often
referred to as Catastrophic Forgetting, is a result of the tendency of NNs to
destroy internal state representations used in learning previously acquired
tasks when learning a new task [13, 14, 15]. This phenomenon is especially
pronounced in continuous learning paradigms with a low degree of inter-task
relatedness [16, 17, 18], posing a challenge to creating multitask models able
to perform prediction or classification across very different tasks, input
data types, or input shapes. Effective mitigation of catastrophic forgetting
is therefore a key prerequisite to achieving the flexibility and adaptability
of a multidomain multitask artificial intelligence.

Isolating parameters for a given task, thereby fixing parameter subsets for a
given task, is a means of achieving maximal stability in a continuous learning
context [19]. Parameter isolation methods localize the performance of learned
tasks over a physically [20, 21, 22] or logically [23, 24, 25] isolated region
of an ANN, reducing or eliminating the interaction of weights responsible for
a given task, therefore minimizing the opportunity for the acquisition of new
tasks to interfere with the ability of an ANN to continue to perform
previously learned tasks. The limitation of interactions between functional
groups of neurons has its basis in the biological necessity to not only
functionally integrate information in the brain, but to isolate the processing
of that information into distinct modules, as evidenced by the importance of
locally specialized regions in processing sensory information [26]. While the
isolation of subnetworks is the surest way to prevent interference arising
from learning new tasks, this method can scale poorly under memory and
computational constraints and therefore demands a flexible approach to
limiting network capacity [20] or minimizing the size of the network utilized
for each task. Our methodology aims to negate the high computational cost of
parameter isolation by applying our RSN2 methodology to represent multidomain
multitask models as sparse tensors, thereby minimizing their computational
footprint.

##  3 Methodology

In this section, we will discuss how we enable a single model to learn
multiple tasks across multiple domains and datatypes through both a task-
specific weight representational structure and a modified training procedure
which selects disjoint subsets of network weights for each individual task.

###  3.1 Multitask Representational Structure

The weights of fully connected neural network layers traditionally consist of
a kernel tensor, $k$, of shape $m\times n$, and a bias vector, $b$, of length
$n$, where $m$ denotes the number of columns in the input data and $n$ denotes
the number of neurons within the layer. To process input data $\bar{x}$, the
decision function is thus $\Phi(k\bar{x}+b)$, where $\Phi$ denotes the layer’s
activation function. Suppose then that there are inputs from two different
distributions, $\bar{x_{1}}$ and $\bar{x_{2}}$. Given the two-dimensional
structure of the kernel tensor in this traditional representation, there is no
way to alter the decision function given the distribution currently selected.

To address this problem, we add a new dimension, $t$, to the kernel tensor and
the bias vector within the layer to denote which distribution or task the
input data belongs to, leaving the kernel tensor with a shape of $t\times
m\times n$ and transforming the bias vector into a bias matrix of shape
$t\times n$. The decision function for this new layer is thus altered to that
in Equation 1.

$F(\bar{x},i)=\Phi(k_{i}\bar{x}+b_{i})$ (1)

We take a similar approach for convolutional layers. For a two-dimensional
convolutions with color channels, the kernel is of shape $s_{1}\times
s_{2}\times c\times f$, where $s_{1}$ and $s_{2}$ denote the width and height
of the convolutional filters, $c$ represents the number of channels in the
input, and $f$ corresponds to the number of filters. The bias vector in this
scenario has length $f$. In our multitask representation of the convolutional
layer, we once more add a new dimension to the front of both the kernel tensor
and the bias vector to denote the task.

Algorithm 1 RSN2 Training Procedure

Input: Neural Network $F(x)$, Training data for multiple tasks,
$X_{i},Y_{i},i\in\\{1,2,..,N\\}$

for $i\in\\{1,2,..,N\\}$ do

Choose $p_{i},p_{i}\in(0,100)$, ensuring $\sum_{i=1}^{N}p_{i}<=100$

Deactivate all weights which have been used for previous tasks

Activate all weights which have not been used for previous tasks

Select $(x_{i},y_{i})$ as a subset of training data and labels for the current
task

Calculate active weight gradients with respect to $(x_{i},y_{i})$

Set all inactive weight gradients to 0

Identify the quantity $q_{i}$ such that $p\%$ of weight gradients with respect
to $(x_{i},y_{i})$ is less than $q_{i}$

Deactivate all weights with gradients less than $q_{i}$ and do not allow them
to train

Train the network as usual, only altering identified active weights and
keeping all other weights inactive

Save all weight values for the specified task

end for

At inference time for task $i$, activate only stored weights for task $i$ and
perform inference as usual

###  3.2 Sparsification and Subnetwork Identification

While the multitask representational structure described above could
theoretically be used to enable a network to learn multiple tasks as-is, it
alone does not help us answer our hypothesis. Alone, this structure allows
multiple networks to be combined into a single model, as the number of
parameters within each layer is increased by a factor of the number of tasks.
To fully test our Multiple Subnetwork Hypothesis, we devised the RSN2 training
procedure, which ensures that only one weight along the task dimension is
active for all other fixed indices in multitask layers.

$I(x)=\left\\{\begin{array}[]{lr}1,&\text{if }x\neq 0\\\
0,&\text{otherwise}\end{array}\right\\}$ (2)

Mathematically, we utilize the indicator function from Equation 2 as an
activation function to indicate whether an individual weight is active or
inactive within a layer. Using this function, we can therefore demonstrate
whether disjoint subnetworks are active for all tasks by taking the sum of the
indicator function across the task dimension for all other fixed indices. If
for all fixed indices, $b$ and $c$, the condition present in Equation 3 holds,
then each weight is only active once across all tasks, meaning the task
dimension adds no additional active weights to the network layer.

$\sum_{a=1}^{t}I(k_{a,b,c})\leq 1;\text{ }b,c\text{ fixed}$ (3)

We utilize binary masking in our training procedure to impose these
conditions. These masks therefore ensure only select weights are active during
training through the bitwise multiplication of the weight value and the mask
value.

Figure 2: Diagram of the RSN2 training procedure and multitask
representational structure for a single fully-connected layer with two tasks.
On the left, we begin with the initialized layer weight matrix (center) and
create two binary mask matrices (top and bottom). These binary masks are then
bitwise-multiplied to the weight matrix, resulting in two disjoint subsets of
weights which are then separated across the task dimension in our
representational structure.

For weight selection and pruning, we utilize a gradient-based approach to
identify which weights are to be selected for each task. By performing pruning
in this manner, all pruning is done in a one-shot manner after initialization
rather than iteratively as the model trains, with the pruning rate a
hyperparameter selected at training time. The entirety of the RSN2 training
procedure for a single network layer can be found in Algorithm 1. For
performing inference, the masks used for training can be removed, thus
reducing the number of active weights to a maximum value of $t*p*W$, where $t$
represents the number of tasks, $p$ the maximum proportion of weights active
for any one task, and $W$ the number of weights which would be present in the
network had a traditional, one-task structure been used. Note that to satisfy
Equation 1, $t*p\leq 1$. As a result, the RSN2 procedure, when coupled with
the multitask representational structure defined above, identifies and
separates disjoint subnetworks within a traditional network to allow each to
individually perform separate tasks. Due to the lack of interaction between
these subnetworks in the resulting model, similarity between tasks is not
required, and no transfer learning will occur between tasks. A visual
representation of the RSN2 training procedure and multitask representational
structure can be found in Figure 2.

###  3.3 Experiments

We test our methodologies across eight experiments utilizing multiple datasets
to test the Multiple Subnetwork Hypothesis and RSN2 training procedure. In our
first experiment, we test the merits of the hypothesis by identifying whether
a single network can learn the same task multiple times. To test this, we
create a single convolutional multitask network and isolate five separate
subnetworks within it, each utilizing a disjoint 20% of the weights within the
network. Each of these subnetworks is then trained on the Fashion MNIST
dataset [27]. We then compare the results of each of these subnetworks, as
well as the results of the ensemble consisting of all five subnetworks,
against the performance of a dedicated network with identical architecture.

Our second experiment builds upon the first by identifying if a single network
can learn related tasks from a single dataset. The classes from the same
Fashion MNIST dataset are broken into two groups for this experiment; the
first group consists of the five individual classes which were most easily
identified in the previous experiment.111We identified those which were
learned most easily by the resulting F1 score on test data by the control
model. The second group consists of the remaining five classes. The results of
this experiment are compared to the performance of a dedicated network with
identical architecture.

For our third experiment, we train a fully-connected network to perform both
the MNIST Digit Recognition [28] and Fashion MNIST tasks and compare the
performance of a single network to two dedicated networks. This experiment is
the first we perform to test whether a single network can truly learn
different tasks from multiple different domains and distributions, and it is
the first experiment designed to show whether a fully connected network can
perform multiple tasks.

In our fourth experiment, we train a convolutional network to perform both
MNIST tasks from the previous experiment. This network is identical in
architecture to all convolutional architectures trained previously, and once
again this network is compared to two dedicated networks of the same
architecture.

Our fifth, sixth, and seventh experiments continue to explore our hypothesis
to a greater degree, as we train individual networks to perform multiple tasks
across a wider array of data modalities. In the fifth experiment, we train a
fully connected network to perform three tasks: both MNIST tasks as well as a
regression task on house prices in the Boston area [29]. Experiment 6 trains a
three-task convolutional network across both MNIST tasks as well as the
CIFAR-10 dataset [30]. Our seventh and final experiment tests whether a single
network can learn multiple tasks across an even wider array of input data
types. For this experiment, we train a single fully-connected network to
perform four tasks: both MNIST tasks, the Boston housing regression task, and
an IMDB reviews sentiment classification task [31].

Our final experiment seeks to explore the feasibility of our methodologies in
the context of more complicated tasks. For this experiment, we trained a
single convolutional model to perform four tasks. The first three of these are
the age, gender, and race prediction tasks from the UTKFace dataset [32]. The
fourth task is the CIFAR10 task [30]. This experimental network is tested
against four dedicated networks of identical architectures.

##  4 Results

In this section, we present the results for each experiment conducted.
Firstly, we present the results for each control model for all tasks, as well
as describe the model architectures and training schedules, as well as any
data preprocessing steps which were taken. For control results, each control
model was created only once and will therefore be presented with greater
detail upon first presentation, but its results will be included in every
pertinent experiment. For more detailed indicators of performance for all
models, as well as mappings between integer labels and their respective
classes, we refer the reader to the appendices.

###  4.1 Experiment 1: MNIST Fashion Ensemble, Convolutional Architecture

Our first experiment involved creating a single convolutional network with
five disjoint subnetworks each trained on the MNIST Fashion dataset. We
utilized a convolutional architecture with two convolutional blocks, with the
first block containing two convolutional layers with thirty-two $3\times 3$
filters and ReLU activation [33] followed by maximum pooling over a $2\times
2$ filter area. The second convolutional block was identical to the first,
except each of the convolutional layers utilized sixty-four filters. The
output to the convolutional blocks were then flattened and fed into two fully-
connected layers with $256$ neurons each and ReLU activation. A final fully-
connected layer with ten neurons and softmax activation was used to provide
the final outputs. The only preprocessing which was done on input data was a
division of pixel values by $255$ to ensure all input values were within the
interval from $0$ to $1$. The model was trained using a batch size of $512$,
and early stopping was initialized after three epochs with no improvement of
greater than $0.01$ in loss on validation data. The same architecture,
preprocessing steps, and training procedure was completed in the experimental
case as well, with each subnetwork pruned so that 20% of the network weights
were allocated to each task. A summary of results can be found in Table 1.

Table 1: Experiment 1 Results.
Network | Accuracy | Pruning Rate
---|---|---
Experimental
Subnetwork 1 | 90% | 80%
Subnetwork 2 | 90% | 80%
Subnetwork 3 | 90% | 80%
Subnetwork 4 | 90% | 80%
Subnetwork 5 | 92% | 80%
Ensemble | 92% | 80% Each
Control
MNIST Fashion | 92% | N/A

###  4.2 Experiment 2: MNIST Fashion Two-Task Model, Convolutional
Architecture

For our second experiment, we utilized an identical architecture, data
preprocessing, and training procedure to the previous experiment but trained
two individual subnetworks within that architecture. We trained the first
subnetwork on the five classes which were most easily identified by the
control model, measured by F1 score, and we trained the second subnetwork on
the five classes which were the most difficult for the control model to
identify, also measured by F1 score. The results of the experiment can be
found in Table 2.

Table 2: Experiment 2 results.
Network | Accuracy | Pruning Rate
---|---|---
Experimental
Easy Task | 98% | 80%
Hard Task | 84% | 80%
Control
MNIST Fashion | 92% | N/A

###  4.3 Experiment 3: MNIST Digits and MNIST Fashion Model, Fully-Connected
Architecture

Our third experiment is the first experiment to truly test whether a single
model can learn disparate tasks across a variety of data modalities. To do
this, we trained a single fully-connected network to perform both the MNIST
Digit Recognition and the MNIST Fashion Recognition tasks. For this
experiment, we preprocessed the data for both tasks once more by dividing each
pixel value by $255$. Each image was then flattened into a two-dimensional
vector. The images would then be passed through six fully connected layers
each containing $1000$ artificial neurons each and activated using the ReLU
activation function. The final layer contained ten neurons which were
activated with the softmax activation function. The model was trained with the
same early stopping conditions as all previous models, and a batch size of
$512$ was used once more. The experimental model was trained using the same
procedure and with the same architecture, with each task pruned to utilize 10%
of the network weights. The results of this experiment can be found in Table
3.

Table 3: Experiment 3 results.
Network | Accuracy | Pruning Rate
---|---|---
Experimental
MNIST Digit | 97% | 90%
MNIST Fashion | 86% | 90%
Control
MNIST Digit | 97% | N/A
MNIST Fashion | 88% | N/A

###  4.4 Experiment 4: MNIST Digit and MNIST Fashion Model, Convolutional
Architecture

In experiment four, we repeated the modeling goals of the previous experiment,
but this experiment utilized the convolutional architecture from the previous
experiments, once again with an 80% pruning rate for each task. Results for
this experiment can be found in Table 4.

Table 4: Experiment 4 results.
Network | Accuracy | Pruning Rate
---|---|---
Experimental
MNIST Digit | 99% | 80%
MNIST Fashion | 90% | 80%
Control
MNIST Digit | 99% | N/A
MNIST Fashion | 92% | N/A

###  4.5 Experiment 5: MNIST Digit, MNIST Fashion, and Boston Housing Model,
Fully-Connected Architecture

The fifth experiment provides the greatest test for our Multiple Subnetwork
Hypothesis up to this point, as we train a model with a wider variety of
datatypes and tasks, and we also train some tasks in sequence rather than all
at once. For this experiment, we trained a single fully-connected network
first on both MNIST tasks, then trained the same network on the Boston Housing
dataset. The MNIST portions of the network were identical in architecture to
the ones used in previous experiments, but due to difference in input shapes
between the MNIST images and the Boston Housing dataset222The MNIST images are
both 28 pixels wide and 28 pixels tall, while the Boston Housing dataset
contains only 13 input features. an additional fully-connected layer is added
after the Boston Housing input layer to create shape alignment. This layer is
activated with ReLU activation, and it is also pruned in the experimental
model. In the test case, all subnetworks are pruned to only use 10% of
available weights.

Data preprocessing for MNIST tasks is identical to previous MNIST
preprocessing for fully-connected networks. For the Boston Housing dataset,
input features were scaled such that the minimum feature value in the training
data corresponded to a value of $0$ and the maximum value of the feature in
the training data corresponded to a value of $1$. The predicted values were
also scaled using the same procedures. During training, as stated before, both
MNIST tasks are learned simultaneously, utilizing the same early stopping
criteria and batch sizes as the previous cases. For the Boston Housing task,
the same early stopping procedure is used, but the batch size is decreased to
$32$ in both the control and test cases due to the limited amount of data
available. The results for this experiment can be found in Table 5.

Table 5: Experiment 5 results. In the performance column, percentages
correspond to accuracy percentage on test data, while decimals represent mean
squared error on test data.
Network | Performance | Pruning Rate
---|---|---
Experimental
MNIST Digit | 97% | 90%
MNIST Fashion | 87% | 90%
Boston Housing | 0.016 | 90%
Control
MNIST Digit | 97% | N/A
MNIST Fashion | 88% | N/A
Boston Housing | 0.017 | N/A

Furthermore, to identify whether any forgetting occurred after sequential
training, we recorded loss values on test data for current and previously-
trained tasks after each subsequent task was trained. These results are
visualized in Figure 3, where it can be seen that no performance degradation
occurred on the first two tasks after training the third task.

Figure 3: Task losses obtained on test data for multitask model after training
each set of tasks.

###  4.6 Experiment 6: CIFAR10, MNIST Digit, and MNIST Fashion Model,
Convolutional Architecture

Experiment 6 was designed to expand upon the findings of previous experiments
by testing whether a single convolutional network could converge across
multiple classification tasks across a variety of input images. In this
experiment, we test whether a single convolutional network can converge on
first the CIFAR-10 image classification task and then train the same network
on both MNIST tasks we have utilized to this point. To ensure that all images
are of the same shape, we downsize the CIFAR-10 images so that they have the
same height and width as the MNIST data. Additionally, because of the color
channels present in the CIFAR images, a separate first convolutional layer
must be created to process the CIFAR-10 inputs in the multitask model. This
dedicated layer is still pruned to the same degree as all other processing
layers within the model. During training, the same early stopping procedures
that have been used previously were employed, and batch sizes of $512$ were
used for all tasks. The results for this experiment can be found in Table 6,
and task losses after training can be found in Figure 4. The test model was
pruned to 90% in this experiment.

Table 6: Experiment 6 results.
Network | Accuracy | Pruning Rate
---|---|---
Experimental
CIFAR-10 | 60% | 80%
MNIST Digit | 98% | 90%
MNIST Fashion | 89% | 90%
Control
CIFAR-10 | 47% | N/A
MNIST Digit | 99% | N/A
MNIST Fashion | 92% | N/A

Figure 4: Task losses obtained on test data for multitask model after training
each set of tasks.

###  4.7 Experiment 7: MNIST Digit, MNIST Fashion, Boston Housing, and IMDB
Reviews Model, Fully-Connected Architecture

Our seventh experiment provides perhaps the most wide-ranging set of tasks for
a single network, and as a result it is perhaps the best test of our Multiple
Subnetwork Hypothesis. In this experiment, we utilized a single fully-
connected neural network and trained the network on four separate tasks; the
first two of these tasks, both the MNIST Digit and MNIST Fashion tasks, were
trained simultaneously. The third task, the Boston Housing regression task,
was then trained. Finally, the IMDB reviews sentiment classification task was
trained. The same architecture as previously used in all fully-connected
experiments was used in this experiment. For the MNIST and Boston Housing
tasks, the same preprocessing and reshaping efforts took place as previous
experiments with fully-connected architectures. For the IMDB task, only the
most common $10000$ words were utilized and sequences were padded (or
truncated) to $128$ words, with both padding and truncating occurring at the
end of the review. Furthermore, each of the input tokens was passed through an
embedding layer, which embedded the token in a two-dimensional vector space.
The embedded tensors were then flattened into a two-dimensional vector and
passed through the architecture used throughout this experiment333This
architecture involves six fully-connected layers of $1000$ neurons and ReLU
activation.. For the multitask model in this experiment, the first fully-
connected layer which processed inputs from the IMDB task was dedicated solely
to the IMDB task due to the differences in input shapes between all tasks.
During training, the same early stopping criteria were used for all training
iterations, batches sizes of $512$ were used for the MNIST and IMDB tasks,
while a batch size of $32$ was used for the Boston Housing task. The results
for this experiment can be found in Table 7, and task losses can be found in
Figure 5. The model was pruned such that each task utilizes 10% of the
available weights in fully-connected layers; the embedding layer was not
pruned.

Table 7: Experiment 7 results. In the performance column, percentages
correspond to accuracy percentage on test data, while decimals represent mean
squared error on test data.
Network | Performance | Pruning Rate
---|---|---
Experimental
MINST Digit | 97% | 90%
MNIST Fashion | 87% | 90%
Boston Housing | 0.011 | 90%
IMDB | 82% | 90%
Control
MNIST Digit | 97% | N/A
MNIST Fashion | 88% | N/A
Boston Housing | 0.017 | N/A
IMDB | 80% | N/A

Figure 5: Task losses obtained on test data for multitask model after training
each set of tasks.

###  4.8 Experiment 8: UTKFace and CIFAR10 Four-Task Model, Convolutional
Architecture

In our final experiment, we tested the RSN2 training procedure on a larger
network with more complex tasks involved. We created a single network with a
convolutional architecture on four individual tasks, three tasks from the
UTKFace dataset [32] and one multiclass classification task from CIFAR10 [30].
For the age task in the UTKFace data, we grouped values into classes by
decade, excluding a final class containing all individuals with age greater
than 90.

Architecturally, the model trained in this experiment contains three
convolutional blocks containing of $16$, $32$, and $64$ $3\times 3$ filters,
respectively. Each block contains two convolutional layers with padding to
preserve shape, followed by a max pooling layer. These blocks then feed into a
fully connected architecture of three layers, each with $128$ neurons each and
activated using ReLU activation [33], followed by an output layer of the
required shape to perform the specified task.

For the multitask model, we utilized a shared convolutional embedding between
all UTKFace tasks. This resulted in a two-task structure for convolutional
layers within the model, with one task channel processing the UTKFace images
and the other task channel processing the CIFAR10 images. For the fully-
connected layers, each of the UTKFace tasks was isolated and a four-task
architecture was used. Along both the convolutional and the fully-connected
architectures, each task was pruned to utilize only $25\%$ of the model’s
available weights per task, meaning $50\%$ of the weights in the convolutional
layers were utilized444This is due to the two-task structure of the
convolutional layers, as mentioned above. and $100\%$ of the weights across
the shared fully-connected architecture was utilized.555This is due to the
four-task structure of the fully-connected layers, as mentioned above. All
four tasks were trained simultaneously, with early stopping criteria utilized
with patience of five epochs. A summary of the results of this experiment can
be found in 8. In addition to these performance results, which interestingly
showed that the dedicated control models were unable to converge but that the
multitask model was able to converge, it was also found that the multitask
model only utilized $130MB$ of disk space when saved, while the combined
single-task models utilized $203MB$ of disk space, over $56\%$ more than the
multitask model.

Table 8: Experiment 8 results.
Network | Accuracy | Pruning Rate
---|---|---
Experimental
UTKFace Age | 55% | 75%
UTKFace Gender | 89% | 75%
UTKFace Race | 77% | 75%
CIFAR10 | 50% | 75%
Control
UTKFace Age | 32% | N/A
UTKFace Gender | 48% | N/A
UTKFace Race | 14% | N/A
CIFAR10 | 10% | N/A

##  5 Discussion

This study was designed to provide insight into answering our Multiple
Subnetwork Hypothesis, which states that a dense, randomly-initialized
feedforward neural network contains within its architecture multiple disjoint
subnetworks which can be isolated to each separately learn and make accurate
predictions on multiple tasks, regardless of the degree of similarity between
tasks or input types. To test this hypothesis, we developed the RSN2 procedure
as well as a customized neural network representational structure designed to
isolate individual subnetworks within the overall architecture of the global
model on a layer-by-layer basis. We then utilized these techniques in a series
of seven experiments across a range of datasets utilizing both fully-connected
and convolutional architectures.

Across each of these experiments, we consistently saw that a single network
was able to converge on multiple tasks by utilizing only a small fraction of
network weights for each task. Furthermore, multitask models were able to
perform each task with near identical performance to dedicated models with all
weights active, even exceeding dedicated model performance in a few cases.
Additionally, we saw that there needed to be no similarity between tasks, as
in our most extreme case we were able to teach a single model to perform two
separate image classification tasks, a tabular regression task, and a
sentiment classification task on natural language.

The beneficial implications of this approach arise from the simplicity of
interacting with a single model rather than a collection of models as in the
traditional machine learning deployment paradigm. Take for example the human
visual system: a unified stream of visual information is collected and passed
to the optic lobe, where it is then passed to the relevant regions of the
brain for further downstream analysis. This paradigm is reflected in the
unified data pipeline of a multitask model trained using the RSN2 procedure,
where invoking the model involves invoking the relevant task or tasks
automatically. Therefore, a collection of models capable of performing
multiple tasks can be deployed as a single entity within an organization’s IT
infrastructure, thereby improving efficiency and lowering costs. Take, for
instance, a simple example of an organization which wants to deploy five
separate models trained on five separate tasks. A typical deployment scenario
would involve this organization deploying one model each to five separate
computational nodes. Using a model trained with the RSN2 procedure to perform
all five tasks, this organization could therefore reduce costs by deploying
this single multitask model to a single node, saving 80% of the organization’s
original costs. Furthermore, to improve system robustness by introducing
redundancy, the organization could deploy this single model to a set of three
nodes and still reduce costs by 40%.

Additionally, in our final experiment, even though we do not utilize sparse
tensors to save the model, the pruned multitask model utilizes less disk space
when saved compared to the total amount of the identical single-task models.
This indicates that utilizing RSN2 and our multitask representational
structure already leads to more efficient representation of the models
themselves, thus potentially further reducing costs by decreasing the hardware
requirements needed to save and use models for inference.

###  5.1 Limitations

There are a number of limitations to this study which limit the conclusions we
can draw from the results. Firstly, our experiments focus primarily on
computer vision related tasks in this work, with all experiments containing
some image classification task. While we do also include experiments which
test our methods on both tabular input and natural language inputs, further
studies should still be conducted to more rigorously test our hypotheses and
methods on various input and data types.

Another limitation to this study is the limited size of both the datasets and
models which were studied. Many state of the art models have been trained on
millions of data samples, and the models themselves consist of many tens of
millions of parameters. Our study, on the other hand, considered labeled data
with up to tens of thousands of samples and models which contained hundreds of
thousands to millions of parameters, orders of magnitude fewer (in both cases)
than the state of the art in most cases. It is therefore unknown whether these
methods would need to be modified666In particular, it is unknown whether our
one-shot sparsification technique would generalize to much larger models. to
accommodate larger modeling paradigms.

###  5.2 Conclusions

Despite the limitations to this study, there are still a number of conclusions
we can draw from its results. Each multitask model was clearly able to
converge on all its assigned tasks, achieving performance similar to, and in
some cases exceeding, the control cases each time. These results therefore
confirm our Multiple Subnetwork Hypothesis, showing that a single neural
architecture is capable of utilizing small portions of itself to learn
individual tasks. Additionally, we showed that our methodology for identifying
and logically separating network weights is robust to catastrophic forgetting,
and we showed that we were therefore able to train multitask models across
various domains and tasks both in parallel or in sequence without any adverse
effects on previously-learned tasks.

Because the resulting model would have the ability to robustly perform
multiple tasks, we can also conclude that a single model trained with these
techniques could substitute multiple models in a deployed scenario. This
substitution would therefore reduce resource requirements for organizations
deploying and utilizing models, thus saving costs and simplifying the
deployment architectures these organizations use. These models would also be
able to acquire more downstream tasks after previously being trained on
initial tasks, meaning adding new predictive analytics would be a much simpler
undertaking as well, as it would only involve updating an already-deployed
model and not require creating a completely new deployment for every new
analytic.

###  5.3 Future Work

Following studies should be designed to address some of the limitations of
this work as well as to improve the methods used within. Future studies should
provide a more thorough analysis on how these methods perform on a larger
variety of data modalities, including data types, task types, data
availability, and model sizes. Secondly, we intend to identify whether a more
precise pruning method can be applied as training occurs instead of in a one-
shot manner as was done in this study. If pruning can be applied in a more
intelligent manner, we believe the resulting model would potentially be able
to more consistently surpass unpruned model performance while also achieving
greater sparsification rates than possible with one-shot techniques.

Additionally, this study does not perform any analysis to identify the maximum
amount of tasks which can be learned by a single network. Future work should
therefore address finding the upper bounds of the number of tasks an
individual model can support. Finally, the methods in this work do not allow
for any transfer learning across tasks; instead, it is assumed that new tasks
should utilize a completely disjoint subset of weights relative to all
previously-learned tasks. Additional studies should be conducted to identify
whether transfer learning can be leveraged during the acquisition of similar
tasks to enable further reduction in the number of active weights and the
quicker acquisition of new tasks.

## References

  * [1] Kumar Chellapilla, Kevin Larson, Patrice Simard, and Mary Czerwinski.  Computers beat humans at single character recognition in reading based human interaction proofs (hips).  01 2005.
  * [2] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489, 2016.
  * [3] Murray Campbell, A.Joseph Hoane, and Feng hsiung Hsu.  Deep blue.  Artificial Intelligence, 134(1):57–83, 2002.
  * [4] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  Nature, 575(7782):350–354, 2019.
  * [5] Claudine Badue, Ranik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius Brito Cardoso, Avelino Forechi, Luan Ferreira Reis Jesus, Rodrigo Ferreira Berriel, Thiago Meireles Paixão, Filipe Wall Mutz, Thiago Oliveira-Santos, and Alberto Ferreira de Souza.  Self-driving cars: A survey.  CoRR, abs/1901.04407, 2019.
  * [6] Zhen Qin, Yicheng Cheng, Zhe Zhao, Zhe Chen, Donald Metzler, and Jingzheng Qin.  Multitask Mixture of Sequential Experts for User Activity Streams, page 3083–3091.  Association for Computing Machinery, New York, NY, USA, 2020.
  * [7] Yann LeCun, John Denker, and Sara Solla.  Optimal brain damage.  In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1990.
  * [8] Kenji Suzuki, Isao Horiba, and Noboru Sugie.  A simple neural network pruning algorithm with application to filter synthesis.  Neural processing letters, 13(1):43–53, 2001.
  * [9] Song Han, Huizi Mao, and William J Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  arXiv preprint arXiv:1510.00149, 2015.
  * [10] Mieszko Lis, Maximilian Golub, and Guy Lemieux.  Full deep neural network training on a pruned weight budget.  Proceedings of Machine Learning and Systems, 1:252–263, 2019.
  * [11] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019.
  * [12] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin.  The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects.  Frontiers in psychology, 4:504, 2013.
  * [13] Robert French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3:128–135, 05 1999.
  * [14] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  arXiv preprint arXiv:1312.6211, 2013.
  * [15] Benedikt Pfülb and Alexander Gepperth.  A comprehensive, application-oriented study of catastrophic forgetting in dnns.  arXiv preprint arXiv:1905.08101, 2019.
  * [16] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366–3375, 2017.
  * [17] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1930–1939, 2018.
  * [18] Marc Masana, Tinne Tuytelaars, and Joost van de Weijer.  Ternary feature masks: zero-forgetting for task-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3570–3579, 2021.
  * [19] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [20] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  arXiv preprint arXiv:1708.01547, 2017.
  * [21] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [22] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * [23] Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov.  Rotate your networks: Better weight consolidation and less catastrophic forgetting.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262–2268. IEEE, 2018.
  * [24] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In International Conference on Machine Learning, pages 4548–4557. PMLR, 2018.
  * [25] Arun Mallya, Dillon Davis, and Svetlana Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 67–82, 2018.
  * [26] Elizabeth Zavitz and Nicholas SC Price.  Understanding sensory information processing through simultaneous multi-area population recordings.  Frontiers in neural circuits, 12:115, 2019.
  * [27] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.  cite arxiv:1708.07747Comment: Dataset is freely available at https://github.com/zalandoresearch/fashion-mnist Benchmark is available at http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/.
  * [28] Yann LeCun and Corinna Cortes.  MNIST handwritten digit database.  http://yann.lecun.com/exdb/mnist/, 2010.
  * [29] David Harrison and Daniel L Rubinfeld.  Hedonic housing prices and the demand for clean air.  Journal of Environmental Economics and Management, 5(1):81–102, 1978.
  * [30] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).
  * [31] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
  * [32] Zhifei Zhang, Yang Song, and Hairong Qi.  Age progression/regression by conditional adversarial autoencoder.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
  * [33] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375, 2018.

##  Appendix A Class labels for classification tasks.

Table 9: Class labels for computer vision tasks.
Dataset | Integer | Label
---|---|---
MNIST Digit |  |
| 1 | Digit “0”
| 2 | Digit “1”
| 3 | Digit “2”
| 4 | Digit “3”
| 5 | Digit “4”
| 6 | Digit “5”
| 7 | Digit “6”
| 8 | Digit “7”
| 9 | Digit “8”
| 10 | Digit “9”
MNIST Fashion |  |
| 1 | T-shirt/top
| 2 | Trouser
| 3 | Pullover
| 4 | Dress
| 5 | Coat
| 6 | Sandal
| 7 | Shirt
| 8 | Sneaker
| 9 | Bag
| 10 | Ankle Boot
CIFAR-10 |  |
| 1 | Airplane
| 2 | Automobile
| 3 | Bird
| 4 | Cat
| 5 | Deer
| 6 | Dog
| 7 | Frog
| 8 | Horse
| 9 | Ship
| 10 | Truck
UTKFace Age |  |
| 1 | $0-10$
| 2 | $11-20$
| 3 | $21-30$
| 4 | $31-40$
| 5 | $41-50$
| 6 | $51-60$
| 7 | $61-70$
| 8 | $71-80$
| 9 | $81-90$
| 10 | $91+$
UTKFace Gender |  |
| 1 | Male
| 2 | Female
UTKFace Ethnicity |  |
| 1 | White
| 2 | Black
| 3 | Asian
| 4 | Indian
| 5 |  “Other” 777Defined by dataset creators and includes ethnicities such as Hispanic, Latino, and Middle Eastern.
Table 10: Class labels for IMDB task.
Integer | Label
---|---
1 | Negative Review
2 | Positive Review

##  Appendix B Additional model performance metrics.

When conducting our study, we collected additional performance measures for
all models which were not included in the main manuscript above. These metrics
are presented in the following tables.

Table 11: Per-class F1 and overall accuracy scores for control computer vision
models. “FC” indicates fully-connected architecture. “Conv” indicates
convolutional architecture.
Network | Class Number | Accuracy
---|---|---
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
MNIST Digit FC | 0.98 | 0.99 | 0.97 | 0.97 | 0.98 | 0.97 | 0.98 | 0.97 | 0.96 | 0.96 | 0.97
MNIST Digit Conv | 0.99 | 0.99 | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.98 | 0.99
MNIST Fashion FC | 0.84 | 0.98 | 0.79 | 0.88 | 0.79 | 0.97 | 0.69 | 0.95 | 0.97 | 0.96 | 0.88
MNIST Fashion Conv | 0.85 | 0.99 | 0.87 | 0.92 | 0.86 | 0.98 | 0.76 | 0.97 | 0.99 | 0.97 | 0.92
CIFAR-10 Conv (Exp 6) | 0.55 | 0.59 | 0.35 | 0.26 | 0.38 | 0.39 | 0.51 | 0.53 | 0.55 | 0.52 | 0.47
UTKFace Age Conv | 0.00 | 0.00 | 0.49 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.32
UTKFace Gender Conv | 0.00 | 0.64 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.48
UTKFace Ethnicity Conv | 0.00 | 0.00 | 0.25 | 0.00 | 0.00 | N/A | N/A | N/A | N/A | N/A | 0.14
CIFAR-10 Conv (Exp 8) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.18 | 0.00 | 0.00 | 0.00 | 0.00 | 0.10
Table 12: Per-class F1 and overall accuracy scores for experimental computer
vision models.
Network | Class Number | Accuracy
---|---|---
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
Experiment 1 |  |  |  |  |  |  |  |  |  |  |
Subnetwork 1 | 0.86 | 0.98 | 0.85 | 0.90 | 0.84 | 0.98 | 0.72 | 0.96 | 0.97 | 0.97 | 0.90
Subnetwork 2 | 0.85 | 0.97 | 0.84 | 0.89 | 0.83 | 0.98 | 0.72 | 0.96 | 0.98 | 0.96 | 0.90
Subnetwork 3 | 0.84 | 0.98 | 0.82 | 0.88 | 0.83 | 0.98 | 0.72 | 0.96 | 0.97 | 0.97 | 0.90
Subnetwork 4 | 0.85 | 0.98 | 0.84 | 0.90 | 0.84 | 0.98 | 0.72 | 0.96 | 0.98 | 0.97 | 0.90
Subnetwork 5 | 0.87 | 0.99 | 0.88 | 0.91 | 0.88 | 0.98 | 0.78 | 0.96 | 0.98 | 0.97 | 0.92
Ensemble | 0.87 | 0.98 | 0.87 | 0.91 | 0.86 | 0.98 | 0.76 | 0.97 | 0.98 | 0.97 | 0.92
Experiment 2 |  |  |  |  |  |  |  |  |  |  |
Easier Task Subnetwork | N/A | 1.00 | N/A | N/A | N/A | 0.97 | N/A | 0.96 | 0.99 | 0.96 | 0.98
Harder Task Subnetwork | 0.85 | N/A | 0.87 | 0.92 | 0.85 | N/A | 0.73 | N/A | N/A | N/A | 0.84
Experiment 3 |  |  |  |  |  |  |  |  |  |  |
MNIST Digit | 0.99 | 0.99 | 0.98 | 0.97 | 0.97 | 0.97 | 0.98 | 0.98 | 0.97 | 0.96 | 0.98
MNIST Fashion | 0.82 | 0.97 | 0.77 | 0.87 | 0.79 | 0.96 | 0.62 | 0.94 | 0.97 | 0.95 | 0.87
Experiment 4 |  |  |  |  |  |  |  |  |  |  |
MNIST Digit | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.98 | 0.98 | 0.99
MNIST Fashion | 0.85 | 0.98 | 0.85 | 0.90 | 0.85 | 0.98 | 0.71 | 0.96 | 0.97 | 0.96 | 0.90
Experiment 5 |  |  |  |  |  |  |  |  |  |  |
MNIST Digit | 0.98 | 0.99 | 0.97 | 0.97 | 0.97 | 0.97 | 0.97 | 0.98 | 0.97 | 0.96 | 0.97
MNIST Fashion | 0.82 | 0.98 | 0.78 | 0.88 | 0.78 | 0.95 | 0.68 | 0.94 | 0.96 | 0.95 | 0.87
Experiment 6 |  |  |  |  |  |  |  |  |  |  |
MNIST Digit | 0.98 | 0.99 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.97 | 0.98
MNIST Fashion | 0.84 | 0.98 | 0.83 | 0.89 | 0.83 | 0.97 | 0.70 | 0.95 | 0.97 | 0.96 | 0.89
CIFAR-10 | 0.65 | 0.72 | 0.46 | 0.43 | 0.49 | 0.49 | 0.69 | 0.70 | 0.70 | 0.69 | 0.60
Experiment 7 |  |  |  |  |  |  |  |  |  |  |
MNIST Digit | 0.98 | 0.99 | 0.98 | 0.96 | 0.97 | 0.96 | 0.97 | 0.96 | 0.96 | 0.96 | 0.97
MNIST Fashion | 0.82 | 0.97 | 0.78 | 0.87 | 0.78 | 0.95 | 0.64 | 0.93 | 0.95 | 0.95 | 0.87
Experiment 8 |  |  |  |  |  |  |  |  |  |  |
UTKFace Age | 0.89 | 0.45 | 0.68 | 0.37 | 0.14 | 0.44 | 0.34 | 0.29 | 0.45 | 0.08 | 0.55
UTKFace Gender | 0.90 | 0.88 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.89
UTKFace Ethnicity | 0.84 | 0.83 | 0.79 | 0.68 | 0.27 | N/A | N/A | N/A | N/A | 0.77 |
CIFAR-10 | 0.58 | 0.63 | 0.27 | 0.35 | 0.44 | 0.47 | 0.53 | 0.54 | 0.61 | 0.52 | 0.50
Table 13: Per-class F1 and overall accuracy scores for experimental and
control IMDB sentiment analysis models.
Network | Class Number | Accuracy
---|---|---
| 1 | 2 |
Experiment 7 IMDB | 0.83 | 0.81 | 0.82
Control IMDB | 0.80 | 0.80 | 0.80
Table 14: Boston Housing model losses.
Network | Loss
---|---
Experiment 5 | 0.016
Experiment 7 | 0.011
Control | 0.017