Novel Multicolumn Kernel Extreme Learning Machine for Food Detection via
Optimal Features from CNN

Ghalib Ahmed Tahira (ghalib@siswa.um.edu.my), Chu Kiong Looa<EMAIL_ADDRESS>
a Department of Artificial Intelligence, Faculty Of Computer Science And
Information Technology, University of Malaya, 50603 Kuala Lumpur, Malaysia

Corresponding Author:
Chu Kiong Loo
Department of Artificial Intelligence, Faculty Of Computer Science And
Information Technology,
University of Malaya, 50603 Kuala Lumpur, Malaysia
Email<EMAIL_ADDRESS>
###### Abstract

Automatic food detection is an emerging topic of interest due to its wide
array of applications ranging from detecting food images on social media
platforms to filtering non-food photos from the users in dietary assessment
apps. Recently, during the COVID-19 pandemic, it has facilitated enforcing an
eating ban by automatically detecting eating activities from cameras in public
places. Therefore, to tackle the challenge of recognizing food images with
high accuracy, we proposed the idea of a hybrid framework for extracting and
selecting optimal features from an efficient neural network. There on, a
nonlinear classifier is employed to discriminate between linearly inseparable
feature vectors with great precision. In line with this idea, our method
extracts features from MobileNetV3, selects an optimal subset of attributes by
using Shapley Additive exPlanations (SHAP) values, and exploits kernel extreme
learning machine (KELM) due to its nonlinear decision boundary and good
generalization ability. However, KELM suffers from the ’curse of
dimensionality problem’ for large datasets due to the complex computation of
kernel matrix with large numbers of hidden nodes. We solved this problem by
proposing a novel multicolumn kernel extreme learning machine (MCKELM) which
exploited the k-d tree algorithm to divide data into N subsets and trains
separate KELM on each subset of data. Then, the method incorporates KELM
classifiers into parallel structures and selects the top k nearest subsets
during testing by using the k-d tree search for classifying input instead of
the whole network. For evaluating a proposed framework large food/non-food
dataset is prepared using nine publically available datasets. Experimental
results showed the superiority of our method on an integrated set of measures
while solving the problem of ’curse of dimensionality in KELM for large
datasets.

###### keywords:

Multicolumn, Kernel Extreme Learning Machine, MobileNet, Food Detection,
Explainable AI, SHAP

††journal: Expert Systems with Applications

##  1 Introduction

Automatic detection of food images applications includes visual-based dietary
assessment and detecting eating activities from wearable camera photos. The
visual-based dietary assessment method reduces the burden of manually
collecting the food data by helping users in refreshing their memory using the
food pictures of the previous dietary intake. Filtering of non-food images
from users is an essential step in these mHealth apps to ensure relevant
images are analyzed.
Similarly, food detection from photos of a wearable camera of the surrounding
environment tackles malnutrition in low-income countries and helps to
facilitate the process of accessing food intake. Besides this, filtering the
non-food images from food images in the user profile on social media fasten
the process of analyzing dietary preferences, with minimum human intervention.
Visual-based methods for food detection provide alerts to users that are
bringing food items to restricted areas where all or certain items are not
allowed. One of the related use-case is to ensure the enforcement of a ’no
food policy’ at the workplaces due to safety concerns. The recent pandemic has
also given birth to several use cases of food detection, especially in
improving the effectiveness of enforcing the ban on eating in public places or
transport to combat the spread of the COVID-19 virus. Despite the significant
importance of this area, there involved research challenges in preparing large
datasets and novel machine learning methods, which should be solved in a
highly accurate framework of food detection. Figure 1 illustrates the basic
steps in a pipeline of food recognition.

Figure 1: Shows general pipeline of food detection

In line with this idea, this study employed an efficient convolutional neural
network for real-time feature extraction and then selected optimal features
from high dimensional feature space using SHAP activation values from the
gradient explainer. Finally, this work contributed mainly to the
classification phase of the pipeline by proposing a novel algorithm to address
challenges with existing methods, as recent approaches are using convolutional
neural networks, both for feature representations of the images and
classification. However, linear classifiers in a convolutional neural network
(CNN) do not suit linearly inseparable features. Consequently, requiring
machine learning algorithms that discriminate linearly inseparable feature
vectors with a high degree of accuracy, while achieving the objective of real-
time classification in a distributed cloud environment.

Kernel extreme learning machine is one of the advanced methods in extreme
learning machines for classifying linearly inseparable features, and it has
good generalization ability. However, KELM relies on complex inner matrix
computations to cater to large data volumes, making these operations costlier
in terms of computational resources and processing time. Similarly, unlike
KELM, which has a strongly coupled network structure, preferred machine
learning algorithms take advantage of the cloud’s parallel and distributed
environments. There are certain efforts for other variants of neural networks,
showing improved results and performance in a distributed environment like a
parallel-structured ANN. Cireşan et al. [2012] Shao et al. [2014] . Although
currently implemented strategies in these methods are naive, this paves the
pathway for the researchers to develop advanced methods that take advantage of
the distributed cloud environment while having competitive performance.
One of the possible strategies to solve this challenge is decoupling strongly
coupled neural networks into loosely coupled networks. We achieve this
objective by using a space partitioning data structure ’k-d tree’ that divides
the data into equal distributions. This approach makes it very efficient for
nearest neighbor search, and it ensures the equal distribution of data by
dividing the data into smaller subsets. Using the k-d tree for dividing data
into equal subsets in KELM seems promising as two fundamental problems are
resolved in KELM making them suitable for large vision-based datasets 1) Curse
of dimensionality and 2) Strongly coupled network structure.
This novel classifier is named as multicolumn kernel extreme learning machine
(MCKELM) classifier as it solves the dimensionality problem and takes
advantage of a distributed cloud-based environment by decoupling the network
structure of KELM during the classification stage of food detection. The
proposed classifier uses reliable features from the efficient neural network
after filtering irrelevant features using feature selection methods that use
SHAP values from Gradient Explainer. Lastly, the visualizations by
GradientExplainer highlight the pixels in the photo based on SHAP values,
which helps to understand the contribution of every region in the image. Hence
improving the interpretability of the proposed architecture compared to
existing techniques during the feature selection process.

In summary, the contribution of this paper is summarized as follows.

1) Prepared a large dataset for food detection using nine publically available
datasets.

2) Exploited efficient neural networks for real-time feature extraction

3) Employed feature selection strategy for reliable feature selection using
SHAP values from gradient explainer

4) Proposed novel MCKELM by successfully integrating k-d tree with KELM to
solve the curse of dimensionality problem.

We organized our work as follows. The relevant studies of Food detection are
discussed in section 2, followed by a detailed explanation of our proposed
approach in section 3. In section 4, we discussed the evaluation criteria and
the results of our proposed methods in the context of food recognition.
Finally, in section 5, we discussed the results, followed by a conclusion and
future research work in this area.

##  2 Literature Review

Recent methods employ CNN for food detection as CNN consists of convolution
layers and pooling layers which learns efficiently high-level features of the
data. However, linear-fully connected classifiers in CNN have reduced
classification performance for feature vectors that are not linearly
inseparable. Similarly, the training process of CNN is sensitive to the local
minimum in the training error surface. Therefore, the generalization
performance of fully connected layers in CNN is not always strong enough.
Moreover, a non-linear classifier that uses visual features via CNN faces a
challenge in selecting optimal features from high dimension space. This
increases the importance of new efficient hybrid methods for food detection
that solves challenges involved in choosing optimal attributes from CNN and
reducing the high complexity of non-linear classifiers. This section discusses
the existing literature on food detection in context with these challenges.
There on summarized the research gap of existing methods followed by the
contribution of this work to existing literature.

Kitamura et al. [2008] have used handcrafted features with a support vector
machine for food/non-food classification. The attributes are based on the
color histograms, detected image patterns and DCT coefficients, etc. Their
method has used handcrafted features and outperformed the present methods that
use visual feature vectors via a convolutional neural network.
Kagaya et al. [2014] has proposed a CNN network for food/non-food recognition.
The neurons in the multilayer neural network process small patches of the
previous layers and are robust against small rotations and shifts. They have
used a Cuda-Covent c++ implementation of CNN on a graphics processing unit
(GPU), and they also prepared a midsize food/non-food dataset for this
purpose. However, they have tested their method on a small size food/non-food
dataset and had limited generalizability.
Kagaya and Aizawa [2015] have proposed a CNN-NIN network for food detection.
The network consists of four convolutional layers with two ’mplconv’ layers.
They employed the pre-trained model to copy mid-level image representations
and model parameters. The dimension of output classes is reduced from 1000 to
2 for the food/non-food classification. The main advantage of their model is
memory efficiency. However, they tested against a small food/non-food dataset.
Singla et al. [2016] has used the pre-trained model google-net for food/non-
food classification and conducted experiments on their datasets gathered from
various online sources. They have achieved an overall accuracy of 99.2% and
83.6% for food class recognition.
Several studies proposed deep learning models to detect food portions from
images of egocentric food datasets Jia et al. [2018] after gathering their
datasets from wearable devices using the e-buttons. The major objective of
their research is to detect food items from egocentric images. The gathered
dataset contains 29515 images from research participants from week-long
recordings, and they have achieved an accuracy of 91.5% for food detection.
However, their method is evaluated on limited data, and secondly, the fully
connected layer in CNN can not discriminate features with high precision. 109
[2021] proposed composite architecture for the real-time classification of
egocentric food/non-food images. It consists of the deep neural network,
shallow learning network, and probabilistic network interface. Their combined
method performs better than other deep learning approaches based on an
integrated set of measures.
The extensive review of existing methods showed that the CNN network has
attained state-of-the-art performance on various food/non-food data sets.
However, there are several research gaps. At first, large datasets do not
exist for the food detection problem. Most of the methods are evaluated for
small datasets. Secondly, the linear classifier in CNN is less appropriate for
linearly inseparable features. To address this, we evolved KELM and proposed
MCKELM to solve the ’curse of dimensionality problem faced by a single KELM
for a large dataset. MCKELM has a good generalization ability for linearly
inseparable features. Moreover, existing frameworks are not explainable in
contrast to our framework, which uses Gradient Explainer to highlight the
pixels in the photo hence helping in visualizing areas that are given more
importance during the feature selection process. The following section briefly
describes our methodology.

##  3 Methodology

The proposed framework extracts the features from a fully connected layer of
an efficient convolutional neural network and then selects reliable attributes
for reducing complexity. There on, it employs a non-linear classifier for
classification. Figure 2 shows the architecture diagram of the proposed
framework. It is to be, noted that we fine-tuned the pre-trained Imagenet
model by using transfer learning on our dataset. After feature extraction, we
introduced the strategy based on Shapley-value-based interpretations to rank
and select the best features. Finally, for the classification phase, we
proposed a novel MCKELM. The proposed classifier has good generalization
ability and solves the problem faced by KELM for large datasets.

Figure 2: General architecture of the proposed framework for food detection.

###  3.1 Efficient Feature Extraction Using MobileNet

At first, all the variants of MobileNet are finetuned on our food detection
dataset. Figure 3 shows the finetuning process on the food/non-food detection
dataset. There on, fine-tuned model of efficient neural network is employed
for real-time features extraction, followed by reliable feature selection and
detection of food/non-food classes with high accuracy.

Figure 3: Transfer learning to fine-tune pre-trained models on food detection
datasets for feature extraction.

Then we evaluated performance of MobileNetV1 Howard et al. [2017], MobileNetV2
Sandler et al. [2018], and MobileNetV3 Howard et al. [2019] for feature
extraction from food/non-food dataset. The brief details of MobileNet,
including their strength and weakness, are as follows. MobileNetV1 is
streamlined architecture and uses depthwise separable convolutions for
constructing lightweight deep convolutional neural networks. In it, depthwise
convolution performs a single convolution on each of the input channels, and
the point convolution filter then linearly combines the output using
convolutions. Figure 4 shows the MobileNetV1 block diagram.

Figure 4: MobileNetV1 block

MobileNetV2 expands over the idea of MobileNetV1 and adds the expansion layer
in the block for getting the system of expansion-filtering-compression Figure
5. Inverted Residual Block introduced in MobileNetV2 improves the performance
as each block consists of narrow input and output layer does not have
nonlinearity. This approach is followed by higher dimensional space and
projection to output. Eventually, the residual connects the bottleneck.

Figure 5: MobileNetV2 block with residual

MobileNetV3 solves the challenges faced by MobileNetV2 by introducing the
squeeze and excitation strategy, which gives unequal weights to different
input channels as opposed to equal weights from a normal CNN. In contrast to
the separate addition of the Squeeze and excitation layer in the ’resnet’
block, it applies them in parallel for improving performance and accuracy.
Figure 6 shows the MobileNetV3 block diagram.

Figure 6: MobileNetV3 block

The results section has briefly discussed the comparison results of variants
of MobileNet on the food detection dataset leading to the conclusion that
MobileNetV3 selected for feature extraction has superior performance on the
food detection dataset in contrast to other variants of MobileNet. Following
feature extraction selecting a reliable set of features from a high
dimensional space of an efficient neural network is vital to improve overall
performance. The following section discusses our method for reliable feature
selection from high dimensional space to achieve the objective of optimal
performance.

###  3.2 Gradient Explainer for Feature Selection From High Dimensional
Feature Vectors

For reliable feature selection, the proposed method uses SHAP values from
GradientExplainer Chen et al. [2021]. The GradientExplainer combines the ideas
from integrated gradients, SHAP, and Smooth gradients into a single expected
equation and uses the entire dataset as a background distribution instead of a
single reference value after allowing local smoothing for approximating the
model as a linear function between each background sample and currently
explained input.

The proposed method firstly uses the GradientExplainer method to compute the
SHAP score of features, showing the importance of features on the output
layer. There on, proposed method has calculated the SHAP score from the
pooling layer (3,3,1280) of the MobileNetV3 for training images and then
applies average pooling to get a single vector. Equation 1 has aggregated the
SHAP scores of the features of each class, while Equation 2 computes average
mean SHAP score of each class.
Based on the aggregated score, the method ranked the features, determined the
indexes of the top 500 attributes with the maximum SHAP score. Figure 7 showed
the process for attribute selection.

$\displaystyle\begin{array}[]{@{}l}\varphi^{c}=\;{\textstyle\displaystyle\frac{\sum_{i=1}^{i=t}\phi^{i}}{t}}\end{array}$
(1)

$\displaystyle\begin{array}[]{@{}l}A=\;{\textstyle\displaystyle\frac{\sum_{i=1}^{i=n}\varphi^{i}}{n}}\end{array}$
(2)

During the testing phase, features are selected corresponding to the indexes
with the maximum Shap score on the training dataset.

Figure 7: Shows the process of feature selection.

###  3.3 Parallel Multiclassification for Big Data

Following efficient feature extraction and reliable feature selection, the
final phase in our framework is highly accurate food detection. To achieve the
objective, we proposed a novel method for food detection by inheriting the
deep representation of the CNN and the approximability of KELMs. Compared to
the linear-fully connected classifier, KELM discriminates features better due
to its nonlinear function. However, large datasets suffer from the curse of
dimensionality. Reduced kernel extreme learning machine (RKELM) reduces the
dimensionality of the kernel matrix by randomly selecting the 10 percent of
nodes as mapping nodes. However, it is not suitable due to variation in the
classification performance. Especially for a performance-critical task like
food/non-food classification, detecting the non-food image as food or vice
versa affects the usability. Furthermore, RKELM also suffers from the
dimensionality problem for a huge dataset. To solve this problem, our
introduced MCKELM uses the k-d tree algorithm to divide the data into multiple
subsets and then train separate KELM on each subset. A brief discussion of the
background and mathematical working of the proposed method for food detection
is as follows. Table 1 and Table 2 shows the acronyms definition and notation
definitions.

Table 1: Acronyms definition  Symbol |  Description
---|---
ELM |  extreme learning machine
KELM |  kernel extreme learning machine
k-d tree |  k dimensional tree
MCKELM |  Multi column kernel extreme learning machine
Table 2: Notation definition  Symbol |  Description
---|---
t |  Total number of images in each class.
c |  Total number of classes.
$\varphi$ |  SHAP score of all attributes for each class.
A |  SHAP score vector of all attributes.
$\eta$ |  Number of chopping process.
N |  Number of resultant subsets.
C |  Single chop of the dataset.
$D^{i}$ |  Average density of chopped dataset.
O |  Original dataset density.
$\mathfrak{H}$ |  Denotes single subser in MCKLEM.
$x_{k}$ |  Training input vector of the kth training data set.
$\widehat{x}$ |  input vector
$x^{i}$ |  ith element of the input vector x
$K$ |  kernel matrix of KELM
$\beta$ |  Output Weights
Y |  Output Labels

####  3.3.1 Kernel Extreme Learning Machine

KELM is a unified learning model where they removed optimization restrictions
of support vector machine (SVM), least-squares support vector machines (LS-
SVM), and proximal support vector machines (PSVM) from the bias term. With
lower optimization constraints, the generalization performance is enhanced,
while also reducing the complexity.
According to Huang et al. Huang et al. [2012] if hidden layer ‘h.’ is unknown,
they can apply Mercer’s conditions on an extreme learning machine (ELM) by
defining the kernel matrix for ELM by using eq. (3).

$\displaystyle\begin{array}[]{@{}l}K_{ELM}\;=\;HH^{T}\;:\;K_{ELMi,j}\;=\;h(x_{i})\;=\;h(x_{j})\;=\;K(x_{i},\;x_{j})\end{array}$
(3)

Where, $\;k(x_{i},y_{i})$ is the output of the single-layer hidden neural
networks.

The output function of KELM is denoted by eq. (4)

$\displaystyle\begin{array}[]{@{}l}f(x)\;=\;\begin{bmatrix}\begin{array}[]{c}k(x,x_{1})\\\
\vdots\end{array}\\\
k(x,x_{n})\end{bmatrix}^{T}\;(CI+K_{KELM})^{-1}T\end{array}$ (4)

According to Frenay and Verleysan Frénay and Verleysen [2011] when h(x) is
known, KELM can be defined by using eq. (5).

$\displaystyle\begin{array}[]{@{}l}k(u,v)\;=\;\lim_{L\rightarrow\infty}\displaystyle\frac{1}{L}h(u).h(v)\end{array}$
(5)

There are various kernels from the existing research work that satisfies
mercer’s condition. This research work has used the radial basis function
(RBF) kernel and chi-square kernel. 1) RBF Kernel

$\displaystyle\begin{array}[]{@{}l}k(\mu_{i},\sigma_{i},\gamma,x)\;=\;exp\left(\displaystyle\frac{\gamma\left\|x-\mu_{i}\right\|^{2}}{\sigma_{i}^{2}}\right)\end{array}$
(6)

2) Chi-square Kernel

$\displaystyle\begin{array}[]{@{}l}k(\mu_{i},\sigma_{i},\gamma,x)\;=\;exp\left(-\sigma\;Sum\;\left[\displaystyle\frac{(x\;-\;\mu)^{2}\;}{(x\;+\mu)}\right]\;\right)\end{array}$
(7)

The features must be normalized between 0 and 1 when using the chi-square
kernel, as $\mu$ must contain values for this kernel.

In the above kernel adjustable parameter, $\sigma$ significantly affects the
classifier performance and should be adjusted based on the dataset. In
contrast to the ELM algorithm, knowing the feature mapping in the hidden layer
is not required, and there is no need to select the number of hidden neurons.
Compared to the linear-fully connected classifier, KELM has a superior ability
for discriminating feature categories. Figure 8 shows the visual comparison
between KELM and linear fully connected layer.

Figure 8: Visual illustration of the decision boundary of linear classifier
and KELM.

####  3.3.2 k-d tree

k-d tree is an important data structure used to store a finite set of points
in a k-dimensional space. It is a primarily multi-dimensional binary tree,
proposed by Jon Louis Bentley in 1975, which was originally introduced to
solve multiple problems emerging in geometric databases Bentley [1979].
Additionally, the k-d tree is well-known for curbing the construction of
subsets with zero data and ensuring uniform data distribution, hence enhancing
the speed of learning in neural networks Omohundro [????]

For food detection tasks k-d tree algorithm produces subsets of training data
to avoid excessive computations and time delays resulting from a high number
of hidden nodes in the single kernel matrix.

Since each non-leaf node in the k-d tree creates a binary-splitting
hyperplane. it distributes the points into left and right sub-trees, with each
leaf node representing a k-dimensional point. Consequently, it reduces network
training and testing time due to fewer hidden nodes in each subset. Each
subset act as independent training data set for each KELM. In the k-d tree,
subsets possess the same density as the original dataset and contain enough
instances for training purposes. Algorithm 1 explains the steps for chopping
the dataset into subsets and the training process of the KELM algorithm on
these subsets.

Figure 9: 3-dimensional visualization of the k-d tree shows the splitting
process. The first split (red) cuts the root dataset into two subsets. Each of
these subsets is further split (green) into two subsets. As we cannot divide
it further, the final eight subsets are called leafs (final subsets).

####  3.3.3 Multicolumn Kernel Extreme Learning Machine

The proposed multicolumn kernel extreme learning machine network (MCKELM)
method reduces the complexity of inner matrix computations during the training
and testing process in KELM. It uses parallel structures and requires fewer
hidden units per KELM and fewer instances of datasets to train each KELM.
MCKELM operates by dividing the large dataset into smaller subsets using the
k-d tree algorithm and considers the resulting ’N’ subsets as individual
datasets required to train ’N’ separate KELM. Consequently, the individual
KELMs are then stacked in a parallel manner as shown in Figure 10 and it only
selects top k subsets during the testing phase of MCKELM.
As it trains KELM on their dataset in isolation from other KELMs, MCKELM is
considered a manageable and highly developed parallel structure instead of
traditional KELM, which suffers from the drawback of computing-intensive
kernel inner product calculations. Thus, MCKELM is an ideal approach for
transitioning from a single large structure to a parallel mechanism.
During testing, when the model processes a feature vector of the input image,
the input subset selector using the k-d tree forwards it to the appropriate
KELMs. Following this, the selected KELMs give their output using the majority
voting method to compute the final results. We described the three stages as
follows. Algorithm 2 shows the step of the testing process.
1\. Individual subset selector:
Whenever a new instance comes, the input subset selector selects K KELMs based
on the k-d tree algorithm. The k-d tree algorithm determines the euclidean
distance between the test vectors and all the nodes in subsets by using
Equation (8). $d_{k}$denotes the kth minimum euclidean distance between the
test input vector $\widehat{x}$and kth training node vector $x_{k}$;
$\widehat{x}^{i}$ is the ith value of the input vector $\widehat{x}$ and
$x_{n}^{i}$ is the ith value of the nth training vector $x_{n}$. It then
selects k nearest vectors with minimum euclidean distance d as shown in Figure
11. The selected vectors belong to K subsets, and it selects only KELMs
trained on these subsets during the food detection.

$\displaystyle\begin{array}[]{@{}l}d_{k}=\underset{1...k}{min}(\left\|\widehat{x}\;-\;x_{n}\right\|)\;=\underset{1...k}{min}\left(\sqrt{\sum_{i=1}^{I}}(\widehat{x}^{i}-x_{n}^{i})^{2}\right)\end{array}$
(8)

2\. Individual KELMs:
Every ‘k’ neighbor belongs to the subset of the entire dataset required to
train separate KELMs, and only selected nearest KELMs work in a parallel
manner to generate ‘n’ results.
3\. Output combiner:
It only selects the output from K nearest KELMs. The remaining KELMs do not
contribute to the output result. MCKELM computes the final output result by
using the simple majority method.
MCKELM method is also significantly advantageous in reducing testing time due
to parallelized structure in contrast to large connected KELM as smaller
subsets essentially avoid generalization problems and instead specify it by
informing the MCKELM about the regional experience. Moreover, MCKELM also
improves the training and testing times while having superior performance,
compared to the softmax classifier. In the results section, we briefly discuss
the performance of the classifier.

⬇

1Input:

2$\eta$ Number of chopping process.

3Deep Features of food/non-food training dataset.

4Kernel Extreme Learning Machine model.

5Output:

6$2^{\eta}\;\ast\;KELM$ trained models of individual KELM

7Algorithm:

8$N\;=\;2^{\eta}$ Number of subsets and KELM model.

9Compute original density of dataset O.

10for $\mathbb{C}$ = 1 to $\eta$

11 for subset= 1 to $(2^{\mathbb{C}}-1)$

12 Find the median of each feature

13 Compute average density $D^{i}$of chopped dataset

14 Choose feature with minimum $\left|D^{i}-O\right|$

15 Divide along the median in to two subsets.

16 end for

17end for

18for subset = 1 to $\eta$

19 Train KELM on the subset

20end for

Code Snippet 1: Dividing food/non-food dataset into N subsets and training of
KELMs

⬇

1Input:

2Test Input

3Output:

4y, classification results

5Algorithm:

6Load $2^{\eta}$ KELM model, where $\eta$ is the total number of chops

7for n = 1 to t

8 find the k nearest training nodes to the $\widehat{x}$

9 for each subset belonging to k nearest nodes

10 Compute the output $y_{k}$ using KELM trained on the subset

11 end for

12end for

13Majority voting method for final result $y$

Code Snippet 2: Testing process  Figure 10: Multicolumn Kernel Extreme
Learning Machine Figure 11: Selection of k subsets and k KELM based on the KNN
algorithm

##  4 Experiments and Results

This section discusses the food/non-food dataset gathered for this study to
evaluate the food detection framework, implementation of the framework, and
detailed results of our analysis of each phase. In the first experiment, we
determined the best feature extractor from the three variants of the MobileNet
for food detection. The second experiment shows that the feature selector
method based on Gradient Explainer reduces the training time. The third
experiment analyzes the classification performance of MCKELM in contrast to
other state-of-the-art methods. Finally, we presented interpretations of the
features by using the Shapley-value-based explanations.

###  4.1 Food/non-food Dataset

We prepared a large-scale food/non-food dataset by using the existing image
datasets of UECFOOD100 Matsuda and Yanai [2012], UECFOOD256 Kawano and Yanai
[2014], Caltech 256 Griffin et al. [2007], Instagram Images, Flickr Image
Dataset Zhou et al. [2020] , Food101 Bossard et al. [2014], Malaysian Food
Dataset, Indoor Scene recognition Dataset Quattoni and Torralba [2009] , 15
scene dataset. Inclusion of datsets of various types and from various sources
has introduced the diversity in the food detection dataset. As these dataset
includes food items of various cultures and images from popular social media
platforms. Figure 12 shows the sample images from included datasets. To
prepare a dataset for food detection, we define the process which initially
filtered the images based on their metadata information. There on, an expert
has manually verified food images of the dataset with unclear metadata
information to improve the authenticity of the food detection dataset. Figure
13 diagrammatically illustrates the process of preparation of the food
detection dataset. After preparation, the food detection dataset consists of
two classes: food class and non-food class. It has 222430 images for the
training and 55096 images for testing, and we open-sourced the dataset for
research purposes Tahir and Kiong [2020]. To the best of our knowledge, it is
the largest publically available food/non-food dataset, hence making it
suitable for verifying the effectiveness of our proposed multicolumn approach
in solving the ’curse of dimensionality problem’ faced by KELM. Figure 14
shows the sample images from our dataset.

Table 3: Attributes of food/non-food dataset  Dataset |  Total class |  Total instance |  Train/Test instance
---|---|---|---
food/non-food |  2 |  277,526 |  222,430/55,096
Figure 12: Sample images from various dataset used to prepare food detection
dataset. Figure 13: Food detection dataset preparation process. Figure 14:
Sample images from food detection dataset.

###  4.2 Implementation

We experimented on the Google Colab (12 GB GPU) and 12 GB random access memory
(RAM) to determine the best feature extractor from three variants of the
MobileNet. Then for feature extraction and feature selection, we have also
used Google Colab. Finally, we trained our classifier using an AMAZON EC2
instance with 64 GB of memory. The trained model is accessible through a web
service implemented in python Django.

###  4.3 Efficient Feature Extraction Using MobileNet

We compared MobileNetV1, MobileNetV2, and MobileNetV3 for feature extraction
after selecting the network for our framework with the highest classification
performance by finetuning all the models for ten epochs at a learning rate of
0.01 and a momentum value of 0.9. The kernel regularizer parameter l2 is set
to 0.0005 as shown in Table 4 . We used stochastic gradient descent (SGD)
optimizer, categorical cross-entropy, and softmax activation function. Table 5
shows the experimental configurations and online data augmentation used for
increasing the generalization while Table 6 shows the comparison for
classification performance. MobileNetV3 outperforms its predecessors in terms
of accuracy, and it is faster in contrast to the previous architectures of
MobileNet. For this reason, we employed MobileNetV3 in our final framework.

Table 4: Parameter settings  Parameter |  Value
---|---
Epochs |  10
Learning Rate |  0.01
Momentum |  0.9
L2 |  0.0005
Optimizer |  SGD
Table 5: Experimental settings of data augmentation  Data-Augmentation Type |  Value
---|---
Wide Shift |  0.2
Height Shift |  0.2
Horizontal Shift |  Randomly flips horizontally
Zoom Range |  0.8 to 1
Channel Shift Range |  30
Full Mode |  Reflect
Table 6: Comparison of classification accuracy on linear fully connected classifier  Datasets |  MobileNetV1 |  MobileNetV2 |  MobileNetV3
---|---|---|---
food/non-food |  98.33 |  98.56 |  98.73

###  4.4 Gradient Explainer for Image Feature Selection From High Dimensional
Data

Feature vectors extracted from the CNN model have a very high dimension.
This work employed the strategy which uses SHAP values from GradientExplainer
to select the best top 500 features. The comparison of the training time, and
testing time in Table 7 shows that the feature selection reduces the
computational complexity while having competitive classification performance.
The reduction in training time is significant when the total number of subsets
is less in MCKELM. However, this difference decreases when subsets are more
due to reduced complexity of inner matrix computations, as experimental
analysis shows that it reduces testing time by 77.14% for 16 subsets and 52%
for 32 subsets.

Table 7: Comparison of Training/Testing Time by using SHAP feature selection  |  Training Time |  Testing Time
---|---|---
Model |  All Features |  SHAP |  All Features |  SHAP
MCKELM
(16 Subsets) |  660 s |  343 s |  0.35 s |  0.08 s
MCKELM
(32 Subsets) |  302 s |  306 s |  0.23 s |  0.11 s

###  4.5 Visual Interpretation of Features From CNN

We employed ’GradientExplainer’ to visualize the pixels and their contribution
to the output, as these pixels provide features that cause the classifier to
lean towards a particular class. Our fast approximation algorithm computes the
game theory-based SHAP values Chen et al. [2021] which considers multiple
background samples instead of a single baseline. In our experiment, we set the
number of background samples to 500 after randomly selecting these samples
from the training dataset of both classes. We set the ranked output parameter
to 2. Figure 15 shows the visualizations generated by GradientExplainer
method. The red pixels provide the features that make the classifier lean
toward the positive classification, and the blue pixels provide features that
make the classifier tends toward the negative classification. These
visualizations indicate regions in the image, which tend to bend the decision,
hence increasing our confidence in framework predictions by ensuring that the
classifier is making decisions based on the features from the desired regions
of the photos.

Figure 15: The red pixels provide the features that lean the classifier
towards positive classification, while the blue pixels provide features that
tend the classifier towards the negative class.

###  4.6 Classification Performance on Big Data

This section compares the experimental results of the proposed MCKELM with
other classifiers. Section 4.5.1 presents the measures used for analysis, and
Section 4.5.2 presents the results of the MCKELM in contrast with other
methods.

####  4.6.1 Measures

We used the following metrics to measure classification performance.

##### Accuracy

It measures the ability of the classifier to predict the class accurately.
Equation (8) denotes the accuracy of the network.

$\displaystyle\begin{array}[]{@{}l}Accuracy\;=\displaystyle\frac{\;(TP+FN)}{(TP+FP+FN+TN)}\times
100\end{array}$ (8)

##### F1 Score

It is interpreted as a weighted average of precision and recall as both recall
and precision contributed equally to the weighted average. Equation (9)
denotes the F1 Score.

$\displaystyle\begin{array}[]{@{}l}F1\;Score\;=\;\displaystyle\frac{(2\ast(Precision\ast
Recall))}{Precision+Recall}\end{array}$ (9)

##### Recall

It is the network’s ability to find all positive instances. Equation (10)
denotes the Recall score.

$\displaystyle\begin{array}[]{@{}l}Recall\;=\;\displaystyle\frac{TP}{(TP+FN)}\end{array}$
(10)

##### Precision

It measures the ability of the network to not label negative samples as
positive. Equation (11) computes the precision of the model.

$\displaystyle\begin{array}[]{@{}l}Precision\;\;=\displaystyle\frac{TP}{(TP\;+FP)}\end{array}$
(11)

####  4.6.2 Results

As food detection is a binary problem, we assumed a batch learning fixed-class
scenario for training MCKELM using processed features. In our experiments
kernel parameter is set to 1 for MCKELM, and the value of the k-d tree’s leaf
size in MCKELM is set to 13000 for 32 subsets and 25000 for 16 subsets. In
K-nearest neighbor (KNN), the number of the nearest neighbor k value is 1. For
RKELM, the kernel matrix randomly selects 10 percent of nodes as mapping
nodes. For ELM, the number of hidden neurons value is determined by using the
following heuristic.

We cannot evaluate the KELM algorithm for such a large dataset due to the
curse of dimensionality as it requires more than 384 GB of ram for processing
the feature matrix of food/non-food datasets. However, our proposed MCKELM
resolves the dimensionality problem faced by KELM. Table 12 compares the
classification results of deep learning models and MCKELM. We reported the
results of MCKELM for 16 subsets and 32 subsets. The experimental analysis
shows that novel MCKELM for classification improves performance compared to
MobileNetV3 by 0.93%. It is worth noting that the dataset gathered by us for
food/non-food recognition is large, and the improvement of the performance by
MCKELM is significant as it correctly classifies 513 more images compared to
MobileNetV3. The comparison with the Extreme learning machine shows that ELM
misclassifies 66 instances more than MCKELM. Similarly, reduced kernel extreme
learning machine has misclassified 105 instances, k nearest neighbor has
misclassified, 54 instances, and Naive Bayesian classifier misclassified 109
more instances compared to proposed MCKLEM. Figure 16 presents the confusion
matrix of MobileNetV3 and MCKELM at different subsets that further elaborates
the improvement by MCKELM.
Figure 17 shows the comparison of the training and testing time at various
subsets of MCKELM. For MCKELM increasing the number of subsets reduces
training time and testing time for our case.

Table 8: Comparison of food/non-food Classification.  Model |  Accuracy |  Precision |  Recall |  F1 Score
---|---|---|---|---
MobileNetV1 |  98.33 |  96.76 |  99.39 |  98.05
MobileNetV2 |  98.56 |  97.51 |  99.14 |  98.32
MobileNetV3 |  98.73 |  97.45 |  99.60 |  98.52
MobileNetV3 (Ensemble) |  99.12 |  99.02 |  99.18 |  99.10
Naive Baysian (AF) |  99.00 |  99.00 |  99.00 |  99.00
Naive Baysian (SHAP) |  99.46 |  99.39 |  99.49 |  99.44
KNN (AF) |  99.56 |  99.56 |  99.55 |  99.55
KNN (SHAP) |  99.58 |  99.56 |  99.57 |  99.57
ELM (AF) |  99.58 |  99.57 |  99.58 |  99.58
ELM (SHAP) |  99.54 |  99.53 |  99.54 |  99.55
RKELM (AF) |  98.98 |  98.89 |  99.04 |  98.96
RKELM (SHAP) |  99.47 |  99.45 |  99.47 |  99.46
MCKELM (RBF)
(16-subsets) (AF) |  99.66 |  99.67 |  99.66 |  99.67
MCKELM (RBF)
(32-subsets) (AF) |  99.66 |  99.67 |  99.66 |  99.67
MCKELM (RBF)
(16-subsets) (SHAP) |  99.66 |  99.64 |  99.66 |  99.65
MCKELM (RBF)
(32-subsets) (SHAP) |  99.65 |  99.63 |  99.64 |  99.66
MCKELM (Chi-Square)
(16-subsets) (SHAP) |  99.66 |  99.65 |  99.66 |  99.66
MCKELM (Chi-Square)
(32-subsets) (SHAP)
|  99.66 |  99.64 |  99.66 |  99.66
Figure 16: Confusion matrix of linear fully connected classifier in
MobileNetV3 and MCKELM at 16 subsers and 32 subsers for food/non-food
classification. Figure 17: Comparison of training and testing time at
different subsers of MCKELM

##  5 Discussion

We state the following considerations based on the experiments carried out for
this research work. Existing datasets of food detection including Food5k, are
mid-size datasets, and models trained on these datasets are not generalized
enough to tackle a wide array of food detection challenges in a real-world
environment. For this purpose, at first, we prepared a large dataset for the
food detection problem and achieved this objective by using nine publically
available datasets and then divided them into two categories: Food and non-
food. Following that, we have proposed a novel framework that extracts
features from the efficient neural network, employs SHAP values for reliable
feature selection, and then multicolumn kernel extreme learning machine for
highly accurate food detection.

For feature extraction, we have explored the potential of efficient neural
networks due to their fast inference. Firstly, we used transfer learning on a
pre-trained model to fine-tune variants of efficient neural networks for the
food detection problem. Following that, our analysis in the results section
shows MobileNetV3 has superior performance as it uses the squeeze and
excitations strategy giving unequal weights to different channels in contrast
to normal weights, hence leading to the superior performance of MobileNetV3,
compared to other models for feature extraction in our framework.
In the second step of our pipeline, we have selected the most reliable
features for real-time classification. The attributes from the deep model have
high dimensions, which increases the training and testing time. Shapley-value-
based explanations showed that all attributes of feature vectors do not
contribute equally towards the correct class. We proposed a strategy that
selects the top features with the highest Shapley score for the prediction.
Table 7 shows that feature selection significantly reduces the training time
while having competitive performance. The advantage of this method over
popularly employed RELIEF F is that Shapley-value-based explanations help us
to visualize and understand the regions in the image based on which classifier
has made the decision.

Finally, in the last step, we solved the challenge faced by a linear-fully
connected layer and employed MCKELM to discriminate linearly inseparable
features for highly accurate food detection. MCKELM has solved two major
problems faced by KELM, especially in the scenario of large datasets. 1) Curse
of dimensionality 2) Strongly coupled network structure. Our results on a
large food/non-food dataset show that the proposed framework improves
performance by 0.93%, in contrast to MobileNetV3. The improvement is
significant as the dataset is large, and the proposed framework classifies 513
images correctly as compared to MobileNetV3. Similarly, the comparison with
other classifiers in our framework also shows the superior performance of
MCKELM in terms of accuracy, precision, recall, and F1 score.

##  6 Conclusion

Food detection is a performance-critical task and requires high classification
accuracy. This work has proposed a hybrid framework that takes optimal
features from the deep model and uses a separate nonlinear classifier after
successfully employing the efficient neural network MobileNetV3 for fast
feature extraction. There on, select the optimal subset of features by using
SHAP values from the gradient explainer. As Gradient Explainer has explained
the contribution of the attributes towards the correct class. This approach
has improved the interpretability of the feature selection process. Finally
and importantly, we have evolved KELM to MCKELM for the classification stage
of our pipeline. KELM is unsuitable for large datasets as increasing the
number of hidden nodes has also increased the complexity of inner matrix
computations. Moreover, KELM has a strongly coupled network structure which
makes it inappropriate for a distributed environment in the cloud. Our
proposed MCKLEM for the classification has addressed these two problems by
employing a multicolumn approach after successfully using the k-d tree to
divide the data into N equal subsets. Then, it has trained separate KELM on
each subset of data. During the testing process, we selected only the top k
nearest subsets instead of the whole network. The comparison on a large
food/non-food dataset prepared by us has demonstrated that the proposed
algorithm has superior performance than the linear fully-connected classifier
in the deep model and other variants of ELM. The multicolumn approach
significantly has reduced the computational time by successfully decoupling
the network structure.

## Author Contributions

Ghalib Ahmed Tahir was responsible for writing algorithms, analysis, writing
the article and literature search, and approved the final version as
submitted. Loo Chu Kiong contributed to the study design, reviewed the study
for intellectual content, and confirmed the final version as submitted.

## Acknowledgements

This research was supported by the Universiti Malaya Impact-oriented
Interdisciplinary Research Grant Programme (IIRG) - IIRG002C-19HWB, Universiti
Malaya Covid-19 Related Special Research Grant (UMCSRG) CSRG008-2020ST and
Partnership Grant (RK012-2019) from University of Malaya.

##  7 Conflict of Interest

The authors declare there is no conflict of interest.

## References

  * 109 [2021] (2021) Food/Non-Food Classification of Real-Life Egocentric Images in Low- and Middle-Income Countries Based on Image Tagging Features. Frontiers in Artificial Intelligence 4:28, DOI 10.3389/frai.2021.644712
  * Bentley [1979] Bentley JL (1979) Multidimensional Binary Search Trees in Database Applications. IEEE Transactions on Software Engineering SE-5(4):333–340, DOI 10.1109/TSE.1979.234200
  * Bossard et al. [2014] Bossard L, Guillaumin M, Gool LV (2014) Food-101 – Mining Discriminative Components with Random Forests. pp 446–461, DOI 10.1007/978-3-319-10599-4˙29
  * Chen et al. [2021] Chen H, Lundberg S, Lee SI (2021) Explaining Models by Propagating Shapley Values of Local Components. DOI 10.1007/978-3-030-53352-6˙24
  * Cireşan et al. [2012] Cireşan D, Meier U, Schmidhuber J (2012) Multi-column Deep Neural Networks for Image Classification. Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition IEEE Computer Society Conference on Computer Vision and Pattern Recognition
  * Frénay and Verleysen [2011] Frénay B, Verleysen M (2011) Parameter-insensitive kernel in extreme learning for non-linear support vector regression. Neurocomputing 74(16):2526–2531, DOI https://doi.org/10.1016/j.neucom.2010.11.037
  * Griffin et al. [2007] Griffin G, Holub A, Perona P (2007) Caltech-256 Object Category Dataset. CalTech Report
  * Howard et al. [2017] Howard A, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
  * Howard et al. [2019] Howard A, Sandler M, Chen B, Wang W, Chen LC, Tan M, Chu G, Vasudevan V, Zhu Y, Pang R, Adam H, Le Q (2019) Searching for MobileNetV3. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp 1314–1324, DOI 10.1109/ICCV.2019.00140
  * Huang et al. [2012] Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme Learning Machine for Regression and Multiclass Classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42(2):513–529, DOI 10.1109/TSMCB.2011.2168604
  * Jia et al. [2018] Jia W, Li Y, Qu R, Baranowski T, Burke L, Zhang H, Bai Y, Mancino J, Xu G, Mao ZH, Sun M (2018) Automatic food detection in egocentric images using artificial intelligence technology. Public Health Nutrition 22:1–12, DOI 10.1017/S1368980018000538
  * Kagaya and Aizawa [2015] Kagaya H, Aizawa K (2015) Highly Accurate Food/Non-Food Image Classification Based on a Deep Convolutional Neural Network. In: Murino V, Puppo E, Sona D, Cristani M, Sansone C (eds) New Trends in Image Analysis and Processing - ICIAP 2015 Workshops, Springer International Publishing, Cham, pp 350–357
  * Kagaya et al. [2014] Kagaya H, Aizawa K, Ogawa M (2014) Food Detection and Recognition Using Convolutional Neural Network. DOI 10.13140/2.1.3082.1120
  * Kawano and Yanai [2014] Kawano Y, Yanai K (2014) Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation. pp 3–17, DOI 10.1007/978-3-319-16199-0˙1
  * Kitamura et al. [2008] Kitamura K, Yamasaki T, Aizawa K (2008) Food Log by Analyzing Food Images. In: Proceedings of the 16th ACM International Conference on Multimedia, Association for Computing Machinery, New York, NY, USA, pp 999–1000, DOI 10.1145/1459359.1459548
  * Matsuda and Yanai [2012] Matsuda Y, Yanai K (2012) Multiple-food recognition considering co-occurrence employing manifold ranking. pp 2017–2020
  * Omohundro [????] Omohundro SM (????) Efficient Algorithms with Neural Network Behavior
  * Quattoni and Torralba [2009] Quattoni A, Torralba A (2009) Recognizing indoor scenes. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp 413–420, DOI 10.1109/CVPR.2009.5206537
  * Sandler et al. [2018] Sandler M, Howard AG, Zhu M, Zhmoginov A, Chen LC (2018) Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation. CoRR abs/1801.04381
  * Shao et al. [2014] Shao L, Wu D, Li X (2014) Learning Deep and Wide: A Spectral Method for Learning Deep Networks. IEEE Transactions on Neural Networks and Learning Systems 25(12):2303–2308, DOI 10.1109/TNNLS.2014.2308519
  * Singla et al. [2016] Singla A, Yuan L, Ebrahimi T (2016) Food/Non-food Image Classification and Food Categorization using Pre-Trained GoogLeNet Model. pp 3–11, DOI 10.1145/2986035.2986039
  * Tahir and Kiong [2020] Tahir G, Kiong LC (2020) Food/Non-food Image Classification. DOI 10.21227/dsqp-4609
  * Zhou et al. [2020] Zhou L, Palangi H, Zhang L, Hu H, Corso J, Gao J (2020) Unified Vision-Language Pre-Training for Image Captioning and VQA. Proceedings of the AAAI Conference on Artificial Intelligence 34:13041–13049, DOI 10.1609/aaai.v34i07.7005