#  A Review of the Vision-based Approaches for Dietary Assessment

Ghalib Tahir<EMAIL_ADDRESS>Chu Kiong Loo<EMAIL_ADDRESS>Department of Artificial Intelligence, Faculty of Computer Science and
Information Technology, University of Malaya, 50603 Kuala Lumpur, Malaysia

###### Abstract

Last ten years have witnessed the growth of many computer vision applications
for food recognition. Dietary studies showed that dietary-related problem such
as obesity is associated with other chronic diseases like hypertension,
irregular blood sugar levels, and increased risk of heart attacks. The primary
cause of these problems is poor lifestyle choices and unhealthy dietary
habits, which are manageable by using interactive mHealth apps that use
automatic visual-based methods to assess dietary intake. This review discusses
the most performing methodologies that have been developed so far for
automatic food recognition. First, we will present the rationale of visual-
based methods for food recognition. The core of the paper is the presentation,
discussion and evaluation of these methods on popular food image databases. We
also discussed the mobile applications that are implementing these methods.
The review ends with a discussion of research gaps and future challenges in
this area.

###### keywords:

Food recognition, Feature Extraction, Automatic-diet monitoring, Image
Analysis, Interactive segmentation, Food Datasets

##  1 Introduction

Despite recent advancements in medicine, the number of people affected by
chronic diseases is still significantly large. This rate is primarily due to
their unhealthy lifestyles and irregular eating patterns. Some of the more
notable chronic diseases include obesity, hypertension, blood sugar,
cardiovascular diseases, and different kinds of cancers. Out of these, obesity
and weight issues are becoming increasingly common around the globe. It
affects almost every part of the world, from middle to low-income countries.
In 2016, 1.9 billion adults 18 years and older were overweight, while 650
million were obese. With time, children are also becoming affected by obesity
at an alarming rate. According to World Health Organization (WHO), over 340
million children and adolescents between 5 and 19 years were overweight or
obese. [1].

The prevalence of these alarming statistics poses a serious concern. However,
determining the effective remedial measures depends on different factors,
ranging from a person’s genetics to lifestyle choices. To cope with chronic
weight problems, people usually keep notes to track their dietary intake. In
turn, dieticians require these records to estimate a patient’s nutrient
consumption. Many dietary mobile applications automate this process by
developing food recognition models that directly classify food categories,
ingredients from smartphone camera pictures.

However, automatic food recognition using a smartphone camera in the real
world is considered a multi-dimensional problem. Unlike other image
classification problems, food recognition is a complex task that involves
several challenges. There is no spatial layout information that it can
exploits like, in the case of the human body, there is a spatial relationship
between body parts. The head is always present over the trunk of the human
body [2, 3, 4] and feet towards the lower end. Similarly, the non-rigid
structure of the food and intra-source variations make it even more
complicated to classify food items correctly as preparation methods and
cooking styles vary from region to region. Moreover, inter-class ambiguity is
also a source of potential recognition problems as different food items may
look very similar (e.g. soups). Moreover, in many dishes, some ingredients are
concealed from view that can limit the performance of food ingredient
classification models.

In addition to this, image quality from the smartphone camera is dependent on
different types of cameras, lighting conditions, and orientations. As a
result, the poor performance of food recognition models is highly susceptible
to image distortions.

Despite these challenges, many food images possess distinctive properties to
distinguish one food type from another. Firstly, the visual representations of
food images are of fundamental importance as it significantly impacts
classification performance. Therefore, many food recognition methods employ
handcrafted features such as shape, colour, texture, local. Recent techniques
are using deep visual features for image representation. Some of these methods
implement a combination of handcrafted and deep visual features for image
feature representation. Secondly, for enhanced classification performance and
reduced computational complexity, an appropriate selection of attributes is
essential for removing redundant features from feature vectors. Finally,
wisely selecting classification techniques is crucial to address food
recognition challenges effectively.

The following research paper aims to scrutinize state-of-the-art vision-based
approaches developed for food recognition. It provides deep insight into
multi-label methods for food ingredient classification and existing mobile
applications which implements these approaches. Section II examines existing
datasets. Section III examines feature extraction methods for food image
representation including, handcrafted and deep visual features. In sections IV
and V, we presented the most performing classifiers for food category
classification techniques, food ingredient classification methods. In section
VI, we provide brief information about mobile applications that are
implementing these methods. Conclusively, we highlight our findings and future
works related to this topic.

##  2 Datasets Used for Food Recognition

Performance of feature extraction and classification techniques is highly
dependent on the detail-oriented collection of images, which, in our case,
happen to be food images. As consolidated large food image datasets, for
example, UECFOOD-100, Food-101, UECFOOD-256, UNCIT-FD1200, UNCIT-FD889 are
eventually used as benchmarks to collate recognition performance of existing
approaches with new classifiers. Such datasets can be distinctive in terms of
characteristics, such as the total number of images in a particular dataset,
cuisine type, and included food categories. Table 1 summarizes the
characteristics of food datasets surveyed in this research endeavour.

For instance, UECFOOD-100 contains 100 different sorts of food categories, and
each food category has a bounding box that indicates the location of the food
item in the photograph. Food categories in this dataset mainly belong to
popular foods in Japan [75]. Similarly, UECFOOD-256 is another variant of
UECFOOD-100. However, it differs in terms of the number of images as it
contains 256 food images of different kinds [75]. Food-101 contains 101000
real-world images that are classified into 101 food categories. It includes
diverse yet visually similar food classes [26]. Similarly, the PFID food
dataset is composed of 1098 food images from 61 different categories. The PFID
collection currently has three instances of 101 fast foods [99]. UNCIT-FD1200
is composed of 4754 food images of 1200 types of dishes captured from actual
meals. Each food plate is acquired multiple times, and the overall dataset
presents both geometric and photometric variability. Similarly, UNICT-FD 889
dataset has 3583 images [86] of 889 different real food plates captured using
mobile devices in uncontrolled scenarios (e.g., different backgrounds and
light environmental conditions). Moreover, they capture each dish image in
UNICT-FD899 multiple times to ensure geometric and photometric variability
(rotation, scale, point of view changes) [97].

Several datasets mainly consist of various food images collected through
various sources such as web crawlers, social media platforms like Instagram,
Flickr, and Facebook. Also, most of these datasets contain images of foods
that are specific to certain regions, such as Vireo-Food 172 [32] and
ChineseFoodNet [34]. Both datasets contain Chinese dishes. Similarly, Food-50
[16], Food-85 [16], Food log [103], UECFOOD-100 [75] and UECFOOD-256 [26]
contains Japanese food items. Turkish foods-15 [89] is limited to Turkish food
items only. Also, the Pakistani Food Dataset [58] accommodates Pakistani
dishes, and the Indian Food Database incorporates Indian cuisines. In addition
to this, few datasets only include fruits and vegetables like VegFru [80],
Fruits 360 Dataset [79], and FruitVeg-81 [67]. Furthermore, Table 1 provides
brief description about food image datasets. Figure 2 shows the sample images
from the food datasets.

Therefore, it is evident from the survey that there is an immense need for
broad and generic food datasets for better food recognition and enhanced
performance. This necessity is because region-specific food items or datasets
with fewer food categories can undermine the accuracy and performance of
classification and extraction methods.

Table 1: Food Image Datasets

Authors | Dataset | Food Category | Total # images/ Class | Image Source | Year | Citations | Ref.
---|---|---|---|---|---|---|---
Chen et al. | PFID | Fast Food Dataset | 1038(61) | Fast Food data captured in multiple restaurants | 2009 | 214 | [99]
Mariappan et al. | TADA | Artificial And Generic Food | 256(11) | Controlled Environment | 2009 | 58 | [74]
Joutou and Yanai et al. | Food-50 | Japanese Food | 5000(50) | Crawled From Web | 2010 | 169 | [16]
Hoashi et al. | Food-85 | Japanese Food | 8500(85) | Acquired From Food databases | 2010 | 90 | [16]
Miyazaki et al. | Foodlog | Japanese Food | 6512(2000) | Captured by Users | 2011 | 54 | [76]
Matsuda et al. | UECFOOD-100 | Japanese Food | 14,361(100) | Captured by Mobile Camera | 2012 | 229 | [75]
Chen et al. | ChineseFoodNet | Chinese dishes. | 192,000(208) | Gathered from web | 2012 | 108 | [34]
M.-Y. Chen et al. | Chen | Chinese Food | 5000/50 | Photos were taken manually or crawled from the Internet | 2012 | 126 | [98]
Bossard et al. | Food-101 | American Fast Food | 101000(101) | Crawled From Web | 2014 | 490 | [36]
L. Bossard and et al. | ETHZ Food-101 | American Fast Food | 100,000(101) | Crawled From Web | 2014 | 664 | [83]
Kawano et al. | UECFOOD-256 | Japanese Food | 25,088(256) | Captured By Mobie Cameera | 2014 | 132 | [26]
G. M. Farinella et al. | UNCIT-FD889 | Italian | 3583(899) | Acquired with a smartphone | 2014 | 66 | [97]
Meyers et al. | FOOD201-Segmented | American Fast Food | 12625 | Manually Annotated Existing Dataset | 2015 | 240 | [65]
Xin Wang et al. | UPMC Food-101 | Generic | 100,000(101) | Crawled From Web | 2015 | 111 | [82]
Cioccoa et al | UNIMB 2015 | Generic | 2000(15) | Using a Samsung Galaxy S3 smartphone | 2015 | 32 | [84]
Xu et al. | Dishes | Restaurant food datasets | 117,504(3,832) | Download from dianping | 2015 | 55 | [77]
Beijbom et al. | Menu-Match | Restaurant datasets of Italian, Asian etc. | 646(41) | Captured From social media | 2015 | 89 | [35]
Feng Zhou and Yuanqing Lin | Food-975 | Chinese Food | 37,785(975) | Collected from restaurants | 2016 | 95 | [90]
J. chen et al. | Vireo-Food 172 | Chineese Food | 110,241(172) | Downloaded from WEb | 2016 | 147 | [32]
Cioccoa et al. | UNIMB 2016 | Italian Foods | 1,027(73) | Captured From Dinning Tables | 2016 | 92 | [85]
Hui Wu et al. | Food500 | Generic | 148,408 (508) | Crawled candidate images from various search engines | 2016 | 33 | [91]
Singla et al. | Food-11 | Generic Food Dataset | 16643(11) | Collected From Other Food Datasets | 2016 | 87 |  [78].
Farinella et al. | UNCIT-FD1200 | Generic | 4754(1200) | Acquired using smartphone | 2016 | 72 | [86]
Jaclyn Rich et al. | Instagram 800k | Generic | 808,964(43) | Social Media | 2016 | 25 | [92]
Yanchao Liang and Jianhua Li | ECUSTFD | Generic | 2978(19) | Acquired using smartphone | 2017 | 164 | [93]
Güngör et al. | Turkish-Foods-15 | Turkish Dishes | 7500/15 | Collected From Other datasets | 2017 | 1 | [89]
Pandey et al, | Indian Food Database | Indian Foods | 5000(50) | Downloaded From WEb | 2017 | 26 | [42]
Termritthikun et al. | THFood-50 | Thai Foods | 700/50 | Downloaded From Web | 2017 | 9 | [47]
Ciocca et al. | FOOD524DB | Generic | 247,636(524) | Merge from existing food database | 2017 | 29 | [87]
Hou et al. | VegFru | Fruit and Vegetable Images | 160,731(292) | Collected From Search Engine | 2017 | 26 | [80]
Waltner et al. | FruitVeg-81 | Fruit and Vegetable Images | 15,630(81) | Collected usingmobile phone | 2017 | 9 | [67]
Muresan et al. | Fruits 360 Dataset | Fruit Dataset | 71,125(103) | Camera | 2018 | 80 | [79]
Qing Yu et al. | FLD-469 | Japanese Food | 209,700(469) | SmartPhone Camera | 2018 | 9 | [88]
Kaur et al. | FoodX-251 | Generic | 158,000(251) | Collected from web | 2019 | 3 | [81]
Ghalib et al. | Pakistani Food Dataset | Pakistani Dishes | 4928(100) | Crawled From Web | 2020 | 1 | [58]
Narayanan et al. | AI-Crowd Food Dataset |  | 25389 |  |  |  | [96]
Bolaños M. and Radeva P. | EgocentricFood | Generic | 5038(9) | Taken by a wearable egocentric vision camera | 2016 | 75 | [94]
E. Aguilar et al. | MAFood-121 | Spain | 21,175 | Google Search Engine | 2019 | 14 | [95]

Figure 1: System Flow Figure 2: Sample images from few food datasets

##  3 Representation of Food Images

Feature extraction plays a vital role in automated food recognition
applications due to its noticeable impact on the recognition efficiency of an
employed system. Feature extractors methods extract different food image
representations. The process of feature extraction involves the identification
of visual characteristics like colour, shape, and texture. The main objective
of feature extraction is to reduced dimensionality space [5] and extract more
manageable groups from raw vectors of food images.

Moreover, selecting the right set of features ensures that relevant
information is extracted from input images to perform the desired task. We
categorized the feature extraction techniques into two main types: Hand-
crafted and Deep visual features. The term ‘handcrafted’ refers to identifying
relevant feature vectors of appropriate objects such as shape, colour, and
texture. In contrast to that, the deep model provides state-of-the-art
performance due to automatic feature extraction through series of connected
layers. For this reason, recent studies have adopted combinations of both
hand-crafted and deep visual features for food image representation.

###  3.1 Handcrafted Features

Existing literature exhibits a large number of methods to employ manually
designed or handcrafted features. Handcrafted features are properties obtained
through algorithms using help from information available in the image. Figure
3 categorizes the handcrafted feature extraction methods.In the scenario of
food image recognition, there is a variation among different food types in
terms of texture, shape, and colour.

The term ‘texture’ refers to homogenous visual patterns that do not result
from single colours such as sky and water [7]. Texture features usually
consist of regularity, coarseness, and frequency. Texture-based
characteristics are classified into two classes, namely statistical model and
transform-based. Similarly, shape features attempt to quantify shape in ways
that agree with human intuition or aid in perception based on relative
proximity to well-known shapes. Based on the analysis, these shapes can either
be declared perceptually similar to human perception or different. Also,
extracted features should remain consistent concerning rotation, location, and
scaling (changing the object size) of an image. Unlike shape and texture
features, colour features are prevalent for image retrieval and image
classification because of their invariant property concerning image
translation, scaling, and rotation. The key items of the colour features
extraction process are colour quantization and colour space. Therefore, the
resulting histogram is only discriminative, when it projects the input image
is to the appropriate colour space. Different methods like; HSV, CIELab, RGB,
normalized RGB, opponent colour spaces, colour k-means clustering, a bag of
colour features, colour patches, and colour-based kernel, are widely employed
for food classification. Although, the colour features from the food images
distinguish between different food items. However, due to intra-class
similarity, these features alone are not enough to accurately classify food
images. For this reason, most of the researchers have used colour features in
combination with other feature extraction methods.

Hoashi et al. [16] employed Bag of features, colour histogram, Gabor features,
and gradient histogram with multiple kernel learning for automatic food
recognition of 85 different food categories. Similarly, Yang et al., [17] deal
with pairwise statistics between local features for food recognition purposes
using the PFID dataset. For real-time food image recognition Kawano and Yanai
et al., 2014 [26] utilized handcrafted features like colour, HOG, and fisher
vector. Moreover, the cloud-based food recognition method proposed by
Pouladzadeh et al., 2015 [27] involves features like colour, texture, size,
shape, and Gabor filter. They evaluated their framework on single food
portions consisting of fruit and a single item of food. Furthermore, mobile
food recognition systems proposed by Kawano and Yanai, 2013 [21], and Oliveira
et al. 2014 [31] also used handcrafted features like colour and texture. Table
2 summarizes the details of proposed methods that employ handcrafted features
for food recognition.

However, identification of food involves challenges due to varying recipes and
presentation styles used to prepare food all around the globe resulting in
different feature-sets[15]. For instance, the shape and texture of a salad
containing vegetables differ from the shape, and texture of a salad containing
fruits. For this reason, we should optimize the feature extraction process by
extracting relevant visual information from food images. Such data is present
in general information descriptors, which are a collection of visual
descriptors that provide information about primary features like shape,
colour, texture, and so forth. Some important descriptors used in existing
studies include Gabor Filter, Local Binary Patterns (LBP), Scale-invariant
Feature Transform (SIFT), and colour information to extract features of a food
image [19]. These descriptors can be applied individually or in combination
with other descriptors for enhanced accuracy.

Nonetheless, feature selection remains a complex task for food types that
involve mixed and prepared foods. Such food items are difficult to identify
and are not easily separable due to the proximity of ingredients in terms of
colour and texture features. Contrastingly, the evolution of deep learning
methods has remarkably reduced the use of handcrafted features. It is due to
their superior performance for both food categorization and ingredient
detection tasks. However, handcrafted methods for feature extraction may still
serve as the foundation for automated food recognition systems in the future.

Table 2: Handcrafted Features

Reference | Visual Features | Dataset | Recognition type
---|---|---|---
| Hoashi et al., 2010
---
[16]
| Bag-of-features(BOF),
---
Color histogram, Gabor features,
and gradient histogram
with Multiple Kernel learning.
| Used for recognition
---
of 85 food categories
| Automatic
---
food recognition
| Yang et al., 2010
---
[17]
| Deals with
---
pair wise statistics
between local features
| Pittsburgh
---
Food Image Dataset
(PFID)
Food recognition
kong and Tan, 2011 [18] |  | SIFT,
---
Guassian Region detector
| Pittsburgh Food Image Dataset (PFID)
---
and dataset consisting of
food images collected
from local restaurants.
| Regular shaped
---
foods recognition
Bosh et al. 2011 [19] |  | Global feature classes: texture and color
---
Local features: local entropy color, local color,
Garbor filter, SIFT, Haar, Daisy descriptor,
Steerable filters and Tamura perceptual filter
| Database consisting of
---
food images collected under controlled
conditions, from nutritional studies
conducted at
Prudue University [29]
| Food recognition
---
and quantification
| Zhang et al.,
---
2011 [30]
Color, SIFT, Shape, RGB histograms |  | Dataset came from online sources,
---
which includes three types of
cuisines, two dishes per cuisines were
represented by 76 images
| Classification
---
of Cuisines
| Matsuda et al.,
---
2012 [20]
| Gabor texture features,
---
Histogram of Oriented Gradient (HoG),
Bag-of-features of SIFT and CSFIT
with Spatial pyramid.
| Food image dataset
---
containing 100 different
food categories.
| Multiple
---
food images
recognition
| Kawano and Yanai,
---
2013 [21]
| Bag-of-features and Color histogram, HOG
---
patch descriptor and color patch descriptor.
- |  | Mobile food
---
recognition
| Anthimopoulos et al.
---
2014 [22]
| Bag-of-features,
---
SIFT and HSV color space
| Visual dataset consisting of 5000
---
food images organized
into 11 different classes
| Food recognition
---
system for
diabetic patients
| Tammachat and
---
Pantuwong,
2014 [23]
| Bag-of-features (BoF) , Texture
---
and Color
| Database consisting of 40 types of
---
Thai food consisting of 100 images
of each food type.
| Food image
---
recognition
| Pouladzadeh et al.,
---
2014 [24]
Graph cut, Color and Texture |  | Dataset consisting of 15 different
---
categories of fruits and food.
| Food image
---
recognition
for calorie estimation
| He et al.,
---
2014 [25]
| Color, Texture, DCD, SCD, SIFT,
---
MDSIFT, EFD and GFD
| Food image dataset containing 1453
---
images
| Food image
---
analysis
| Kawano and Yanai,
---
2014 [26]
Color, HOG and Fisher Vector | UECFOOD-256 food image dataset |  | Real-time food
---
image recognition
| Oliveira et al.,
---
2014 [31]
Color, Texture |  | Images were gathered using
---
mobile’s camera
| Mobile Food
---
Recognition
| Pouladzadeh et al.,
---
2015 [27]
Color, Texture, Size, Shape, Gabor filter |  | System was tested on single food
---
portions consisting of fruits and
single piece of food. 100 images were
chosen for training and 100 for
testing purposes.
| Cloud-based
---
food recognition.
| Farinella et al.,
---
2016 [28]
SIFT, Bag of Textons, PRICoLBP | UNICT-FD1200 dataset. |  | Food image
---
recognition

###  3.2 Deep Visual Features

Recently, deep learning techniques have gained immense attention due to their
superior performance for image recognition and classification. The deep
learning approach is a sub-type of machine learning, and it trains more
constructive neural networks. The vital operation of deep learning approaches
includes automatic feature extraction through the sequence of connected layers
leading up to a fully connected layer, which is eventually responsible for
classification. Moreover, in contrast to conventional methods, deep learning
techniques show outstanding performance while processing large datasets and
have excellent classification potential [6][7].

Deep learning methods like convolutional neural networks (CNN) [8], Deep
Convolutional Neural Networks (DCNN) [37], Inception-v3 [9] and Ensemble net
are implemented by existing food recognition methods for feature extraction.
Convolutional Neural network is one of the widely used deep learning
techniques in the area of computer vision due to its impressive learning
ability regarding visual data and achieves higher accuracy in contrast to
other conventional techniques [10]. DCNN technique gained popularity owing to
its large-scale object recognition ability. It incorporates all major object
recognition procedures like feature extraction, coding, and learning.
Therefore, DCNN is an adaptive approach for estimating adequate feature
representation for datasets [11]. Similarly, Inception-v3 is also a new deep
convolutional neural network technique introduced by Google. It is composed of
small inception modules which are capable of producing very deep networks. As
a result, this model has proved to have higher accuracy, decreased number of
parameters, and computational cost in contrast to other existing models.
Likewise, Ensemble Net is a deep CNN-based architecture and is a suitable
method for extracting features. It is due to the outstanding performance of
CNN feature descriptors as compare to handcrafted features.

Asymmetric multi-task CNN and spatial pyramid CNN [12] provides highly
discriminative image representations. Jing et al. [32] proposed ARCH-D
architecture for multi-class multilabel food recognition, and their model
provides feature vectors for both food category and ingredient recognition.
Although the feature vectors from multi-scale multi-view deep network [13] has
a very high dimension however they were successful in achieving state-of-art
performance. Ghalib et al. [58] proposed ARCIKELM for open-ended learning.
They have employed InceptionResnetV2 for feature extraction due to their
superior performance over other deep feature extraction methods such as
ResNet-50 and DenseNet201. Table 3 further provides a brief description of
deep visual features.

Table 3: Deep Visual Features

Reference | Features | Dataset | Recognition type
---|---|---|---
Kawano and Yanai, 2014 [45] | Fisher Vector and DCNN |  | UECFOOD-100 and
---
100-class food Dataset
Food Image Recognition
Yanai and Kawano, 2015 [37] | DCNN |  | UECFOOD-100
---
and UECFOOD- 256
Food Image recognition
Christodoulidis et al.., 2015 [43] | CNN |  | Manually annotated dataset
---
with 573 food items
Food Recognition
Pouladzadeh et al., 2016 [44] | Graphcut and DCNN |  | Database consisting of
---
10000 high res images
| Food Recognition for
---
Calorie Measurement
Hassannejad et al., 2016 [40] | Inception |  | Food-101, UECFOOD-100
---
and UECFOOD-256
Food Image Recognition
Liu et al., 2016 [41] | DCNN | Food-101, UECFOOD-256 | Mobile Food Image Recognition
Liu et al., 2016 [d28] | Inception | UECFOOD-256, Food -101 | Food Recognition
Chen and Ngo, 2016 [32] | Arch-D | Chinese Food |  | Ingredient recognition
---
and food categorization
Ciocca et al.., 2017 [46] | VGG | UNIMIB 2016 | Food recognition
Termritthikun et al., 2017 [47] | NU-InNet | THFOOD-50 | Food Recognition
Pandey et al., 2017[42] |  | AlexNet, GoogLeNet
---
and ResNet
| ETH Food-101 and
---
Indian Food Image Database
Food Recognition
Liu et al., 2018 [39] | GoogleNet |  | UECFOOD-100,UECFOOD-256
---
and Food-101
| Food Recognition
---
for dietary assessment
McAllister et al. 2018 [48] | ResNet-152, GoogLeNet |  | Food 5k, Food-11,RawFooT-DB
---
and Food-101
Food Recognition
Martinel et al. 2018 [49] | WISeR |  | UECFOOD-100, UECFOOD-256
---
and Food-101
Food Recognition
E. Aguilar et al., 2018 [51] | AlexNet | UNIMIB2016 | Automatic food tray analysis
S. Horiguchi et al., 2018 [56] | GoogleNet | Built their own food dataset FoodLog | Food image Recognition
Gianluigi Ciocca et al., 2018 [61] | ResNet50 | Food 475 |  | Food image recognition
---
and classification
B. Mandal et al., 2019 [57] | SSGAN | ETH Food-101 and Indian Food Dataset |  | Food Recognition
---
of Partially Labeled Data
G.Ciocca et al.,2020 [50] |  | GoogleNet, Inception-v3,
---
MobileNet-V2 and ResNet-50
| Own dataset containing 20 different
---
food categories of fruit and
vegetables.
| Food category recognition,
---
Food state recognition
L. Jiang et al., 2020 [52] | VGGNet |  | UECFOOD-100, UECFOOD-256 and
---
introduced new
dataset based on FOOD-101.
Food Recognition and Dietary Assesment
C. Liu et al.,2020 [53] | VGGNet, ResNet | Vireo-Food 172 | Food Ingredient Recognition
H. Liang et al., 2020 [54] |  | ChineseFoodNet and Vireo-Food 172 | Chinese Food Recognition
H. Zhao et al., 2020 [55] | VGGNet, ResNet and DenseNet | UECFOOD-256 and Food-101 | Mobile Food Recognition
G. A. Tahir and C. K. Loo 2020 [58] |  | ResNet-50, DenseNet201
---
and InceptionResNet-V2
| Pakistani Food Dataset, UECFOOD-100,
---
UECFOOD-256, FOOD-101 and PFID
Food Recognition
C. S. Won, 2020 [59] | ResNet50 |  | UECFOOD-256, Food-101
---
and Vireo-Food 172
| Fine grained
---
Food image Recognition
Zhidong Shen et al., 2020 [60] | Inception-v3, Inception-v4 |  | Dataset was created including
---
hundreds and thousands of images
of several food categories
| Food Recognition
---
and Nutrition Estimation

Figure 3: Handcrafted Feature Extraction Methods

##  4 Food Category Classification

The primary requirement of any food recognition system is accurate
identification and recognition of food components in the meal. Therefore,
robust and precise food classification methods are crucial for several health-
related applications like automated dietary assessment, calorie estimation,
and food journals. Image classification refers to a machine learning technique
that associates a set of unspecified objects with a subset (class) learned by
the classifier during the training phase. In the scenario of food image
classification, food images are used as input data to train the classifier.
Hence, an ideal classifier must recognize any food category explicitly
included during the learning phase. The accuracy of a classifier mainly
depends on the quantity and quality of images, as there are several variations
in food images such as rotation, distortion, lightning distribution, and so
forth. In this section, we discussed classification techniques used by
traditional approaches which use handcrafted features. Following that, we
analyzed state-of-the-art deep learning models for food recognition.

###  4.1  Traditional Machine Learning Methods

Major classifiers used by several traditional approaches in the domain of food
image recognition include Support Vector Machines (SVM)[16], Multiple Kernel
Learning (MKL) [16] and K-Nearest Neighbor (KNN)[32]. It is due to their
outstanding performance as compared to other classification methods.

Food recognition method proposed by [33] employs colour, SIFT, and texture
features to train KNN classifier. In contrast to SVM, KNN achieved higher
classification accuracy i.e. 70%, whereas the accuracy of the SVM classifier
was only 57%. Similarly, Anthimopoulos et al.,[22] applied the Bag-of-features
model using SIFT features. They also trained SVM linear image classifier to
recognize 11 different food classes and acquired a classification accuracy of
78%.

Chen et al.[34], employed a multi-class SVM classifier for the identification
of 50 different classes of Chinese food. It includes 100 food images in each
category. However, obtained classification accuracy was 62.7% only. They
further implemented a multi-class Adaboost algorithm and enhanced their
classification accuracy up to 68.3%. Furthermore, Bejibom et al., [35] used
LBP, colour, SIFT, MR8, and HOG features to train an SVM image classifier.
They evaluated their work on two different datasets and achieved a
classification accuracy of 77.4% on the dataset presented by [34], their
classification accuracy was 51.2% when applied to the menu-matched dataset.
Table 4 summarizes classifiers implemented by traditional classification
methods along with their achieved classification accuracies.

Table 4: Traditional Machine Learning Methods For Food Category Classification

Reference | Classification Technique | Classification accuracy
---|---|---
Top 1 | Top 5
Hoashi et al., 2010 [16] | Multiple Kernel Learning (MKL) |  | Own Food Dataset = 62.5%
---
N/A
Yang et al., 2010 [17] | Support Vector Machine (SVM) | PFID = 78.0% | N/A
kong and Tan, 2011 [18] | Multi-class SVM | PFID = 84% | N/A
Bosh et al. 2011 [19] | Support Vector Machine (SVM) |  | Dataset collected =86.1%
---
using nutritional
studies Conducted
at Prudue University
N/A
Zhang et al., 2011 [30] | SVM regression with RBF kernel |  | Own Food Dataset= 82.9%
---
N/A
Matsuda et al., 2012 [20] |  | Multiple Kernel Learning (MKL)
---
and Support Vector Machine (SVM)
Own food Dataset = 55.8% | N/A
Kawano and Yanai, 2013 [21] | Linear SVM and fast toookernel | N/A | 81.6%
Anthimopoulos et al. 2014 [22] | Linear SVM |  | Own Food Dataset = 78.0%
---
N/A
| Tammachat and
---
Pantuwong, 2014 [23]
Support Vector Machine (SVM) |  | Own Food Dataset =70.0%
---
N/A
Pouladzadeh et al.., 2014 [24] | Support Vector Machine (SVM) | Own Food Dataset = 95% | N/A
He et al., 2014 [25] |  | K- nearest Neighbors
---
and Vocabulary Trees
| Own Food Dataset= 64.5%
---
N/A
Kawano and Yanai, 2014 [26] | One-vs-rest | UECFOOD-256 = 50.1% | UECFOOD-256 = 74.4%
Oliveira et al. 2014 [31] | Support Vector Machine (SVM) |  | Own Food Dataset
---
Top 3 classification
achieved between 84 and 100 %
N/A
Pouladzadeh et al., 2015 [27] | Cloud-based Support Vector Machine |  | Own Food Dataset = 94.5%
---
N/A
Farinella et al., 2016 [28] | Support Vector Machine (SVM) | UNICT-FD1200 =75.74% | UNICT-FD1200 = 85.68%

###  4.2 Deep Learning Models

Deep learning approaches have gained significant attention in the field of
food recognition. It is due to their exceptional classification performance in
comparison to traditional approaches [56] [57]. Convolutional Neural Network
(CNN), Deep Convolutional Neural Network (DCNN), Ensemble Net, and
Inception-v3 are some of the most prominent techniques used by existing
methods for food image recognition purposes.

Yanai and Kawano [45] employed a deep convolutional neural network (DCNN) on
three food datasets: Food-101, UECFOOD-256, and UECFOOD-100. They explored the
effectiveness of pre-training and fine-tuning a DCNN model using 100 images
from each food category obtained from each dataset. During evaluation,
classification accuracy achieved was 78.77% for UECFOOD-100, 67.57% for
UECFOOD-256 and 70.4% for Food-101. Similarly, the study presented by [40]
implemented Inception-v3 deep network established by Google [9] on the same
datasets i.e. Food-101, UEC FOOD-100, and UECFOOD-256. Classification accuracy
achieved using fine-tuned model V3 was greater than classification accuracy of
the fine-tuned version of DCNN i.e. 88.28%, 81.45%, and 76.17% for
UECFOOD-100, UECFOOD-256, and Food-101 respectively. The food recognition
method proposed by [41] implemented CNN based approach using the Inception
model on the same three datasets.

Classification accuracy achieved was 77.4%, 76.3% and 54.7% for UECFOOD-100,
UECFOOD-256 and Food-101 respectively. Table 5 provides the overview of
existing food recognition methods based on deep learning approaches and their
classification performance.

Table 5: Deep Learning Models For Food Category Classification

Reference | Classification Technique | Classification Performance
---|---|---
Top1 | Top 5
Yanai and Kawano, 2015 [37] | DCNN |  | UECFOOD-100 =78.8%
---
UECFOOD-256 =67.6%
N/A
Christodoulidis et al.., 2015 [43] | DCNN | Own dataset = 84.9% | N/A
Chen and Ngo, 2016 [32] | DCNN |  |
Pouladzadeh et al., 2016 [44] | DCNN + Graph cut | Own dataset =99% | N/A
Hassannejad et al., 2016 [40] | DCNN |  | ETH Food-101 = 88.3%
---
UECFOOD-100 = 81.5%
UECFOOD-256 = 76.2%
| ETH Food-101 = 96.9%
---
UECFOOD-100 = 97.3%
UECFOOD-256 = 92.6%
Liu et al., 2016 [41] | CNN |  | UECFOOD-100 = 76.3%
---
Food-101 =77.4%
| UECFOOD-100 =94.6%
---
Food-101 = 93.7%
Pandey et al., 2017[42] | CNN |  | ETH-Food101 = 72.1%
---
Indian Food =73.5%
Database
| ETH-Food101 = 91.6%
---
Indian Food =94.4%
Database
Ciocca et al.., 2017[46] | CNN | UNIMIB 2016 = 78.3% | N/A
Termritthikun et al., 2017 [47] | CNN | THFOOD-50 = 69.8% | THFOOD-50 = 92.3%
McAllister et al. 2018 [48] |  | CNN+ANN+SVM+
---
Random Forest
| Food-5K =99.4%
---
Food-11 = 91.3%
RawFooT-DB = 99.3%
Food-101 = 65.0%
N/A
Liu et al., 2018 [39] | DCNN |  | UECFOOD-256 = 54.5%
---
UECFOOD-100 = 77.5%
Food 101 = 77.0%
| UECFOOD-256 = 81.8%
---
UECFOOD-100 = 95.2%
Food 101 = 94.0%
Martinel et al. 2018 [49] | DNN |  | UECFOOD-100 =89.6%
---
UECFOOD-256 =83.2%
Food-101 = 90.3%
| UECFOOD-100 =99.2%
---
UECFOOD-256 =95.5%
Food-101 = 98.7%
E. Aguilar et al., 2018 [51] | CNN+SVM | UNIMIB 2016 = 90.0% | N/A
Gianluigi Ciocca et al., 2018 [61] | CNN | Food-475 = 81.6% | Food-475 = 95.5%
S. Horiguchi et al., 2018 [56] |  | Sequential Personalized Classifier
---
(SPC) with
fixed-class and incremental
classification
| FoodLog = 40.2%
---
(t251-t300)
| FoodLog = 56.6%
---
(t251-t300)
B. Mandal et al., 2019 [57] | DCNN |  | ETH Food-101 = 75.3%
---
IndianFood Database = 85.3%
| ETH Food-101 = 93.3%
---
Indian Food Database = 95.6%
L. Jiang et al.,2020 [52] | DCNN (Faster R-CNN) | FOOD20-with-bbx = 71.7% | FOOD20-with-bbx = 93.1%
C. Liu et al.,2020 [53] |  |  |
H. Zhao et al., 2020 [55] | JDNet |  | UECFOOD-256 = 84.0%
---
FOOD-101 = 91.2%
| UECFOOD-256 = 96.2%
---
FOOD-101 = 98.8%
G. A. Tahir and C. K. Loo 2020 [58] |  | Adaptive Reduced Class
---
Incremental Kernel Extreme
Learning Machine (ARCIKELM)
| Food-101 = 87.3%
---
UECFOOD-100 = 88.7%
UECFOOD-256= 76.51%
PFID = 100%
Pakistani Food = 74.8%
N/A
C. S. Won, 2020 [59] | Three-scale CNN |  | UECFOOD-256 = 74.1%
---
Food 101 = 88.8%
Vireo-Food 172 = 91.3%
| UECFOOD-256 = 93.2%
---
Food-101 = 98.1%
Vireo-Food 172 = 98.9%
Zhidong Shen et al., 2020 [60] | CNN | Own dataset =85.0% | N/A

##  5 Food Ingredient Classification

Over the past few years, nutritional awareness among people is increasing due
to their intolerance towards certain types of food, mild or severe obesity
problems, or simply because of interest in maintaining a healthy diet. This
rise in nutritional awareness has also stirred a shift in the technological
domain, as several mobile applications facilitate people in keeping track of
their diet. However, such applications hardly offer features for automated
food ingredient recognition.

For this purpose, several proposed models use multi-label learning for food
ingredient recognition. It [101] can be defined as the prediction of more than
one output category for each input sample. Therefore, food ingredient
recognition is known as a multi-label learning problem. Marc Bolanos et al.
have deployed CNN as a multi-label predictor to discover recipes in terms of
the list of ingredients from food images [102]. Similarly, Yunan Wang et al.
[103], used multi-label learning for mixed dish recognition, as they have no
distinctive boundaries among them. Therefore labelling bounding boxes for each
dish is a challenging task. Another system proposed by Amaia Salvador et al.,
[104] regenerates recipes from provided food images along with cooking
instructions. On other hand, Jingjing Chen and Chong-Wah Ngo [32] proposed
deep architectures for food ingredient recognition and food categorization and
evaluated their proposed system on a large Chinese food dataset with highly
complex food images.

Furthermore, Table 6 provides brief information about accuracy scores of
proposed systems along with methods and dataset used.

Table 6: Proposed Methods for Food Ingredient Classification

Reference | Dataset | Method | Recall | Precision | F1
---|---|---|---|---|---
Chen et al., 2016 [32] | Vireo-Food 172 |  | Arch-D
---
(Multi-task)
- | - |  | 67.17 (Micro-F1)
---
47.18 (Macro-F1)
UECFOOD-100 |  | Arch-D
---
(Multi-task)
- | - |  | 82.06 (Micro-F1)
---
95.88 (Macro-F1)
Bolaños et al., 2017 [102] | Food-101 |  | ResNet50+
---
Ingredients 101
73.45 | 88.11 | 80.11
Recipe 5k |  | ResNet50+
---
Recipe 5k
19.57 | 38.93 | 26.05
Recipe 5k |  | Inception-v3+
---
Recipe 5k (Simplified)
42.77 | 53.43 | 47.51
Wang, Yunan, et al.; 2019 [103] | Economic Rice |  | Inception-V4* + NS
---
(multi-scale)
71.90 | 72.10 | 71.40
Economic Behoon |  | Inception-V4* + NS
---
(multi-scale)
77.60 | 68.50 | 69.70
| Salvador, Amaia, et al.
---
2019 [104]
Recipe 1M |  | CNN
---
Auto-Encoder
75.47 | 77.13 | 48.61

##  6 Existing Mobile Application for Food Recognition using these Methods

Today, several mobile applications have been developed to monitor diet and
facilitate users to choose healthier alternatives regarding food consumption.
Initially, these mobile applications were dependent on manually inputting food
items by selecting from limited food databases. Therefore, such applications
were not very reliable as they were prone to inaccuracies in dietary
assessment, mainly extending from limited exposure to numerous food
categories. With the advancement in the area of food image recognition, a
large number of mHealth applications for dietary assessment use images to
recognize food categories. For this purpose, existing mobile applications use
different combinations of traditional and deep visual feature extraction, and
classification methods for food recognition described earlier in sections III
and IV. Aizawa et al., [62] developed a mobile app ‘food log’ which uses
traditional feature extraction methods like colour, Bag of Features, and SIFT;
and uses an Adaboost classifier for classification purposes. Similarly, Ravi
et al., [66] proposed the ‘FoodCam’ application, which uses traditional
methods for feature extraction (LBP and RGB colour features) and SVM for
classification. Alternatively, Meyers et al., [65] employed a deep visual
technique (GoogleNet CNN model) for feature extraction and classification
purposes. Similarly, the Food Tracker app proposed by Jiang et al., [72] uses
a deep convolutional neural network for feature extraction and classification.
Also, G. A. Tahir and C. K. Loo [58] utilized deep visual methods like
ResNet-50, DenseNet201, and InceptionResNet-V2 for feature extraction and
Adaptive Reduced Class Incremental Kernel Extreme Learning Machine (ARCIKELM)
as classification method for their mobile application “My Diet Cam”. Table 7
summarizes existing mobile applications in terms of feature extraction and
classification methods used. Based on these deep visual method combinations,
food recognition accuracies differ for various existing mobile applications.
Therefore, apps with higher food recognition and classification accuracies
gain more popularity. These apps tend to ease the dietary assessment process.
Figure 4 shows the mobile application by Ravi et al. [66].

Table 7: Summary of Feature Extraction and Classification Methods Used by
Existing Mobile Applications

Reference |  | Application
---
Name
| Food
---
Segmentation
| Feature Extraction
---
Method
| Classification
---
Method
| Aizawa et al.,
---
2013 [62]
FoodLog | No |  | Color, SIFT and
---
Bag of Features
| Adaboost
---
Classifier
| Oliveira et al.,
---
2014 [31]
- | Yes |  | Color
---
and Texture
| Support Vector
---
Machine (SVM)
| Probst et al.,
---
2015 [64]
- | - |  | SIFT, LBP
---
and Color
Linear SVM
| Meyers et al.,
---
2015 [65]
Im2Calories | Yes | GoogleNet CNN |  | GoogleNet CNN
---
model
| Ravi et al.,
---
2015 [66]
FoodCam | No |  | HoG, LBP and
---
RGB Color
Features
Linear SVM
| Waltner et al.,
---
2017 [67]
- | Yes |  | RGB, HSV and
---
LAB Color
values
| Random Forest
---
Classifier
| Mezgec and Seljak
---
2017 [68]
- | - | NutriNet | NutriNet
| Pouladzadeh et al.,
---
2017 [69]
- | Yes | CNN |  | Caffe
---
Framework
| Waltner et al.,
---
2017 [70]
- | Yes | CNN | CNN
| Ming et al.,
---
2018 [71]
DietLens | - | ResNet-50 CNN | ResNet-50 CNN
| Jiang et al.,
---
2018 [72]
- | Yes |  | Colors, Lines,
---
Points,SIFTand
Texture Features
| Reverse Image Search
---
(RIS) and Text Mining
| Jianing Sun et al.,
---
2019 [73]
Food Tracker | Yes | DCNN | DCNN
| G. A. Tahir
---
and C.K. Loo
2020 [58]
MyDietCam | Yes |  | ResNet-50,
---
DenseNet201
and Inception
ResNet-V2
| Adaptive Reduced
---
Class Incremental
Kernel Extreme
Learning Machine
(ARCIKELM)

Figure 4: The application provides the top 5 prediction results. This picture
is taken from the study of Ravi et al. 2015

##  7 Discussion

Our research provides deep insight into computer vision-based approaches for
dietary assessment. It focuses on both traditional and deep learning
methodologies for feature extraction and classification methods used for food
image recognition. Similarly, this survey also explores and compares current
food image datasets in detail, as vision-based techniques are highly dependent
on a comprehensive collection of food images. In contrast to previous research
work, Mohammad A. Sobhi et al. [105], Min, Weiqing, et al.[106]. Our survey
scrutinizes traditional and current deep visual approaches for feature
extraction and classification to enhance clarity in terms of their performance
and feasibility. Unlike existing reviews, our survey also emphasizes existing
solutions developed for food ingredient recognition through multi-label
learning. Finally, our research study also explores real-world applications
using the prior methodologies for dietary assessment purposes.

###  7.1 Findings

Our findings indicate that the ultimate performance of traditional and deep
visual techniques depends on the type of dataset used. Therefore, it has been
observed out of 38 datasets (as shown in Table 1) explored in this survey, 3
most commonly used datasets were UECFOOD-256 [26], UECFOOD-100 [75] and
Food-101 [36]. UECFOOD-256 (25088 images and 256 classes) and UECFOOD-100
(14361 images and 100 classes of food) are Japanese food datasets consisting
of Japanese food images captured by users, whereas Food-101(101000 images and
101 classes) is an American fast food dataset containing images crawled from
several websites. However, these widely used datasets are region-specific.
Therefore, there is an immense need for generic food datasets for excluding
regional bias from experimental results. Besides this, it is also evident from
this survey that traditional machine learning methodologies for food image
recognition are replaced with deep visual techniques. As per our survey,
systems proposed after the year 2015 are mainly using deep learning
technologies for food classification purposes. It is due to their phenomenal
classification performance. Speaking of classification performance of deep
visual techniques, for food-non food classification McAllister et al. 2018
[48] (99.4%) and Pouladzadeh et al., 2016 [44] (99%) achieved the highest top
1 classification accuracy. Pouladzadeh et al., 2016 [44] used DCNN and Graph
cut on their proposed dataset, whereas McAllister et al. 2018 [48] used CNN,
ANN, SVM, and random forest on the food 5k dataset. Table 5 further compares
classification accuracies of proposed deep visual models. Recent advancements
and exceptional performance of food image classification methods have now led
researchers to explore food images from a much deeper perspective in terms of
retrieval and classification of food ingredients from food images. Therefore,
we have also explored several proposed solutions for food ingredient
recognition and classification. According to our survey, the system proposed
by Chen et al.,2016 [32] has achieved the highest F1 score i.e. 95.88%
macro-F1 and 82.06% micro-F1 using the Arch-D method on the UECFOOD-100
dataset (as shown in Table 6). Finally, food image and ingredient
classification techniques helped to provide an automatic dietary assessment
with reduced human intervention in mHealth apps. For this purpose, we have
also surveyed several mobile applications which employ deep learning methods
for dietary assessment.

###  7.2 Limitations and Future Research Challenges

Despite enhanced performance and classification accuracy, food image
recognition through vision-based approaches may continue to present
interesting future research challenges. This is because the performance of the
methodology used for food image identification is highly dependent on the
source of images in a particular food dataset. Although the growing number of
food categories are being incorporated in food image datasets like UECFOOD-256
[26], Food 85 [16], and Food201-segmented [65], there is still an immense need
for generalized, comprehensive datasets for better performance evaluation and
benchmarking. Moreover, we observed that datasets with a large number of food
images significantly positively impact classification accuracy. However,
keeping these large image datasets updated is another challenge, especially
since different types of foods are being prepared every day.

In addition to this, progressive learning during the classification phase is
vital for food image datasets due to the continuous arrival of new concepts
and domain variation within existing concepts. Similarly, developing
frameworks interpretable based on an area of interest will improve the overall
practicality of solutions in a real-world environment.

Finally, there is a need to design and develop solutions that can respond to
situations ethically. In our context, it refers to the removal of any biases
concerning region-specific food preferences. Similarly, it is also vital to
prevent any distortion in the results obtained from the automated food
recognition by ensuring a data security policy. It will help to ensure
transparency in existing models.

##  8 Conclusion

In this work, we explored a broad spectrum of vision-based methods that are
specifically tailored for food image recognition. In practice, the food
recognition process incorporates four tasks: acquiring food images from the
corresponding food datasets, feature extraction using handcrafted or deep
visual, selection of relevant extracted features, and finally, appropriate
selection of classification technique using either traditional machine
learning approach or deep learning models followed by food ingredient
classification to provide better insight of nutrient information.

Despite, impeccable performance exhibited by state-of-the-art approaches,
there exists several limitations and challenges. There is an immense need for
comprehensive datasets for benchmarking and performance evaluation of these
models, as incorporating large food image datasets improves the overall
performance. Consequently, when dealing with open-ended and dynamic food
datasets, the classifier must be capable of open-ended continuous learning.
However, existing methods have several bottlenecks which undermine the food
recognition ability when it comes to open-ended learning. Similarly, some
proposed methods are prone to catastrophic forgetting, as they tend to forget
previous knowledge extracted from images while learning new information. Such
methods work great only for fixed food image datasets. Moreover, proposed
techniques for food ingredient classification are still struggling with
performance issues when applied to prepared and mixed food items.

Therefore, there is substantial room for innovative health care and dietary
assessment applications that can integrate wearable devices with a smartphone
to revolutionize this research area. Moreover, dietary assessment systems
should address these challenges to provide better insights into effective
health maintenance and chronic disease prevention.

## Author contributions

Ghalib Ahmed Tahir was responsible for the literature search and writing the
article and approved the final version as submitted. Loo Chu Kiong contributed
to the study design, reviewed the study for intellectual content, and
confirmed the final version as submitted.

## Conflict of interest

The authors wish to confirm that there are no conflicts of interest.

## Acknowledgement

This research was supported by the UM Partnership Grant: Project No:
RK012-2019 from University of Malaya, IIRG Grant (IIRG002C-19HWB) from
University of Malaya, International Collaboration Fund for project
Developmental Cognitive Robot with Continual Lifelong Learning (IF0318M1006)
from MESTECC, Malaysia and ONRG grant (Project No.: ONRG-NICOP-
N62909-18-1-2086) / IF017-2018 from Office of Naval and Research Global, UK.

## References

  * [1] “Obesity and Overweight.” World Health Organization (WHO) https://www.who.int/news-room/fact-sheets/detail/obesity-and-overweight
  * [2] N. Martinel and C. Micheloni Classification of local eigen-dissimilarities for person re-identification IEEE Signal Process. Lett., 22 (4) (2015), pp. 455-459, 10.1109/LSP.2014.2362573
  * [3] Re-Identification in the function space of feature warps N. Martinel, A.K. Roy-Chowdhury IEEE Trans. Pattern Anal. Mach. Intell., 37 (8) (2015), pp. 1656-1669, 10.1109/TPAMI.2014.2377748
  * [4] N. Martinel, C. Micheloni and G.L. Foresti Kernelized Saliency-Based Person Re-Identification Through Multiple Metric Learning IEEE Trans. Image Process., 24 (12) (2015), pp. 5645-5658, 10.1109/TIP.2015.2487048
  * [5] G. Kumar and P. K. Bhatia, "A Detailed Review of Feature Extraction in Image Processing Systems," 2014 Fourth International Conference on Advanced Computing & Communication Technologies, Rohtak, India, 2014, pp. 5-12, doi: 10.1109/ACCT.2014.74.
  * [6] . Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning, vol. 1. MIT Press Cambridge, 2016.
  * [7] L. Deng and D. Yu, “Deep learning: methods and applications,” Foundations and Trends® in Signal Processing, vol. 7, no. 3–4, pp. 197–387, 2014
  * [8] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, p. 436, 2015.
  * [9] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on
  * [10] O. Russakovsky et al., “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
  * [11] A Krizhevsky, I Sutskever, and G E Hinton, “Imagenet classification with deep convolutional neural networks.,” in Advances in Neural Information Processing Systems, 2012
  * [12] E. Heravi, H. Aghdam, D. Puig, "Classification of Foods Using Spatial Pyramid Convolutional Neural Network," Artificial Intelligence Research and Development, vol. 288, pp. 163-168.
  * [13] J. Shuqiang and M. Weiqing and L. Linhu and L. Zhengdong, "Multi-Scale Multi-View Deep Feature Aggregation," IEEE Transactions on Image Processing, 2020.
  * [14] C. Gianluigi & N. Paolo & S. Raimondo. (2016). Food Recognition: A New Dataset, Experiments, and Results. IEEE Journal of Biomedical and Health Informatics. PP. 1-1. 10.1109/JBHI.2016.2636441.
  * [15] M. Niki & P. Claudio & M. Christian., "A supervised extreme learning committee for food recognition",Computer Vision and Image Understanding 2016\.
  * [16] H. Hoashi, T. Joutou, and K. Yanai, “Image recognition of 85 food categories by feature fusion,” in Multimedia (ISM), 2010 IEEE International Symposium on, 2010, pp. 296–301.
  * [17] S. Yang, M. Chen, D. Pomerleau, and R. Sukthankar, “Food recognition using statistics of pairwise local features,” in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, 2010, pp. 2249–2256.
  * [18] F. Kong and J. Tan, “Dietcam: Regular shape food recognition with a camera phone,” in Body Sensor Networks (BSN), 2011 International Conference on, 2011, pp. 127–132
  * [19] M. Bosch, F. Zhu, N. Khanna, C. J. Boushey, and E. J. Delp, “Combining global and local features for food identification in dietary assessment,” in Image Processing (ICIP), 2011 18th IEEE International Conference on, 2011, pp. 1789–1792
  * [20] Matsuda, H. Hoashi, and K. Yanai, “Recognition of multiplefood images by detecting candidate regions,” in Multimedia and Expo (ICME), 2012 IEEE International Conference on, 2012, pp. 25–30.
  * [21] Y. Kawano and K. Yanai, “Real-time mobile food recognition system,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2013, pp. 1–7.
  * [22] M. Anthimopoulos, L. Gianola, L. Scarnato, P. Diem, and S. G. Mougiakakou, “A food recognition system for diabetic patients based on an optimized bag-of-features model.,” IEEE J. Biomedical and Health Informatics, vol. 18, no. 4, pp. 1261–1271, 2014.
  * [23] N. Tammachat and N. Pantuwong, “Calories Analysis of Food Intake Using Image Recognition,” 2014 6th International Conference on Information Technology and Electrical Engineering (icitee), pp. 67–70, 2014
  * [24] P. Pouladzadeh, S. Shirmohammadi, and A. Yassine, “Using graph cut segmentation for food calorie measurement.,” in MeMeA, 2014, pp. 621–626.
  * [25] Y. He, C. Xu, N. Khanna, C. J. Boushey, and E. J. Delp, “Analysis of food images: Features and classification,” in Image Processing (ICIP), 2014 IEEE International Conference on, 2014, pp. 2744– 2748.
  * [26] Y. Kawano and K. Yanai, “Foodcam-256: a large-scale real-time mobile food recognition system employing high-dimensional features and compression of classifier weights,” in Proceedings of the 22nd ACM international conference on Multimedia, 2014, pp. 761–762.
  * [27] P. Pouladzadeh, S. Shirmohammadi, A. Bakirov, A. Bulut, and A. Yassine, “Cloud-based SVM for food categorization,” Multimedia. Tools Appl., vol. 74, no. 14, pp. 5243–5260, Jul. 2015.
  * [28] G. M. Farinella, D. Allegra, M. Moltisanti, F. Stanco, and S. Battiato, “Retrieval and classification of food images,” Computers in biology and medicine, vol. 77, pp. 23–39, 2016.
  * [29] M. Bosch, T. Schap, N. Khanna, F. Zhu, C. J. Boushey, and E. J. Delp, “Integrated databases system for mobile dietary assessment and analysis,” Proceedings of the 1st IEEE International Workshop on Multimedia Services and Technologies for E-health in conjunction with the International Conference on Multimedia and Expo, Barcelona, Spain, July 2011.
  * [30] Mabel Mengzi Zhang. 2011. Identifying the cuisine of a plate of food. University of California San Diego, Tech. Rep (2011).
  * [31] Luciano Oliveira, Victor Costa, Gustavo Neves, Talmai Oliveira, Eduardo Jorge, and Miguel Lizarraga. 2014. A mobile, lightweight, poll-based food identification system. Pattern Recognition 47, 5 (2014), 1941–1952.
  * [32] J. Chen and C.-W. Ngo, “Deep-based ingredient recognition for cooking recipe retrieval,” in Proceedings of the 2016 ACM on Multimedia Conference, 2016, pp. 32–41.
  * [33] F. Zhu, M. Bosch, N. Khanna, C. J. Boushey, and E. J. Delp, “Multiple hypotheses image segmentation and classification with application to dietary assessment,” IEEE Journal of Biomedical and Health Informatics, vol. 19, no. 1, pp. 377–388, 2015.
  * [34] M.-Y. Chen et al., “Automatic Chinese food identification and quantity estimation,” in SIGGRAPH Asia 2012 Technical Briefs, 2012, p. 29.
  * [35] O. Beijbom, N. Joshi, D. Morris, S. Saponas, and S. Khullar, “Menu-match: Restaurant-specific food logging from images,” in Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on, 2015, pp. 844–851
  * [36] L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101–mining discriminative components with random forests,” in European Conference on Computer Vision, 2014, pp. 446–461.
  * [37] K. Yanai and Y. Kawano, “Food image recognition using deep convolutional network with pre-training and fine-tuning,” in Multimedia & Expo Workshops (ICMEW), 2015 IEEE International Conference on, 2015, pp. 1–6
  * [38] A. Mariappan et al., “Personal dietary assessment using mobile devices,” in Computational Imaging VII, 2009, vol. 7246, p. 72460Z.
  * [39] C. Liu et al., “A new deep learning-based food recognition system for dietary assessment on an edge computing service infrastructure,” IEEE Transactions on Services Computing, vol. 11, no. 2, pp. 249–261, 2018.
  * [40] H. Hassannejad, G. Matrella, P. Ciampolini, I. De Munari, M. Mordonini, and S. Cagnoni, “Food image recognition using very deep convolutional networks,” in Proceedings of the 2nd International Workshop on M
  * [41] C. Liu, Y. Cao, Y. Luo, G. Chen, V. Vokkarane, and Y. Ma, “Deepfood: Deep learning-based food image recognition for computer-aided dietary assessment,” in International Conference on Smart Homes and Health Telematics, 2016, pp. 37–48.
  * [42] P. Pandey, A. Deepthi, B. Mandal, and N. B. Puhan, “Foodnet: recognizing foods using ensemble of deep networks,” IEEE Signal Processing Letters, vol. 24, no. 12, pp. 1758–1762, 2017.
  * [43] S. Christodoulidis, M. Anthimopoulos, and S. Mougiakakou, “Food recognition for dietary assessment using deep convolutional neural networks,” in International Conference on Image Analysis and Processing, 2015, pp. 458–465.
  * [44] P. Pouladzadeh, P. Kuhad, S. V. B. Peddi, A. Yassine, and S. Shirmohammadi, “Food calorie measurement using deep learning neural network,” in IEEE International Instrumentation and Measurement Technology Conference Proceedings (I2MTC), 2016, pp. 1–6
  * [45] Y. Kawano and K. Yanai, “Food image recognition with deep convolutional features,” in Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, 2014, pp. 589–593.
  * [46] G. Ciocca, P. Napoletano, and R. Schettini, “Food Recognition: A New Dataset, Experiments, and Results.,” IEEE J. Biomedical and Health Informatics, vol. 21, no. 3, pp. 588–598, 2017.
  * [47] C. Termritthikun, P. Muneesawang, and S. Kanprachar, “NUInNet: Thai food image recognition using convolutional neural networks on smartphone,” Journal of Telecommunication, Electronic and Computer Engineering (JTEC), vol. 9, no. 2–6, pp. 63–67, 2017.
  * [48] Patrick McAllister, Huiru Zheng, Raymond Bond, and Anne Moorhead. 2018. Combining deep residual neural network features with supervised machine learning algorithms to classify diverse food image datasets. Computers in Biology and Medicine 95 (2018), 217–233.
  * [49] Niki Martinel, Gian Luca Foresti, and Christian Micheloni. 2018. Wide-slice residual networks for food recognition. In IEEE Winter Conference on Applications of Computer Vision. 567–576.
  * [50] G. Ciocca, G. Micali and P. Napoletano, "State Recognition of Food Images Using Deep Features," in IEEE Access, vol. 8, pp. 32003-32017, 2020, doi: 10.1109/ACCESS.2020.2973704.
  * [51] E. Aguilar, B. Remeseiro, M. Bolaños and P. Radeva, "Grab, Pay, and Eat: Semantic Food Detection for Smart Restaurants," in IEEE Transactions on Multimedia, vol. 20, no. 12, pp. 3266-3275, Dec. 2018, doi: 10.1109/TMM.2018.2831627.
  * [52] L. Jiang, B. Qiu, X. Liu, C. Huang and K. Lin, "DeepFood: Food Image Analysis and Dietary Assessment via Deep Model," in IEEE Access, vol. 8, pp. 47477-47489, 2020, doi: 10.1109/ACCESS.2020.2973625.
  * [53] C. Liu, Y. Liang, Y. Xue, X. Qian and J. Fu, "Food and Ingredient Joint Learning for Fine-Grained Recognition," in IEEE Transactions on Circuits and Systems for Video Technology, doi: 10.1109/TCSVT.2020.3020079.
  * [54] H. Liang, G. Wen, Y. Hu, M. Luo, P. Yang and Y. Xu, "MVANet: Multi-Tasks Guided Multi-View Attention Network for Chinese Food Recognition," in IEEE Transactions on Multimedia, doi: 10.1109/TMM.2020.3028478.
  * [55] H. Zhao, K. -H. Yap, A. C. Kot and L. Duan, "JDNet: A Joint-Learning Distilled Network for Mobile Visual Food Recognition," in IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 4, pp. 665-675, May 2020, doi: 10.1109/JSTSP.2020.2969328.
  * [56] S. Horiguchi, S. Amano, M. Ogawa and K. Aizawa, "Personalized Classifier for Food Image Recognition," in IEEE Transactions on Multimedia, vol. 20, no. 10, pp. 2836-2848, Oct. 2018, doi: 10.1109/TMM.2018.2814339.
  * [57] B. Mandal, N. B. Puhan and A. Verma, "Deep Convolutional Generative Adversarial Network-Based Food Recognition Using Partially Labeled Data," in IEEE Sensors Letters, vol. 3, no. 2, pp. 1-4, Feb. 2019, Art no. 7000104, doi: 10.1109/LSENS.2018.2886427.
  * [58] G. A. Tahir and C. K. Loo, "An Open-Ended Continual Learning for Food Recognition Using Class Incremental Extreme Learning Machines," in IEEE Access, vol. 8, pp. 82328-82346, 2020, doi: 10.1109/ACCESS.2020.2991810.
  * [59] C. S. Won, "Multi-Scale CNN for Fine-Grained Image Recognition," in IEEE Access, vol. 8, pp. 116663-116674, 2020, doi: 10.1109/ACCESS.2020.3005150.
  * [60] Zhidong Shen, Adnan Shehzad, Si Chen, Hui Sun, Jin Liu, Machine Learning Based Approach on Food Recognition and Nutrition Estimation, Procedia Computer Science, Volume 174,2020, Pages 448-453, ISSN 1877-0509,https://doi.org/10.1016/j.procs.2020.06.113.
  * [61] Gianluigi Ciocca, Paolo Napoletano, Raimondo Schettini, CNN-based features for retrieval and classification of food images, Computer Vision and Image Understanding, Volumes 176–177, 2018, Pages 70-77, ISSN 1077-3142, https://doi.org/10.1016/j.cviu.2018.09.001.
  * [62] Aizawa, K., Maruyama, Y., Li, H., & Morikawa, C. (2013). Food balance estimation by using personal dietary tendencies in a multimedia food log. IEEE Transactions on Multimedia, 15, 2176–2185.
  * [63] Oliveira, L., Costa, V., Neves, G., Oliveira, T., Jorge, E., & Lizarraga, M. (2014). A mobile, lightweight, poll-based food identification system. Pattern Recognition, 47, 1941–1952.
  * [64] Probst, Y.; Nguyen, D.T.; Tran, M.K.; Li, W. Dietary Assessment on a Mobile Phone Using Image Processing and Pattern Recognition Techniques: Algorithm Design and System Prototyping. Nutrients 2015, 7, 6128-6138. https://doi.org/10.3390/nu7085274
  * [65] Meyers, A, Johnston, N, Rathod, V, Korattikara, A, Gorban, A, Silberman, N, … Murphy, KP (2015). Im2Calories: towards an automated mobile vision food diary. InProceedings of the IEEE International Conference on Computer Vision, 1233–1241.
  * [66] Ravì, D., Lo, B., & Yang, G.-Z. (2015). Real-time food intake classification and energy expenditure estimation on a mobile device. Wearable and implantable body sensor networks (BSN). IEEE 12th International Conference on, IEEE
  * [67] G. Waltner, M. Schwarz, S. Ladstätter, A. Weber, P. Luley, M. Lindschinger, I. Schmid, W. Scheitz, H. Bischof, L. Paletta, Personalized dietary self-management using mobile vision-based assistance, in: International conference on image analysis and processing, Springer, pp. 385–393.
  * [68] Mezgec, S., & Koroušić Seljak, B. (2017). Nutrinet: A deep learning food and drink image recognition system for dietary assessment. Nutrients, 9, 657.
  * [69] Pouladzadeh, P., & Shirmohammadi, S. (2017). Mobile multi-food recognition using deep learning. ACM Transactions on Multimedia Computing, Communications, and Applications, 13, 36
  * [70] G. Waltner, M. Schwarz, S. Ladstätter, A. Weber, P. Luley, H. Bischof, M. Lindschinger, I. Schmid, L. Paletta, Mango-mobile augmented reality with functional eating guidance and food awareness, in: International conference on image analysis and processing, Springer, pp. 425–432.
  * [71] Ming, J. Chen, Y. Cao, C. Forde, C.-W. Ngo, T. S. Chua, Food photo recognition for dietary tracking: System and experiment, in: International conference on multimedia modeling, Springer, pp. 129–141
  * [72] Jiang, H., Starkman, J., Liu, M., & Huang, M.-C. (2018). Food nutrition visualization on google glass: Design tradeoff and field evaluation. IEEE Consumer Electronics Magazine, 7, 21–31
  * [73] Sun, Jianing & Radecka, Katarzyna & Zilic, Zeljko. (2019). FoodTracker: A Real-time Food Detection Mobile Application byDeep Convolutional Neural Networks.
  * [74] A. Mariappan et al., “Personal dietary assessment using mobile devices,”in Computational Imaging VII, 2009, vol. 7246, p. 72460Z.
  * [75] Y. Matsuda and K. Yanai, "Multiple-food recognition considering co-occurrence employing manifold ranking," in Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), 2012, pp. 2017-2020.
  * [76] T. Miyazaki, G. C. de Silva and K. Aizawa, "Image-based Calorie Content Estimation for Dietary Assessment," 2011 IEEE International Symposium on Multimedia, 2011, pp. 363-368, doi: 10.1109/ISM.2011.66.
  * [77] Herranz, Luis & Xu, Ruihan & Jiang, Shuqiang., "A probabilistic model for food image recognition in restaurants," in 2015 IEEE International Conference on Multimedia and Expo (ICME), 2015, pp. 1-6.
  * [78] Singla, Ashutosh, Lin Yuan, and Touradj Ebrahimi. "Food/non-food image classification and food categorization using pre-trained googlenet model." Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management. 2016.
  * [79] Mureşan, Horea, and Mihai Oltean. "Fruit recognition from images using deep learning." arXiv preprint arXiv:1712.00580 (2017).
  * [80] Liu, Chang & Cao, Yu & Chen, Guanling & Vokkarane, Vinod & Ma, Yunsheng & Chen, Songqing & Hou, Peng. "A New Deep Learning-Based Food Recognition System for Dietary Assessment on An Edge Computing Service Infrastructure," IEEE Transactions on Services Computing, pp. 1-1, 2017.
  * [81] Kaur, Parneet, et al. "Foodx-251: a dataset for fine-grained food classification." arXiv preprint arXiv:1907.06167 (2019).
  * [82] Xin Wang, D. Kumar, N. Thome, M. Cord and F. Precioso, "Recipe recognition with large multimodal food dataset," 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2015, pp. 1-6, doi: 10.1109/ICMEW.2015.7169757.
  * [83] L. Bossard and et al., “Food-101 Mining Discriminative Components with Random Forests,” in ECCV, 2014.
  * [84] Gianluigi Ciocca, Paolo Napoletano, and Raimondo Schettini. 2015. Food Recognition and Leftover Estimation for Daily Diet Monitoring. In New Trends in Image Analysis and Processing – ICIAP 2015 Workshops. 334–341.
  * [85] Gianluigi Ciocca, Paolo Napoletano, and Raimondo Schettini. 2017. Food Recognition: a New Dataset, Experiments and Results. IEEE Journal of Biomedical and Health Informatics (2017).
  * [86] Farinella, G.M., Allegra, D., Moltisanti, M., Stanco, F., Battiato, S.: Retrieval and classification of food images. Comput. Biol. Med. 77, 23–39 (2016)
  * [87] Ciocca, G., Napoletano, P., Schettini, R., 2017b. Learning CNN-based features for retrieval of food images. In: New Trends in Image Analysis and Processing – ICIAP 2017, pp.426–434.
  * [88] Q. Yu, M. Anzawa, S. Amano, M. Ogawa and K. Aizawa, "Food Image Recognition by Personalized Classifier," 2018 25th IEEE International Conference on Image Processing (ICIP), 2018, pp. 171-175, doi: 10.1109/ICIP.2018.8451422.
  * [89] C. Güngör, F. Baltacı, A. Erdem, and E. Erdem, “Turkish cuisine:A benchmark dataset with Turkish meals for food recognition,” in Proc.25th Signal Process. Commun. Appl. Conf. (SIU), May 2017, pp. 1–4.
  * [90] Feng Zhou and Yuanqing Lin. 2016. Fine-grained image classification by exploring bipartite-graph labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1124–1133.
  * [91] Hui Wu, Michele Merler, Rosario Uceda-Sosa, and John R. Smith. 2016. Learning to make better mistakes: Semantics-aware visual food recognition. In Proceedings of the ACM on Multimedia Conference. 172–176
  * [92] Jaclyn Rich, Hamed Haddadi, and Timothy M. Hospedales. 2016. Towards bottom-up analysis of social food. In Proceedings of the International Conference on Digital Health Conference. ACM, 111–120.
  * [93] Yanchao Liang and Jianhua Li. 2017. Computer vision-based food calorie estimation: Dataset, method, and experiment. Retrieved from: arXiv preprint arXiv:1705.07632 (2017).
  * [94] Bolaños M. and Radeva P. “Simultaneous Food Localization and Recognition” In Proceedings of the 23rd International Conference on Pattern Recognition (ICPR) 2016 (IN PRESS).
  * [95] E. Aguilar, M. Bolaños, P. Radeva, Regularized Uncertainty-based Multi-Task LearningModel for Food Analysis, J. Vis. Commun. Image R. (2019), doi: https://doi.org/10.1016/j.jvcir.2019.03.011
  * [96] https://www.aicrowd.com/challenges/food-recognition-challenge
  * [97] G. M. Farinella, D. Allegra, F. Stanco, "A Benchmark Dataset to Study the Representation of Food Images", International Workshop on Assistive Computer Vision and Robotics (ACVR) 2014
  * [98] M.-Y. Chen et al., “Automatic Chinese food identification and quantity estimation,” in Proc. SIGGRAPH Asia Tech. Briefs, 2012, p. 29.
  * [99] M. Chen, K. Dhingra, W. Wu, L. Yang, R. Sukthankar and J. Yang, "PFID: Pittsburgh fast-food image dataset," 2009 16th IEEE International Conference on Image Processing (ICIP), 2009, pp. 289-292, doi: 10.1109/ICIP.2009.5413511.
  * [100] Kiyoharu Aizawa and Makoto Ogawa. Foodlog: Multimedia tool for healthcare applications. IEEE MultiMedia, 22(2):48, 2015.
  * [101] Grigorios Tsoumakas and Ioannis Katakis. Multi-label classification: An overview. International Journal of Data Warehousing and Mining, 3(3), 2006.
  * [102] Bolaños, Marc, Aina Ferrà, and Petia Radeva. "Food ingredients recognition through multi-label learning." International Conference on Image Analysis and Processing. Springer, Cham, 2017.
  * [103] Wang, Yunan, et al. "Mixed dish recognition through multi-label learning." Proceedings of the 11th Workshop on Multimedia for Cooking and Eating Activities. 2019.
  * [104] Salvador, Amaia, et al. "Inverse cooking: Recipe generation from food images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  * [105] Subhi, Mohammed Ahmed, Sawal Hamid Ali, and Mohammed Abulameer Mohammed. "Vision-based approaches for automatic food recognition and dietary assessment: A survey." IEEE Access 7 (2019): 35370-35381.
  * [106] Min, Weiqing, et al. "A survey on food computing." ACM Computing Surveys (CSUR) 52.5 (2019): 1-36.


# Reinvestigation on large perpendicular magnetic anisotropy in Fe/MgO
interface from first-principles approach

Nurul Ikhsan,1,2 Tomosato Kanagawa,1 Indra Pardede,1,3
Masao Obata,1,4 and Tatsuki Oda1,4 1Graduate School of Natural Science and
Technology, Kanazawa University, Kakuma, Kanazawa 920-1192, Japan
2School of Computing, Telkom University, Bandung 40257, Indonesia
3Department of Physics, Institut Teknologi Sumatera, Lampung Selatan 35365,
Indonesia
4Institute of Science and Engineering, Kanazawa University, Kakuma, Kanazawa
920-1192, Japan

###### Abstract

We investigated electronic structure and magnetic anisotropy in the Fe/MgO
interface of magnetic metal and dielectric insulator under the Cr layer of
small electronegativity, by means of the first-principles density functional
approach. The result indicates that the interface resonance state gets
occupied unlike a typical rigid band picture as the number of Fe layers
decreases, finding large perpendicular anisotropies in the oscillating
behavior for thickness dependence. We discuss scenarios of the two dimensional
van Hove singularity associated with flat band dispersions, and also the
accuracies of anisotropy energy in comparison with the available experimental
data.

###### pacs:

75.30.Gw, 75.70.-i, 71.15.Rf

Interface between magnetic metal and insulator has been attractive for a long
time in material physics. In particular, the interface of Fe/MgO or its family
play a key role in the magnetic tunnel junction for the development of
magnetoresistive random access memory (MRAM), being used in the typical high
performance system of Fe/MgO/Fe junctions.Yuasa2004 ; Ando2005 ; Butler2001 ;
Koo2014  These devices, currently driven by the mechanism of spin transfer
torque,Slonczewski1996 ; Berger1996  require a large ratio of tunnel
magnetoresistances for MRAM applications.

Surprisingly, such interface has been also investigated for development of the
possible MRAM which works with electric-field (EF) magnetization reversal
mechanism. Such multi-functional interface has a lot of interesting physical
properties and potential applications for magnetic and magnonic
devices.Kruglyak2010  Its unique magnetic characteristic, perpendicular
magnetic anisotropy, voltage/spin-current controlled magnetic anisotropy, also
unique spin structure with voltage controlled Dzyaloshinskii-Moriya
interaction.Nawaoka2015APEX  These features are promising for information
technology device, especially for high density and low power consumption data
storage.ShiotaNM2012

In solid state physics, the two dimensional electronic structures originated
by the van Hove singularity (vHS) of band theory has provided us many
interesting physics as well as those of other (one and three) dimensional
properties; in carbon nanotubes the optical transition between the vHS
states,CarbonNT  the high-$T_{\rm c}$ HxS superconductors, etc.Sano2016  In
the works for development of the devices, the family of Fe/MgO interfaces has
been used as a kernel technological element. They have showed a strong
perpendicular magnetic anisotropy for the thin Fe layer without any heavy
element.Ikeda2010 ; Shimabukuro2010  The multi-functional properties,
mentioned in the previous paragraphs, are mostly originated from the
electronic structure. In such system, the interface resonance state (IRS) has
been discussed in the several works.Zermatten2008  These states are
consequences of the band formation consisting of non-bonding orbitals on the
interface. Although such character has been observed as interesting
characteristics, the van Hove’s singularity(vHS) has never been pointed out so
far in terms of orbital components at the interface.

In recent improvement of computational performance allows us to estimate the
magnetic anisotropy or its EF effect precisely and numerically.Duan2008 ;
Nakamura2009  Such improvement contributes not only to physical and
qualitative explanations in the property of magnetic anisotropy, but also to
semi-quantitative agreements. In particular, the slope in the EF variation has
been proved to have a realistic meaning, when compared with the experimental
results.Nozaki2010  On the magnetic anisotropy energy (MAE) itself, the
comparison with experiment has a distance from explaining the experimental
measurements with a quantitative agreement. The experimental progresses on the
interface magnetic anisotropy in the thin films give us a fascinating
opportunity on a direct comparison between the theoretical and experimental
approaches.

This work was devoted to the discussions on electronic, magnetic, and
structural properties of Fe/MgO(001), as a reinvestigation in the view point
of two dimensional vHS. We obtained remarkable Fe-thickness dependences of
MAE, implying a picture of non-rigid band filling in the IRS. Such result can
be discussed in terms of electronic band theory, compared with the available
experimental data. In the comparison, we also discuss the computational
accuracy on MAE within the current state-of-the-art for density functional
theory (DFT).

Figure 1: Schematic diagram of the slab systems; Cr(6ML)/Fe(xML)/MgO(5ML) for
$x=5$.

In order to investigate the interface of Fe/MgO, we consider a slab system,
vacuum (0.79 nm)/Cr($w$ ML)/Fe ($x$ ML)/MgO (5 ML)/vacuum (0.79 nm)
(ML$=$atomic monolayer), in the computation (Fig. 1). The atoms are specified
by number as Cr(1), Cr(2), …, Fe (1), Fe (2), …, O (1), …, Mg (1), … etc. from
the Fe/MgO interface. At this interface an Fe atom was placed just next to the
O atom due to its stability, and in the Cr and Fe layers the body-centered
layer-stacking sequence was used.Parkin90 ; Johnson2007  The in-plane lattice
constant extracted from bulk Cr was employed. The thickness of Fe thin layer
($x$) was varied from 1 ML until 10 ML, while that of Cr ($w$) was kept at
$w=6$. The latter thickness is enough to give an constant MAE within an
accuracy of 0.1mJ/m2 for a given Fe thickness.

We carried out first-principles density functional calculations, which employ
fully relativistic (with spin-orbit interaction) and scalar-relativistic
(without spin-orbit interaction) Ultrasoft pseudopotentials and planewave
basis,Oda2005 ; Laasonen1993  by using the generalized gradient
approximation.Perdew1992  The MAE originated from spin-orbit interaction (SOI)
was estimated from the total energy difference between the different
magnetization directions [100] ($x$-axis) and [001] ($z$-axis), ${\rm
MAE(SOI)}=E[100]-E[001]$, where [001] specifies the direction of film
thickness (right-hand side in Fig. 1). We used the 32 $\times$ 32 $\times$ 1
mesh in k point samplingMonkhost76  in MAE estimation.Tsujikawa2008  Using the
scalar-relativistic level computation, in which taking a 24 $\times$ 24
$\times$ 1 k-mesh, we induced structural relaxation while keeping both the in-
plane lattice constant and the atomic coordinates of O(3). The MAE from the
shape anisotropy was estimated using the magnetostatic dipole interaction
(MDI) and assuming the atomic magnetic moments.Szunyogh1995

Figure 2: Thickness dependence of the magnetic anisotropy energy (MAE) from
spin-orbit interaction in Cr/Fe(xML)/MgO. The bullet indicates the
experimental value (ref. Koo2014, ), where 1ML thickness is assumed to be
0.142 nm.

We report the thickness dependence of MAE(SOI) in Fig. 2. It indicates an
oscillating perpendicular anisotropy with respect to Fe thickness ($x$), and
the maximum of 2.0 mJ/m2 at Fe 2ML and the maximal values at 5ML and 7ML. The
behavior shows that an odd-even alternating oscillation at the thicker systems
(4ML-10ML). For the thinner systems, the amplitude of oscillation is largely
enhanced since large changes are expected in the electronic structure around
the Fe/MgO interface. The maximum MAE at Fe 2ML is much larger than the
previous theoretical and experimental values in the Fe/MgO interface family,
comparable to the interface contribution extracted from the extrapolation
fitting in the experiment.NozakiPRA2016  At Fe 5ML our value agrees well with
the experimental value.Koo2014

Figure 3 shows the projected density of states (PDOS) at the interface in
$x=2$ and band dispersions with 3d orbital components in the vicinity of the
Fermi level ($E_{\rm F}$). As shown, the $E_{\rm F}$ is located between the
two peaks of PDOS consisting of 3d orbitals. These electronic states forms a
flat band around the k-point ${\bf k}_{1}=\pi/a(1/2,1/2)$ (see Fig. 3) and a
vHS in two-dimensional Brillouin zone (2DBZ). The vHS is more clearly shown in
$x=5$. The band flatness remarkably appears on $\bar{\rm X}$–$\bar{\rm Y}$
line in 2DBZ and the saddle point nature appears near/around the cross point
${\bf k}_{1}$ of $\bar{\rm X}$–$\bar{\rm Y}$ and $\bar{\Gamma}$–$\bar{\rm M}$
lines. These features are the origins of sharp PDOS peaks in the IRS,
appearing more or less in the Fe/MgO and its family systems. However, for
realizing these features, there may be two conditions. The one is an
appropriate orbital hybridization between 1st Fe and 2nd Fe layers. This keeps
splitting the mixed eigenstates of $d_{xz}$ and $d_{yz}$ components at
$\bar{\rm M}$ point to the lower and higher eigen energies, while in the Fe
1ML system, those stays remain on or around $E_{\rm F}$.WangPRB1993  The 2nd
condition is also an orbital hybridization between Fe $d_{3z^{2}-r^{2}}$ and O
$p_{z}$. This keeps the $d_{3z^{2}-r^{2}}$ away from $E_{\rm F}$, not
disturbing at $E_{\rm F}$. The latter has been well known as one of important
origins for realizing perpendicular anisotropy.Shimabukuro2010  This is
because the orbital $d_{3z^{2}-r^{2}}$ always contributes only to in-plane
magnetic anisotropy, assuming that the contribution from the majority spin
state can be neglected due to a large exchange splitting.WangPRB1993  The MAE
maximum in Fe 2ML is obtained as the consequences of the origins discussed
above. Note that there are vertical couplings of SOI around ${\bf k}_{1}$ area
in 2DBZ, which contribute to perpendicular anisotropy; couplings of
$d_{xy}$–$d_{x^{2}-y^{2}}$ and $d_{xz}$–$d_{yz}$.

Figure 3: Electronic structures at the Fermi level in Cr/Fe($x$ML)/MgO
($x=2,5$) without including spin-orbit interaction ; (a) projected densities
of states (PDOS) for the Fe ${\rm 3}d$ orbitals at the interface Fe(1), (b)
1st Brillouin zone, (c) band dispersions for $x=2$, (d) band dispersions for
$x=5$. The colors of grey, orange (yellow), blue, green, and red are used for
indicating the components of total 3$d$ orbital, $d_{xz+yz}$, $d_{xy}$,
$d_{x^{2}-y^{2}}$, and $d_{3z^{2}-r^{2}}$, respectively.

The thickness dependence in MAE has a relationship with the number of
electrons (NOE) in the 3d orbital of minority spin state on the interface Fe.
In the large variation range of MAE ($<$ Fe 4ML), the NOE decreases as
thickness, and in the odd-even alternating range ($\geq$ Fe 4ML) its NOE does
not synchronize so much with the MAE, as shown in the supplemental material
(Fig. S1). For the former, the number of Fe works as hole doping, while in the
latter the MAE may be influenced by the details of electronic structures at
the interface. Interestingly, note that the NOEs of $d_{xy}$ and
$d_{x^{2}-y^{2}}$ show an alternating behavior in their relative values (See
Fig. S1). Indeed, the alternating nature appears at the $d_{xy}$ component in
the band dispersions and PDOS (not shown). The Fe layer, forming a quantum
well structure, is terminated at the Fe/MgO and Cr/Fe interfaces, calling the
amplitudes of wave function at the edges have a feature of odd-even
alternating behavior in the one dimensional model chain of which the atomic
site is connected with electron transfer integral. The $d_{xy}$ component at
the Fe/MgO interface is largely affected due to its non-bonding nature, and
can be sensitive to external perturbation from the other edge of Cr/Fe
interface. The electron transfer between the $d_{xy}$ orbitals of neighboring
Fe MLs is proportional to $T_{xy,xy}({\bf k})\sim e^{i{\bf k}\cdot{\bf
R}}\\{3t(dd\sigma)+2t(dd\pi)+4t(dd\delta)\\}/9$, and is not negligible because
the absolute of transfer integral can reach to about 0.1 eV.WangPRB1993
Indeed, comparing the band dispersions between odd and even Fe MLs, the
behaviors of $d_{xy}$ component are much different, in particular, around the
$\bar{\rm X}$ point in 2DBZ at $E_{\rm F}$.

Figure 4: Projected density of states (PDOS) for the interface resonance state
in Cr/Fe(xML)/MgO; (a)–(f) for $x=1-5$ and 9, respectively. The top, middle,
bottom lows in the panels show the components of $d_{xz+yz}$, $d_{xy}$, and
$d_{x^{2}-y^{2}}$.

Figure 4 shows the series of PDOS for IRS. Except for $x=1$, the sharp peaks
appear, implying the localized nature of wave functions. It is worthy to
notice that the series of $x=2-5$ do not show a simple rigid band filling. The
shape of $d_{xy}$ component changes while that of $d_{x^{2}-y^{2}}$ is kept
without any large change. In details, the peaks of $d_{xy}$ and
$d_{x^{2}-y^{2}}$ are located just below and above the $E_{\rm F}$ level,
respectively, in Fe 2ML, and as the thickness the peaks of $d_{xz+yz}$ moves
to higher energies with that of $d_{x^{2}-y^{2}}$. Additionally, it is
interesting to see the PDOS peak sharpen in the thicker systems (see the case
of $x=9$). The origin of IRS energy shift, showing a property of smaller
electronegativity for Cr, is speculated as an orbital hybridization with the
Cr underlayer. It was observed clearly in the Fe layer with small $x$’s as a
vicinity effect of Cr, and such shift has been discussed in the
literatures.Ogura2011  Indeed, the Cr-orbital component also appears at the
IRS. Interestingly, the similar energy lowering of IRS occurs as the decrease
of in-plane lattice constant,Yoshikawa2013  inducing the modulations in MAE.

Decreasing the thickness of Fe layer to, for example, 2ML, the ferromagnetic
state may become unstable against thermal disturbance. This could be denied in
a discussion of Cr–Fe magnetic interaction. The Fe at the Cr/Fe interface has
a strong antiferromagnetic interaction of Heisenberg type. This stabilizes the
ferromagnetic state of Fe thin layer. The previous works reported large
exchange interactions of $J=69$ meV and $|J|$=59, 20 meV for Fe-Fe and Cr-Fe
nearest neighbors,Ogura2011 ; Hasegawa1997 ; Lavrentiev2011  respectively.
They could be enough to maintain the ferromagnetic states at room
temperatures. However, practically, associated with the existence of magnetic
dead layer in such system,NozakiPRA2016  Fe atom may diffuse into the Cr layer
during the fabricating process at high temperatures. Indeed, when exchanging
the Fe and Cr atoms at the Cr/Fe interface in Fe 4ML, the total energy does
not become so higher (50 meV/in-plane Fe).

The shape anisotropy is realistically important for determining the whole
magnetic anisotrpy. When we compare our MAE(SOI) with those from measurements,
the shape anisotropy needs to be estimated precisely. Here, we estimated the
shape anisotropy energy, MAE(SA), in two ways; from the MDISzunyogh1995  using
the theoretical atomic magnetic moments and the estimation
($-\mu_{0}M_{s}^{2}/2$) using the experimental saturation magnetization
($M_{s}$).NozakiPRA2016  In the former, the 3MLs of Cr attached to the Fe
layer were also taken into account. Figure 5 reports the total MAE, namely
summation of MAE(SOI) and MAE(SA), in comparison with the available
experimental data. This figure shows a good agreement with the experiment. As
shown at the inset in Fig. 5, the experimental MAE(SA) is much reduced in
absolute, compared with the theoretical one, and gets a better agreement with
the experiment on the total MAE.

In the theoretical approach we obtained the magnetic dead layer (MDL) with
0.053 nm thick at most, while the experiment clearly showed the MDL with 0.1
nm thick.NozakiPRA2016  The theoretical magnetization is larger than the
experimental one by about 25 % at the Fe 5 ML, as shown in the supplemental
material (see Fig. S2). These differences could be ascribed to an alloying
effect at the Cr/Fe interface in the experiment. This allows to draw a
suppression of the in-plane anisotropy from MDI contribution, or alternatively
an enhancement in the strength of interface perpendicular anisotropy. The
nearest neighbor pairs of Cr–Cr and Fe–Cr tends to enhances antiferromagnetic
coupling between the magnetic moments, and possibly inducing a noncollinear
magnetic structures due to a competition in magnetic couplings.Lavrentiev2011
Associated with such complexity in magnetic configuration, the MAE(SA)
modulates due to the change in MDI.

Figure 5: Total magnetic anisotropy energy (MAE), compared with the
experimental data (bullets)(ref. NozakiPRA2016, ). The open square symbol
specifies the data estimated theoretically; MAE(SOI)+MAE(MDI), and the cross
data the sum of the theoretical MAE(SOI) and the shape anisotropy MAE(SA)
estimated using the experimental magnetization. The inset shows the
theoretical MAE(MDI) (filled squares) and experimental MAE(SA) (dots). The
dashed lines are only for the guide of eyes.

In summary, we performed the arguments on the vHS in the IRS of Fe/MgO
interface and on the accuracy of MAE from the SOI, using the thickness
dependences of the Fe layer with Cr underlayer. It was found that the DFT
approach can describe the MAE with the numerical accuracy which is comparable
to the experimental thickness dependence. The vHS which appears at ${\bf
k}_{1}$ in the band dispersions are found to play the important role in the
perpendicular anisotropy. In the Fe 2ML, particularly, the flat bands
consisting of $d_{xz}$, $d_{yz}$, $d_{xy}$, $d_{x^{2}-y^{2}}$ are located just
below and above the Fermi level, which contribute to the large MAE of 2 mJ/m2.
This work shows that the potential of Fe/MgO over the previous works can be
drawn by introducing the Cr underlayer, like a proximity effect, implying a
possible materials design in the multi-functional interfaces through
underlayer.

###### Acknowledgements.

The authors thank T. Nozaki, Y. Suzuki, S. Yuasa, H. Sukegawa, M. Tsujikawa,
Y. Miura, K. Nakamura, and M. Shirai for their stimulated discussions. The
autor (T.O.) thanks A. Oshiyama for inspiring an aspect in the computational
accuracies on MAE. The first-principles calculation was performed using the
facilities of the Supercomputer Center, Institute for Solid State Physics,
University of Tokyo, Japan. This research partly used computational resources
of the K computer and other computers of the HPCI system provided by the AICS
and Information Technology Center of Nagoya University through the HPCI System
Research Project (Project ID:hp160227, hp160107). This work was partly
supported by ImPACT Program of Council for Science, Technology and Innovation
(Cabinet Office, Japan Government), and by the Computational Materials Science
Initiative (CMSI), Japan, and by Kanazawa University SAKIGAKE Project. The
author (T.O.) acknowledges the stimulated discussion in the meeting of the
Cooperative Research Project of the Research Institute of Electrical
Communication, Tohoku University (Project No. H25/B04). The author (M.O.)
acknowledges JSPS Research Fellowships for Young Scientists (Grant No.
27-4610) and the authors (N.I. and I.P.) acknowledges Japanese Government
(Monbukagakusho: MEXT) Scholarship in the Program for the Development of
Global Human Resources.

## References

  * (1) S. Yuasa, T. Nagahama, A. Fukushima, Y. Suzuki, and K. Ando, Nat. Mater. 3, 868 (2004).
  * (2) Y. Ando, T. Miyakoshi, M. Oogane, T. Miyazaki, H. Kubota, K. Ando, and S. Yuasa, Appl. Phys. Lett. 87, 142502 (2005).
  * (3) W. H. Butler, X.-G. Zhang, T. C. Schulthess, and J. M. MacLaren, Phys. Rev. B 63, 054416 (2001).
  * (4) J. W. Koo, H. Sukegawa, S. Kasai, Z. C. Wen, and S. Mitani, J. Phys. D: Appl. Phys. 47, 322001 (2014).
  * (5) J. C. Slonczewski, J. Magn. Magn. Mater. 159, L1 (1996).
  * (6) L. Berger, Phys. Rev. B 54, 9353 (1996).
  * (7) V. V. Kruglyak, S. O. Demokritov, and D. Grundler, J. Phys. D: Appl. Phys. 43, 264001 (2010).
  * (8) K. Nawaoka, S. Miwa, Y. Shiota, N. Mizuochi, and Y. Suzuki, Appl. Phys. Express 8, 063004 (2015).
  * (9) Y. Shiota, T. Nozaki, F. Bonell, S. Murakami, T. Shinjo, and Y. Suzuki, Nat. Mater. 11, 39 (2012).
  * (10) R. Saito, G. Dresselhaus, and M. S. Dresselhaus, Physical Properties of Carbon Nanotubes, (Imperial College Press, London, 1998), p. 191.
  * (11) W. Sano, T. Koretsune, T. Tadano, R. Akashi, and R. Arita, Phys. Rev. B 93, 094525 (2016).
  * (12) S. Ikeda, K. Miura, H. Yamamoto, K. Mizunuma, H. D. Gan, M. Endo, S. Kanai, J. Hayakawa, F. Matsukura, and H. Ohno, Nat. Mater. 9 , 721 (2010).
  * (13) R. Shimabukuro, K. Nakamura, T. Akiyama, and T. Ito, Physica E 42, 1014 (2010).
  * (14) P.-J. Zermatten, G. Gaudin, G. Maris, M. Miron, A. Schuhl, C. Tiusan, F. Greullet, and M. Hehn, Phys. Rev. B, 78 033301 (2008).
  * (15) C.-G. Duan, J. P. Velev, R. F. Sabirianov, Z. Zhu, J. Chu, S. S. Jaswal, and E. Y. Tsymbal, Phys. Rev. Lett. 101, 137201 (2008).
  * (16) K. Nakamura, R. Shimabukuro, Yuji Fujiwara, T. Akiyama, T. Ito, and A. J. Freeman, Phys. Rev. Lett. 102, 187201 (2009).
  * (17) T. Nozaki, Y. Shiota, M. Shiraishi, T. Shinjo, and Y. Suzuki, Appl. Phys. Lett. 96, 022506 (2010).
  * (18) S. S. P. Parkin, N. More, and K. P. Roche, Phys. Rev. Lett. 64, 2304 (1990).
  * (19) D. F. Johnson, D. E. Jiang, and E. A. Carter, Surf. Sci. 601, 699 (2007).
  * (20) T. Oda and A. Hosokawa, Phys. Rev. B 72, 224428 (2005).
  * (21) K. Laasonen, A. Pasquarello, R. Car, C. Lee, and D. Vanderbilt, Phys. Rev. B 47, 10142 (1993).
  * (22) J. P. Perdew, J. A. Chevary, S. H. Vosko, K. A. Jackson, M. R. Pederson, D. J. Singh, and C. Fiolhais, Phys. Rev. B 46, 6671 (1992).
  * (23) H. J. Monkhost and J. D. Pack, Phys. Rev. B 13, 5188 (1976).
  * (24) M. Tsujikawa, A. Hosokawa, and T. Oda, Phys. Rev. B 77, 054413 (2008).
  * (25) L. Szunyogh, B. Ujfalussy, and P. Weinberger, Phys. Rev. B 51, 9552 (1995).
  * (26) T. Nozaki, A. Koziol-Rachwal, W. Skowronski, V. Zayets, Y. Shiota, S. Tamaru, H. Kubota, A. Fukushima, S. Yuasa, and Y. Suzuki, Phys. Rev. Appl. 5, 044006 (2016).
  * (27) D. S. Wang, R. Wu, and A. J. Freeman, Phys. Rev. B 48, 15886 (1993).
  * (28) M. Ogura, H. Akai, and J. Kanamori, J. Phys. Soc. Jpn. 80, 104711 (2011).
  * (29) D. Yoshikawa, M. Obata, and T. Oda, JPS Conf. Proc. 5, 011012 (2015).
  * (30) H. Hasegawa, Phys. Rev. B., 42, 2368 (1990).
  * (31) M. Yu. Lavrentiev, R. Soulairol, Chu-Chun Fu, D. Nguyen-Manh, and S. L. Dudarev, Phys. Rev. B 84, 144203 (2011).