# Facilitated machine learning for image-based fruit quality assessment

Manuel Knott<EMAIL_ADDRESS>Fernando Perez-Cruz
<EMAIL_ADDRESS>Thijs Defraeye<EMAIL_ADDRESS>Empa,
Swiss Federal Laboratories for Materials Science and Technology, Laboratory
for Biomimetic Membranes and Textiles, St. Gallen, Switzerland  Swiss Data
Science Center, ETH Zurich and EPFL, Zurich, Switzerland  Institute for
Machine Learning, Department of Computer Science, ETH Zurich, Switzerland

###### Abstract

Image-based machine learning models can be used to make the sorting and
grading of agricultural products more efficient. In many regions, implementing
such systems can be difficult due to the lack of centralization and automation
of postharvest supply chains. Stakeholders are often too small to specialize
in machine learning, and large training data sets are unavailable. We propose
a machine learning procedure for images based on pre-trained Vision
Transformers. It is easier to implement than the current standard approach of
training Convolutional Neural Networks (CNNs) as we do not (re-)train deep
neural networks. We evaluate our approach based on two data sets for apple
defect detection and banana ripeness estimation. Our model achieves a
competitive classification accuracy equal to or less than one percent below
the best-performing CNN. At the same time, it requires three times fewer
training samples to achieve a 90% accuracy.

###### keywords:

Machine Learning , Computer Vision , Food Quality , Postharvest

##  1 Introduction

The automated quality assessment of fruit images by computer vision is
commonplace in the postharvest supply chain and is typically used for sorting
and grading. Sorting is a selection based on a single attribute, such as size
or color. Grading is the classification according to several quality
attributes such as color, size, or defects (Abbas et al., 2019). They provide
several advantages for packhouse operators and retailers in the food supply
chain. First, damaged or off-spec fresh produce is sorted out. Fruits can also
be packed better if they are sorted based on size, for example, in trays or
boxes. In addition, grading of fruits and vegetables into different categories
based on quality at harvest enables producers to sell their produce in
different price ranges for the different grades based on consumers’ needs. In
case of limited cooling capacity, producers can also choose to store only
high-value products in cold storage. Such automated sorting and grading
largely replaced manual quality estimation for fresh produce in high-income
countries due to its increased efficiency and reduced labor cost. At the same
time, algorithm-based solutions establish a common standard for visual
inspection and remove human bias (Abbas et al., 2019).

Color spectrometers are a direct and precise way to assess color-based fruit
properties such as its ripeness level (Alander et al., 2013; Das et al.,
2016). However, assessing more complex visual features – such as surface
patterns that correlate with specific defects – often requires the application
of Computer Vision based on RGB images taken with regular cameras. Early
approaches rely on image processing and manual feature crafting often combined
with shallow machine learning models (e.g. Dubey and Jalal, 2012; Prabha and
Kumar, 2015; Piedad et al., 2018; Mazen and Nashat, 2019; Santos Pereira et
al., 2018). Khan et al. (2018) combine manual features with deep features from
a pre-trained VGG neural network and classify them using a Support Vector
Machine. Later approaches directly train neural networks on smaller domain
data sets (e.g. Selvaraj et al., 2019; Lopes et al., 2022). Ismail and Malik
(2022) achieve excellent classification results for the data sets that are
also used in this research by using sophisticated image pre-processing and
neural networks. Convolutional Neural Networks (CNNs) are the established
approach for machine learning on image data and are used in numerous
applications in different fields. Training such a model from scratch is not
feasible for most use cases, as it requires a large number of labeled training
samples, enhanced computational resources, and extensive experimentation to
find the best model setup. Therefore, machine learning practitioners commonly
make use of transfer learning: Instead of starting from scratch, one uses a
model that is already pre-trained on a large general data set and finetunes it
on a smaller domain-specific data set (Ribani and Marengoni, 2019; Saranya et
al., 2021). While this procedure reduces the required amount of training data
drastically, enhanced computational resources in form of GPUs are still
required. Furthermore, the required number of labeled training samples can
still be high, especially when the task to learn is difficult. For postharvest
sorting and grading applications, retraining can be required for some
applications to account for fruit quality differences every harvest season. In
summary, CNN-based machine learning for automated sorting and grading requires
large labeled image data sets and extensive computational resources in the
form of GPUs for (re-)training models.

While large centralized fruit sorting lines readily meet these requirements,
this does often not apply to regions with decentralized food supply chains
where sorting and grading are done manually (Abbas et al., 2019).

Supply chains in remote areas or in developing countries can often be
characterized by stakeholders that are too small to specialize in high-tech
solutions. According to De-Arteaga et al. (2018) this leads to several
challenges when implementing machine learning solutions. Three of those
directly apply to computer vision for quality assessment in postharvest supply
chains:

  1. 1.

Learning with small data sets: Due to the lack of highly automated sorting
lines, images are often sourced by cold-chain operators, for example, with
smartphone cameras in a non-standardized way. This naturally limits the sizes
of data sets and the quality of the images.

  2. 2.

Learning from multiple messy data sets: To cover different crops at different
ripeness levels, images need to be taken at different times of the season or
at different stages in the supply chain. This non-centralized and non-
standardized data acquisition creates additional challenges. One must address
the risk of introducing unwanted biases when combining data from different
sources.

  3. 3.

Learning with limited computational resources: Farmers, packhouse operators,
and retailers are often too small and not specialized enough, so investment
into advanced computational resources or cloud services would be worthwhile.
By identifying machine learning solutions that can be developed on standard
hardware, the barrier to using such technologies can be lowered.

This research presents an approach focusing on the issues of “small data sets”
and “limited computational resources”. We make use of Vision Transformers
(ViTs) that are pre-trained in a self-supervised manner. We use those models
as “off-the-shelf” feature generators that translate high-dimensional and
unstructured image data into a lower-dimensional domain. Those feature
representations (“embeddings”) can then be input to simpler machine learning
models, which can be trained with minimal effort on low-cost hardware to
perform the desired classification or regression task. In this procedure,
there is no need for resource-intensive deep learning on the domain data set,
which addresses the issue of limited computational resources. We demonstrate
the competitiveness of this approach by benchmarking two different
classification tasks for estimating banana ripeness and defective apples,
respectively. In addition, we show that our method is superior when only a
small amount of labeled training data is available. Therefore, we address the
need for learning algorithms that work on limited data.

##  2 Materials and Methods

###  2.1 Data sets

This study involves data sets that address two types of visual fruit quality
attributes. The first is color-based, which often relates to a fruit’s
ripeness. Here we target banana fruits. The second is based on local features,
such as surface defects or surface spots. These traits indicate, for example,
insect damage. Here we target apple fruits. As such, our study includes one
representative data set for each of those challenges.

The banana ripeness data set from Fayoum University (Mazen and Nashat, 2019)
(short: “Fayoum Banana”) contains 273 single fruit images of bananas of
different ripeness levels with a neutral background. Even though the ripeness
degree of a banana is a continuous scale, in reality, farmers and cold store
managers grade the fruit using a scale from 1 to 7. The researchers who
provided this data set labeled it into four ordinal classes: “green” (104
samples), “yellowish-green” (48 samples), “midripen” (88 samples), and
“overripen” (33 samples). However, using ordinal classes naturally leads to
edge cases where samples can hardly be objectively associated with one class.
Consequently, this leads to a human-induced annotation error. Our own visual
examination of the images disclosed possibly inaccurate labels, especially
between the yellowish-green and midripen classes. From a machine learning
perspective, this classification problem can be considered an easy task since
the classification is mainly dependent on the average color values.

For apple fruits, the Internal Feeding Worm data set of the Comprehensive
Automation for Specialty Crops research project (Li et al., 2009) (short:
“CASC IFW”) was used. It contains 5858 images of apples with and without worm
defects at the apple surface. Images were taken from four apple cultivars on
trees from June to September and, therefore, at different ripeness levels
ranging from green to red apples. We use it for binary classification and
distinguish between “healthy” (2058 samples) and “damaged” (3800 samples)
fruits. The images have varying lighting conditions and backgrounds. The size
and location of the feature of interest — the damaged spots — varies. It can
be unremarkable and hard to differentiate from the apple’s calyx. For those
reasons, this classification problem can be considered an advanced machine
learning task.

Since we are using pre-trained models, our data sets require preprocessing to
match the resolution that was used for pre-training. The CASC IFW images were
upsampled from originally 120x120 pixels to 224x224 using bilinear
interpolation. The Fayoum Banana images were downsampled to 224x224 pixels
keeping the images’ original ratio by applying zero-padding to the height
dimension (top and bottom side). Example images for both data sets can be
found in A.

###  2.2 Machine Learning with Image Data

Images are a high-dimensional and highly unstructured type of data
representation. Traditional (shallow) machine learning models fail to
automatically capture the spatial relations of neighboring pixels in an image.
Therefore, early approaches rely on extracting hand-crafted features (e.g.,
color histograms) to make shallow models work with images. However, these
approaches are unsuitable for complex tasks and require a lot of manual work
by domain experts.

In recent years, Deep Learning models, i.e., Convolutional Neural Networks
(CNNs), have been established as the standard approach for image-based machine
learning. These models can learn complex patterns in images in a self-reliant
way. In simplified terms, a CNN for image classification consists of two high-
level building blocks (see Figure 2, top):

  1. 1.

A feature extractor — usually a combination of Convolutional and Pooling
layers — that translates an image into a lower-dimensional representation that
is easier to work with for subsequent tasks. This part contains a large number
of trainable parameters and requires large data sets to be trained on. In this
study, we will use a pre-trained feature extractor.

  2. 2.

A classifier that uses the latent features to solve the desired downstream
task (e.g., finding the most probable affiliation in a given set of classes).
It usually consists of one to only a few simple feed-forward layers and is,
therefore, fast to train.

It is common practice to combine pre-trained feature extractors with untrained
classifiers for faster convergence on new machine learning tasks. This
concept, called “transfer learning” accelerates the development of new models
significantly. However, the latent features can be hyper-optimized on the task
and/or data set the model was pre-trained on. Therefore, to get satisfying
results with a new smaller domain data set, one often needs to at least fine-
tune the feature extractor’s parameters. In that case, the required training
time is still significantly lower compared to training a model from scratch,
but advanced computational resources can still be required in the form of
GPUs. As detailed below, we can also use pre-trained feature extractors
without re-training or fine-tuning, which we target in this work. (Ribani and
Marengoni, 2019)

####  2.2.1 Vision Transformers

Figure 1: Model overview of a vision transformer. An input image is split into
patches and embedded with positional information (“patch tokens”). An extra
learnable cls-token is added. All tokens are transformed into a new sequence
by a transformer encoder. In a classification setup, an additional simple sub-
model (“classification head”) learns the class based on the transformed cls-
token. The illustration is adapted from Dosovitskiy et al. (2020).

In this research, we propose an alternative method for image classification
with small data sets. It is built upon a pre-trained Vision Transformer. The
Transformer architecture as shown in Figure 1 was originally introduced by
Vaswani et al. (2017) and has been established as the standard method for
Natural Language Processing tasks, such as machine translation or text
classification. Later variants use attention in a self-supervised way, which
enables the use of large unlabelled text corpora as training data (Devlin et
al., 2019; Radford et al., 2019). Recent advances in computer vision have
successfully adapted these models for image data with only small adaptions to
the original approach (Dosovitskiy et al., 2020; Touvron et al., 2021). Unlike
CNNs, transformers can model long-range dependencies and are not limited to
local features. Also, they scale well to large network capacities and data
sets. (Khan et al., 2021)

In the Vision Transformer (ViT) architecture, an input image is split into
patches, while each patch is encoded into an embedding representation using a
trainable linear projection. These patch embeddings are called “tokens,” and
they are the equivalent of words in a natural language context. The sequence
of flattened one-dimensional tokens is extended by positional encoding to
retain positional information. The key component of the architecture – the
Transformer Encoder – acts as a sequence-to-sequence translator and,
therefore, generates a new output sequence of identical length and shape. The
standard approach is to extend the patch tokens by an additional learnable
class (“cls”) token. In a supervised classification setup, this token
comprises the learned class-related features. This condensed vector is
translated to a class label by a simple Multilayer Perceptron111A feed-forward
neural network with a small number of hidden layers (“classification head”),
equivalent to the classifier in a CNN.

The key component in the transformer encoder is the self-attention mechanism.
Attention in machine learning describes a method of dynamically weighting
inputs based on their actual value instead of weighting them equally. This is
realized by adapting the key-query-value concept from retrieval systems. All
three components are implemented as vectors and mapped to an output which is
the weighted sum of the values. Self-attention in transformers is usually a
“scaled dot-product attention” as originally introduced by Vaswani et al.
(2017). It can be calculated as follows, while $d_{k}$ describes the number of
key dimensions:

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\dfrac{QK^{\top}}{\sqrt{d_{k}}}\right)V$
(1)

In practice, it has been shown to be beneficial to apply self-attention
multiple times with different query-key-value triplets, which is called
“Multi-Head Attention” (Vaswani et al., 2017).

####  2.2.2 Self-supervised learning of images

Self-supervised learning describes a methodology where no external labels are
required to train a model, but the training objective is generated from the
data itself in the form of a proxy task. In general, this enables training
based on large unlabelled data sets. Chen et al. (2021) present an overview
study of self-supervised vision transformers.

In this work, we use one specific self-supervised learning approach called
“DINO” (self distillation with no labels) Caron et al. (2021). It adapts the
concept of knowledge distillation, where a student model is learning from a
teacher model. Both models share the same architecture, while the teacher is
dynamically created from the exponential moving average of the student’s
parameters. During a training step, the two models receive different
augmentations of the same input image, while the learning objective is to
minimize the divergence of the two outputted probability distributions. As a
result, the student model learns the relevant information in an image in a
self-supervised way.

The DINO framework works particularly well with ViTs but can, in principle,
also be applied to other model architectures, such as CNNs. The authors have
shown that DINO ViTs explicitly capture the scene layout in the form of object
boundaries, which does neither apply to CNNs nor supervised ViTs. Further
experiments have shown that their model can achieve competitive classification
results on benchmark data sets when combined with simple linear classification
models. These properties let us hypothesize that ViTs trained with the DINO
method are particularly suitable as feature extractors. While supervised
methods might be prone to being hyper-optimized on the task they are trained
on, we will evaluate if DINO features are superior in capturing the general
task-independent information of an image. In this study, we use pre-trained
DINO ViTs as “off-the-shelf” feature generators, which means we use them on
domain data sets without retraining or fine-tuning.

###  2.3 Experimental setup

The classification experiments described in this study were implemented using
PyTorch (Paszke et al., 2019) version 1.9.1 and scikit-learn (Pedregosa et
al., 2011) version 0.24.2. To address the robustness of the reported results,
all training runs were executed at least five times with different random
initialization. For all experiments, we use the same training (64%),
validation (16%), and test (20%) splits. To evaluate how different models
perform on different data set sizes, we artificially reduced the available
training data for some runs by sampling class-balanced subsets from the
original data set using weighted random sampling. The validation and the test
split stay at their original size in those cases.

####  2.3.1 Baseline: Convolutional Neural Networks

As a baseline, we evaluate several state-of-research CNNs; all pre-trained on
the ImageNet data set (Deng et al., 2009). The models included in this study
are VGG-11 (Simonyan and Zisserman, 2014), AlexNet (Krizhevsky, 2014),
DenseNet (Huang et al., 2017), SqueezeNet (Iandola et al., 2016), and four
variations of ResNet (He et al., 2016) with different numbers of residual
blocks.

We use pre-trained weights for the feature extractor block in the CNN, and the
classifier is initialized with random weights. Two different types of transfer
learning with CNNs were examined: a) train the whole model on the domain data
set, and b) only train the classifier on the domain data set (see Figure 2 for
reference).

We trained with a batch size of 100 samples using Stochastic Gradient Descent
as an optimization method. Further, we follow well-established practices when
training neural networks: The initial learning rate and momentum are 0.0001
and 0.9, respectively. The learning rate is decreased once the model converges
towards its best state: we reduce it by a factor of 10 every time the
validation loss does not decrease for 15 subsequent epochs. To avoid
overfitting on the training data, training is stopped once the validation loss
does not decrease for 30 epochs in a row, while the final model weights are
loaded from the checkpoint where convergence started.

####  2.3.2 Dino ViTs + shallow classifiers

Figure 2: High-level comparison between the standard approach for machine
learning–based image classification and the method proposed in this work. We
use a pre-trained Vision Transformer for the task of feature extraction. The
generated CLS token can be seen as the equivalent of CNN embeddings. Since the
training of the ViT and the classifier are decoupled, we can use any shallow
model for class prediction.

Figure 2 (bottom) shows a high-level overview of our approach compared to the
baseline procedure. We use DINO ViTs that are pre-trained on ImageNet as
provided by the original authors222https://github.com/facebookresearch/dino to
generate the CLS token from an image. Different ViT configurations are tested:
We test the small (“ViT-S”, 21 million parameters, 384 token dimensions) and
the base architecture (“ViT-B”, 85 million parameters, 768 token dimensions).
Both are combined either with an 8x8 or 16x16 pixel patch size.

Based on the generated CLS token, we fit a set of the most popular shallow
machine learning models for classification tasks. The models included in this
study are K-nearest Neighbors (KNN), Logistic Regression (LR), Support Vector
Machine (SVM), Random Forest (RF), XGBoost (Chen and Guestrin, 2016), and
Multilayer Perceptron (MLP), which is a small neural network. All of those are
classical and well-established techniques in the machine learning community
and have been shown to work well on tabular data.

###  2.4 Visualizing Feature Embeddings

Even though feature embeddings generated by deep learning models have
significantly fewer dimensions than the original images, their dimensions are
still in the hundreds. To visualize the distribution of samples, we use
unsupervised machine learning methods for dimensionality reduction. Those
methods aim to decrease the number of dimensions while maintaining the most
important information. In this study, we used Principal Component Analysis
(PCA) as a linear method and UMAP (McInnes et al., 2018) from the family of
non-linear methods.

##  3 Results and Discussion

###  3.1 Classification results on the full data sets

We evaluate the predictive performance of CNNs and our proposed method in
classifying the banana fruit images. Table 1(a) comprises the test set
classification accuracies for banana ripeness classification using CNNs. The
classification works similarly well, independent of whether the entire model
or only the classification layers are trained. This is plausible since the
task is comparably simple and mainly color-based. The best classification
accuracy for CNNS (0.941) is the same as for the best ViT-based approach (see
Table 1(c)). We hypothesize that even higher accuracies for this data set for
both CNN and DINO ViT are prevented by the annotation blur and possibly
wrongly labeled data points we described in Section 2.1. It can be assumed
that all evaluated approaches can solve the task almost perfectly if labels
are accurate. However, having a certain degree of noise and inaccuracy in
labels comes with most real-world data sets.

Table 1: Test set classification accuracies for both data sets included in
this study. We show the results for CNNs ((a) and (b)) and ViTs combined with
shallow classifiers ((c) and (d)). All deep models are pretrained on ImageNet.

(a) CNN results for the Fayoum banana data set. We report both accuracies when all weights are trained and those when only the dense classification layers are trained. The  first and second best scores for each variation are highlighted. Model |  Full model |  Classifier only
---|---|---
AlexNet |  0.930 ± 0.015 |  0.941 ± 0.015
DenseNet |  0.926 ± 0.029 |  0.907 ± 0.013
ResNet-101 |  0.922 ± 0.008 |  0.893 ± 0.015
ResNet-152 |  0.922 ± 0.024 |  0.941 ± 0.020
ResNet-18 |  0.926 ± 0.013 |  0.904 ± 0.024
ResNet-50 |  0.933 ± 0.021 |  0.907 ± 0.000
SqueezeNet |  0.915 ± 0.021 |  0.933 ± 0.010
VGG |  0.904 ± 0.027 |  0.885 ± 0.020

(b) CNN results for the CASC IFW apple data set. We report both accuracies when all weights are trained and those when only the dense classification layers are trained. The  first and second best scores for each variation are highlighted. Model |  Full model |  Classifier only
---|---|---
AlexNet |  0.938 ± 0.004 |  0.866 ± 0.001
DenseNet |  0.949 ± 0.003 |  0.847 ± 0.004
ResNet-101 |  0.950 ± 0.003 |  0.850 ± 0.002
ResNet-152 |  0.944 ± 0.004 |  0.859 ± 0.001
ResNet-18 |  0.944 ± 0.003 |  0.842 ± 0.002
ResNet-50 |  0.949 ± 0.003 |  0.846 ± 0.005
SqueezeNet |  0.952 ± 0.002 |  0.879 ± 0.004
VGG |  0.958 ± 0.002 |  0.872 ± 0.005

(c) Results for the Fayoum banana data set of shallow classifiers based on embeddings of two different DINO ViTs (base and small model). The transformers are never trained on the domain data set. The  first and second best score are highlighted. Transformer → |  DINO ViT-B/8 |  DINO ViT-S/8
---|---|---
Classifier ↓ |  |
KNN |  0.922 ± 0.008 |  0.904 ± 0.015
LR |  0.930 ± 0.008 |  0.900 ± 0.010
MLP |  0.933 ± 0.010 |  0.915 ± 0.010
RF |  0.937 ± 0.010 |  0.915 ± 0.021
SVM |  0.919 ± 0.017 |  0.904 ± 0.008
XGBoost |  0.893 ± 0.024 |  0.941 ± 0.030

(d) Results for the CASC IFW apple data set of shallow classifiers based on embeddings of two different DINO ViTs (base and small model). The transformers never trained on the domain data set. The  first and second best score are highlighted. Transformer → |  DINO ViT-B/8 |  DINO ViT-S/8
---|---|---
Classifier ↓ |  |
KNN |  0.873 ± 0.002 |  0.903 ± 0.004
LR |  0.917 ± 0.002 |  0.917 ± 0.003
MLP |  0.934 ± 0.002 |  0.930 ± 0.004
RF |  0.872 ± 0.005 |  0.879 ± 0.002
SVM |  0.950 ± 0.000 |  0.936 ± 0.001
XGBoost |  0.925 ± 0.003 |  0.910 ± 0.004

Table 1(b) shows the CNN results for the apple fruit data set. The
classification accuracy is significantly higher when training the full model
compared to when only training the classification layers. This increased
accuracy is likely the case because we are looking for specific and local
features in the image which are not captured by the pre-trained model.

As shown in Table 1(d), the best ViT-based model (a Support Vector Classifier
based on DINO ViT-B/8 embeddings) performs less than one percent worse than
the best CNN (0.950 vs. 0.958). At the same time, it significantly outperforms
CNNs in terms of accuracy, where only the classification layers are trained.
These are remarkable results since we did not train the ViT on the domain data
set. It shows that the pre-trained DINO ViT can capture domain-specific and
local features without having seen comparable images before. Consequently, for
advanced classification tasks, DINO ViTs could either achieve similar results
with less training effort (compared to fully trained CNNs) or better results
with comparable training effort (compared to CNNs where only the classifier is
trained).

These results evoke the impression that DINO ViT embeddings are generally more
accurate than supervised CNN embeddings in capturing the essential features of
images when models are pre-trained on general-purpose data sets. To confirm
this claim, we ran another set of experiments where we combined the best
models from previous results with various pre-trained ViTs and CNNs, including
a DINO-trained ResNet-50 for comparison. The results are shown in Table 2.
Again, the best accuracies are achieved by DINO ViT-based configurations. This
implies that the improved accuracy stems from the feature quality rather than
from a possibly superior classifier.

It is worth mentioning that previous machine learning research using the same
two data sets achieves higher overall accuracies (Ismail and Malik, 2022). The
main difference in their approach is sophisticated and highly optimized image
pre-processing. We are not benchmarking our results with theirs, as this
research focuses on implementation simplicity rather than maxing out
predictive performance for the given data sets.

Table 2: Test set classification accuracies of the best shallow classifier from previous experiments in combination with feature embeddings from both ViTs and CNNs. The  first and second best score for each data set are highlighted. Data set | Fayoum Banana | CASC IFW
---|---|---
Classifier → | XGBoost | SVM
Embeddings ↓ |  |
DINO ViT-B/8 | 0.893 ± 0.024 |  0.950 ± 0.000
DINO ViT-S/8 |  0.941 ± 0.030 | 0.936 ± 0.001
DINO ViT-B/16 | 0.919 ± 0.010 | 0.947 ± 0.002
DINO ViT-S/16 | 0.896 ± 0.025 | 0.909 ± 0.002
DINO ResNet-50 | 0.889 ± 0.029 | 0.893 ± 0.005
ResNet-18 | 0.870 ± 0.029 | 0.863 ± 0.003
ResNet-50 | 0.911 ± 0.008 | 0.869 ± 0.003
ResNet-101 | 0.930 ± 0.030 | 0.889 ± 0.003
ResNet-152 | 0.926 ± 0.029 | 0.882 ± 0.004
VGG | 0.874 ± 0.024 | 0.836 ± 0.004

###  3.2 Classification results with reduced training data

The fact that we need to train any deep neural network on our domain data sets
to achieve good results naturally leads to the hypothesis that our approach
works better on small data sets than CNNs. To test this hypothesis, we chose
the best-performing models from previous experiments to examine their
performance when training data is scarce. We trained the models on randomly
sampled subsets of the training split while the validation and test set stayed
constant and were the same as in previous runs. Figure 3 shows the models’
performances as a function of the available training data using the CASC IFW
apple data set. We can observe that DINO ViT-based classifiers perform
significantly better on small data sets. A test accuracy higher than 0.9 can
be achieved with only 500 samples, while the two CNNs need at least 1500
samples to reach that benchmark. The same experiment for the Fayoum Banana
data set can be found in B. However, the results are harder to interpret since
downsampling an already very small data set leads to sample sizes where all
models perform equally poorly.

Figure 3: Test accuracies on the CASC IFW data set for different numbers of
available training samples. Error bars represent the standard deviation of
experimental results w.r.t. random initialization. The ViT approach performs
significantly better when training data is scarce. $90\%$ accuracy is reached
with three times less training data compared to the best-performing CNNs.

###  3.3 Embedding visualization

This section examines the differences between CNN and DINO ViT embeddings by
visualizing them in one or two dimensions using dimensionality reduction
methods. Subsequently, we only show the results based on PCA. The reader is
encouraged to explore visualizations of further principal components and
visualizations based on the UMAP algorithm in C. It is important to note that
dimensionality reduction is done unsupervised, which means that the
information on class labels is added a posteriori to the visualizations. As
representatives for CNNs, we chose a pre-trained VGG network since it is one
of the strongest models for our data sets from previous experiments and
ResNet-50 since it is a popular and widely used model. These models are
compared to a ViT-S/8 and a ViT-B/8 model. Figure 4 shows the distribution of
banana samples in two dimensions. Although low-dimensional CNN embeddings
cluster the classes to a certain degree, there is a substantial overlap
between them. A closer look at VGG embeddings shows that the model captures
the orientation of bananas in the first PCA dimension and divides them into
two classes (negative and positive x). On the other hand, DINO ViT embeddings
show a clearer separation of clusters and, at the same time, capture the
ordinal character of ripeness classes. The largest overlap exists between the
“yellowish green” and the “midripen” class, which can be traced back to the
aforementioned imprecise labeling. One reason for this clear class separation
is that there are not many signals in the image apart from the fruit’s color
and shape since lighting conditions and background are constant.

Figure 4: Latent representation of the Fayoum banana ripeness data set in two
dimensions (reduced by PCA) for ResNet50 (top-left), VGG-11 (top-right), DINO
ViT-S/8 (bottom-left), and DINO ViT-B/8 (bottom-right). All models are pre-
trained on ImageNet, and there was no training on the domain data set. The ViT
models seem superior in capturing the ordinal property of the four classes.
Overlaps between the classes can be traced back to imprecise labeling.

For reasons of clarity, we only show the distribution of the first principal
component of apple image embeddings in Figure 5. The first principal component
of CNN embeddings generates similar distributions for healthy and damaged
apples. We hypothesize that more dominant features, such as the fruit’s color,
changes in spectra, or background signals, are more salient in low dimensions.
Further analysis of the second and third components (see C) indicates that
local defects are captured there. However, the separation of the classes is
rather marginal. In comparison, one can observe a clear distribution shift of
the healthy/damaged classes using the first PC of DINO ViT embeddings. This
observation allows two possible interpretations: First, DINO ViT embeddings
seem more robust against noise, such as lighting and background. Second, these
embeddings lead to a higher degree of linearization in the embedding space. In
any case, these visualizations support the findings of the classification
experiments: DINO ViT features are a superior low-dimensional representation
of images compared to CNNs when pre-trained models are being used as off-the-
shelf feature generators.

Figure 5: Latent representation of the CASC IFW apple defect data set in one
dimension (reduced by PCA) for ResNet50 (top-left), VGG-11 (top-right), DINO
ViT-S/8 (bottom-left), and DINO ViT-B/8 (bottom-right). All models are pre-
trained on ImageNet, and there was no training on the domain data set. Even
though defects are a non-dominant visual feature, there is a notable shift in
the distribution of the first PCA component of ViT embeddings, especially for
DINO ViT-8.

##  4 Conclusion and Outlook

We presented a simplified development procedure for image-based machine
learning for visual fruit quality assessment. It is particularly suitable for
domains with low availability of both data and computational resources.

Our main contribution is the facilitation of the development process of the ML
model compared to current state-of-research approaches. We show how effective
machine learning models can be established with minimal development effort on
a standard computer without GPU access. In addition, our approach maintains a
high classification accuracy with small training data sets. This can
democratize the use of machine learning in decentralized food supply chains.
Potential users are cold store operators or companies serving farmer-producer
organizations, who often rely on smartphone images for fruit quality
assessment. This research contributes toward “Green AI” which aims to make AI
systems more efficient from a computational standpoint and more inclusive for
researchers and engineers worldwide by not being dependent on the availability
of enhanced computational resources (Schwartz et al., 2020).

Future research could focus on validating our findings on more difficult
classification tasks. We do not address whether and how far fine-tuning pre-
trained Vision Transformers could outperform current state-of-research methods
in achieving the highest possible predictive performance. Many food quality
applications do not rely on image labels but require the localization and/or
quantification of defects. Future studies could address this by adapting the
presented approach to object detection or image segmentation tasks. Our
visualizations of feature embeddings using PCA underline the high
expressiveness of DINO ViT embeddings. This is likely a result of the relevant
features of images being largely linearized in the embedding space. This
property indicates that DINO ViT embeddings may be particularly suitable for
integration with other data sources. One could combine these embeddings (or
some principal components) with other data types in a common tabular form.
This could be, for instance, sensor or meta data from food supply chains to
predict the end quality at arrival.

## CRediT authorship contribution statement

Manuel Knott: Conceptualization, Methodology, Software, Formal analysis,
Investigation, Writing – Original Draft, Visualization. Fernando Perez-Cruz:
Conceptualization, Methodology, Resources, Writing – Review & Editing,
Supervision, Funding acquisition. Thijs Defraeye: Conceptualization,
Resources, Writing – Review & Editing, Supervision, Project administration,
Funding acquisition.

## Declaration of competing interests

The authors declare that they have no known competing financial interests or
personal relationships that could have influenced the work reported in this
paper.

## Acknowledgments

This work was funded partially by the data.org Inclusive Growth and Recovery
Challenge grant “Your Virtual Cold Chain Assistant”, supported by The
Rockefeller Foundation and the Mastercard Center for Inclusive Growth. The
funder was not involved in the study design, collection, analysis,
interpretation of data, the writing of this article or the decision to submit
it for publication. This manuscript has been released as a pre-print at arXiv.
We acknowledge the support of Daniel Onwude for critical commenting on the
work.

## Supplementary materials

The source code of all experiments can be accessed on
Github333https://github.com/manuelknott/DINO-ViT_fruit_quality_assessment.

## References

  * Abbas et al. (2019) Abbas, H.M.T., Shakoor, U., Khan, M.J., Ahmed, M., Khurshid, K., 2019.  Automated sorting and grading of agricultural products based on image processing, in: Proceedings of the 2019 8th International Conference on Information and Communication Technologies (ICICT), Institute of Electrical and Electronics Engineers Inc.. pp. 78–81.  doi:10.1109/ICICT47744.2019.9001971.
  * Alander et al. (2013) Alander, J.T., Bochko, V., Martinkauppi, B., Saranwong, S., Mantere, T., 2013.  A Review of Optical Nondestructive Visual and Near-Infrared Methods for Food Quality and Safety.  International Journal of Spectroscopy 2013, 1–36.  URL: https://www.hindawi.com/journals/ijs/2013/341402/, doi:10.1155/2013/341402.
  * Caron et al. (2021) Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A., 2021.  Emerging properties in Self-Supervised Vision Transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650–9660.  doi:10.1109/iccv48922.2021.00951.
  * Chen and Guestrin (2016) Chen, T., Guestrin, C., 2016\.  Xgboost: A Scalable Tree Boosting System, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794.  doi:10.1145/2939672.2939785.
  * Chen et al. (2021) Chen, X., Xie, S., He, K., 2021.  An Empirical Study of Training Self-Supervised Vision Transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, Institute of Electrical and Electronics Engineers (IEEE). pp. 9640–9649.  doi:10.1109/iccv48922.2021.00950.
  * Das et al. (2016) Das, A.J., Wahi, A., Kothari, I., Raskar, R., 2016\.  Ultra-portable, wireless smartphone spectrometer for rapid, non-destructive testing of fruit ripeness.  Scientific Reports 6, 32504\.  URL: http://www.nature.com/articles/srep32504, doi:10.1038/srep32504.
  * De-Arteaga et al. (2018) De-Arteaga, M., Herlands, W., Neill, D.B., Dubrawski, A., 2018\.  Machine Learning for the Developing World.  ACM Transactions on Management Information Systems 9, 1–14.  doi:10.1145/3210548.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009\.  Imagenet: A large-scale hierarchical image database, in: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Ieee. pp. 248–255.  doi:10.1109/cvpr.2009.5206848.
  * Devlin et al. (2019) Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Stroudsburg, PA, USA. pp. 4171–4186.  doi:10.18653/v1/N19-1423.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N., 2020\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  arXiv preprint doi:10.48550/arXiv.2010.11929.
  * Dubey and Jalal (2012) Dubey, S.R., Jalal, A.S., 2012\.  Detection and classification of apple fruit diseases using complete local binary patterns, in: Proceedings of the 2012 3rd International Conference on Computer and Communication Technology (ICCCT), pp. 346–351.  doi:10.1109/ICCCT.2012.76.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep Residual Learning for Image Recognition, in: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778.  doi:10.1109/cvpr.2016.90.
  * Huang et al. (2017) Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017.  Densely Connected Convolutional Networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700–4708.  doi:10.1109/CVPR.2017.243.
  * Iandola et al. (2016) Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K., 2016\.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size.  doi:10.48550/ARXIV.1602.07360. publisher: arXiv Version Number: 4.
  * Ismail and Malik (2022) Ismail, N., Malik, O.A., 2022\.  Real-time visual inspection system for grading fruits using computer vision and deep learning techniques.  Information Processing in Agriculture doi:10.1016/j.inpa.2021.01.005. publisher: Elsevier.
  * Khan et al. (2018) Khan, M.A., Akram, T., Sharif, M., Awais, M., Javed, K., Ali, H., Saba, T., 2018.  CCDF: Automatic system for segmentation and recognition of fruit crops diseases based on correlation coefficient and deep CNN features.  Computers and Electronics in Agriculture 155, 220–236.  doi:10.1016/j.compag.2018.10.013. publisher: Elsevier.
  * Khan et al. (2021) Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M., 2021\.  Transformers in Vision: A Survey.  ACM Computing Surveys (CSUR) doi:10.1145/3505244. publisher: ACM PUB27 New York, NY.
  * Krizhevsky (2014) Krizhevsky, A., 2014.  One weird trick for parallelizing convolutional neural networks.  arXiv preprint doi:arXiv:1404.5997.
  * Li et al. (2009) Li, G., Holguin, G., Park, J., Lehman, B., Hull, L., Jones, V., 2009.  The CASC IFW Database.  URL: https://engineering.purdue.edu/RVL/Database/IFW/index.html.
  * Lopes et al. (2022) Lopes, J.F., da Costa, V.G.T., Barbin, D.F., Cruz-Tirado, L.J.P., Baeten, V., Barbon Junior, S., 2022\.  Deep computer vision system for cocoa classification.  Multimedia Tools and Applications 81, 41059–41077.  URL: https://link.springer.com/10.1007/s11042-022-13097-3, doi:10.1007/s11042-022-13097-3.
  * Mazen and Nashat (2019) Mazen, F.M.A., Nashat, A.A., 2019\.  Ripeness Classification of Bananas Using an Artificial Neural Network.  Arabian Journal for Science and Engineering 44, 6901–6910.  doi:10.1007/s13369-018-03695-5. publisher: Springer.
  * McInnes et al. (2018) McInnes, L., Healy, J., Saul, N., Großberger, L., 2018\.  UMAP: Uniform Manifold Approximation and Projection.  Journal of Open Source Software 3, 861.  doi:10.21105/joss.00861.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S., 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  Advances in Neural Information Processing Systems (NeurIPS) 32, 8026–8037.
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., 2011\.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research 12, 2825–2830.  Publisher: JMLR. org.
  * Piedad et al. (2018) Piedad, E., Larada, J.I., Pojas, G.J., Ferrer, L.V.V., 2018\.  Postharvest classification of banana (Musa acuminata) using tier-based machine learning.  Postharvest Biology and Technology 145, 93–100.  doi:10.1016/J.POSTHARVBIO.2018.06.004. publisher: Elsevier.
  * Prabha and Kumar (2015) Prabha, D.S., Kumar, J.S., 2015\.  Assessment of banana fruit maturity by image processing technique.  Journal of Food Science and Technology 52, 1316–1327.  doi:10.1007/s13197-013-1188-3. publisher: Springer.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., 2019\.  Language models are unsupervised multitask learners.  OpenAI blog URL: https://openai.com/blog/better-language-models/.
  * Ribani and Marengoni (2019) Ribani, R., Marengoni, M., 2019\.  A Survey of Transfer Learning for Convolutional Neural Networks, in: 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T), IEEE. pp. 47–57.  doi:10.1109/SIBGRAPI-T.2019.00010.
  * Santos Pereira et al. (2018) Santos Pereira, L.F., Barbon, S., Valous, N.A., Barbin, D.F., 2018\.  Predicting the ripening of papaya fruit with digital imaging and random forests.  Computers and Electronics in Agriculture 145, 76–82.  URL: https://linkinghub.elsevier.com/retrieve/pii/S016816991731030X, doi:10.1016/j.compag.2017.12.029.
  * Saranya et al. (2021) Saranya, N., Srinivasan, K., Kumar, S.K., 2021.  Banana ripeness stage identification: a deep learning approach.  Journal of Ambient Intelligence and Humanized Computing 13, 4033–4039.  URL: https://link.springer.com/article/10.1007/s12652-021-03267-w, doi:10.1007/S12652-021-03267-W/TABLES/4. publisher: Springer Science and Business Media Deutschland GmbH.
  * Schwartz et al. (2020) Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O., 2020\.  Green AI.  Communications of the ACM 63, 54–63.  doi:10.1145/3381831. publisher: ACM New York, NY, USA.
  * Selvaraj et al. (2019) Selvaraj, M.G., Vergara, A., Ruiz, H., Safari, N., Elayabalan, S., Ocimati, W., Blomme, G., 2019.  AI-powered banana diseases and pest detection.  Plant Methods 15, 1–11.  doi:10.1186/s13007-019-0475-z. publisher: BioMed Central.
  * Simonyan and Zisserman (2014) Simonyan, K., Zisserman, A., 2014\.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint doi:10.48550/arXiv.1409.1556.
  * Touvron et al. (2021) Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H., 2021\.  Training data-efficient image transformers & distillation through attention.  Proceedings of the 38th International Conference on Machine Learning (PMLR) 139, 10347–10357.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017\.  Attention is All you Need, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 6000–6010.

##  Appendix A Example Images

(a) Example of the Fayoum banana ripeness data set (Mazen and Nashat, 2019),
where three images of each class are depicted.

(b) Example of the CASC IFW apple damage data set, where nine images per class
are depicted.

Figure 1: Example images from the two data sets used in this research.

##  Appendix B Sample size experiments for Fayoum data set

Figure 1: Test accuracies on the Fayoum Banana data set for different numbers
of available training samples. Error bars represent the standard deviation of
experimental results w.r.t. random initialization. In Contrast to the results
shown in Figure 3, the ViT-based approach is not superior with a low number of
samples.

##  Appendix C Additional Embedding Visualizations

Figure 1: Latent representation of the Fayoum banana ripeness data set in two
dimensions (reduced by UMAP) for Resnet50 (top-left), VGG (top-right), DINO
ViT-S/8 (bottom-left), and DINO ViT-B/8 (bottom-right). All models are pre-
trained on ImageNet and there was no training on the domain data set. Figure
2: Second Principal Component (PC1) of the latent representations generated by
different models for the CASC IFW apple defect data set. See Figure 5 for
further explanation. Figure 3: Third Principal Component (PC2) of the latent
representations generated by different models for the CASC IFW apple defect
data set. See Figure 5 for further explanation. Figure 4: Latent
representation of the CASC IFW apple defect data set in two dimensions
(reduced by UMAP) for Resnet50 (top-left), VGG (top-right), DINO ViT-S/8
(bottom-left), and DINO ViT-B/8 (bottom-right). Data points are omitted for
clarity. All models are pre-trained on ImageNet and there was no training on
the domain data set.


11institutetext: Department of Artificial Intelligence Convergence, Chonnam
National University Gwangju 61186, South Korea 22institutetext: Corresponding
author: 22email<EMAIL_ADDRESS>
# Affective Behavior Analysis using Action Unit Relation Graph and Multi-task
Cross Attention

Dang-Khanh Nguyen 11 Sudarshan Pant 11 Ngoc-Huynh Ho 11 Guee-Sang Lee 11 Soo-
Huyng Kim 11 Hyung-Jeong Yang 1122

###### Abstract

Facial behavior analysis is a broad topic with various categories such as
facial emotion recognition, age, and gender recognition. Many studies focus on
individual tasks while the multi-task learning approach is still an open
research issue and requires more research. In this paper, we present our
solution and experiment result for the Multi-Task Learning challenge of the
Affective Behavior Analysis in-the-wild competition. The challenge is a
combination of three tasks: action unit detection, facial expression
recognition, and valance-arousal estimation. To address this challenge, we
introduce a cross-attentive module to improve multi-task learning performance.
Additionally, a facial graph is applied to capture the association among
action units. As a result, we achieve the evaluation measure of $128.8$ on the
validation data provided by the organizers, which outperforms the baseline
result of $30$.

###### Keywords:

multi-task learning, cross attention, action unit detection, facial expression
recognition, valence and arousal estimation, graph convolution network

##  1 Introduction

In affective computing, emotion recognition is a fundamental research topic
and our face is an obvious indicator to analyze the human affect. With the
development of computer vision and deep learning, there are numerous studies
and modern applications related to facial behavior analysis [10], [8], [6].
The ABAW 4th Workshop organized a competition with two challenges which are
the multi-task learning (MTL) challenge involving the development of a multi-
task model using facial images [13] and the learning from synthetic data (LSD)
challenge involving the use of synthetic data [7], [12]. We only participated
in the MTL challenge and the LSD challenge is beyond the scope of this work.

The MTL challenge aims to design a model performing the following three tasks
with facial image data as input:

  1. 1.

Action unit detection (AU detection): a task involving a multi-label
classification with 12 classes of action units that represent various
movements on the subject’s face.

  2. 2.

Facial expression recognition (FER): a multi-class classification task, which
involves identifying the emotion of the subjects among 8 categories:
happiness, sadness, anger, fear, surprise, disgust, neutral and other state.

  3. 3.

Valance Arousal estimation (VA estimation): a regression task, which involves
estimating the valence and arousal. Arousal labels are numeric representations
of the excitement levels of the subject, while valence labels represent the
degree of positive or negative feelings. The output is two continuous values
in the range $[-1,1]$.

This paper proposes utilizing an attention mechanism for MTL problem. By
attending to the output of one specific task, the model can learn to boost the
result of other related tasks. In addition, we leverage the graph-based neural
network to learn the relation among action units (AUs) in the AU detection
task.

##  2 Related Work

Based on the VGG-Face, Kollias [13] devised a multi-task CNN network to
jointly learn the VA estimation, AU detection, and FER. The MT-VGG model was
created by adapting the original VGG-Face for multi-tasking purposes with
three different types of outputs. The author also proposed the recurrent
version to adapt to temporal affect variations and the multi-modal version to
exploit the audio information.

Savchenko [16] introduced a multi-head model with a CNN backbone to resolve
the facial expression and attributes recognition. The model was sequentially
trained with typical face corpora [18], [2], [15] for various facial analysis
tasks. In ABAW 3rd competition, Savchenko [17] also developed a lightweight
model using EfficientNet [19] to effectively learn the facial features and
achieved the top 3 best performances in the MTL challenge.

Kollias [9] showed an association between emotions and AUs distribution. Each
emotion has its prototypical AUs, which are always active along with it; and
observational AUs, which are frequently present with the emotion at a certain
rate. From this assumption, the authors proposed co-annotation and
distribution matching to couple the emotions and AUs. Luo [14] introduced a
graph-based method with multi-dimensional edge features to learn the
association among AUs. The AU detection block in our model is inspired by the
node feature learning in [14].

##  3 Proposed Method

Our method is based on two observations: (1) there are informative connections
among AU activations [14] and (2) the presence of the AUs is related to the
facial expression [9]. Following these statements, we proposed a model with a
graph convolution network (GCN) to exploit the AUs’ connections and a cross
attention module to learn the influence of AUs’ presence on facial emotion
expression.

Figure 1: Block diagram of our multi-task cross attention model

The architecture of the proposed model is illustrated in Fig 1. We used a pre-
trained EfficientNet [16] to obtain the facial feature vector from the input
image. The image feature is then fed to three blocks corresponding to three
tasks. Regarding the AU detection task, we utilized the AU relation graph to
create the AU-specific features. For expression recognition and valance-
arousal estimation, we used two fully connected layers to generate the
predictions. In addition, we devised an attention-based module to learn the
effect of facial AUs on the prediction of the emotion recognition task.

###  3.1 AU relation graph

To learn the representation of AUs, we adopted the Action unit Node feature
learning (ANFL) module proposed by Luo [14]. The ANFL module consists of $N$
fully connected layers corresponding to $N$ AUs. These layers generate the AU-
specific feature vectors $\\{v_{i}\\}^{N}_{i=1}=V$ using the extracted image
feature $X$. Mathematically, the AU-specific feature vectors are given by:

$\displaystyle v_{i}=\sigma\left(W_{i}X+b_{i}\right)$ (1)

Afterward, the Facial Graph Generator (FGG) constructs a fully connected graph
with N nodes corresponding to $N$ AU-specific feature vectors. The edge weight
of two nodes is the inner dot product of the two corresponding vectors. The
graph is simplified by removing edges with low weights. We chose k-nearest
neighbors algorithm to keep valuable connections between nodes. We used the
simplified topology to create the adjacency matrix of a GCN. The GCN is used
to enrich the connection information among AU-specific feature vectors.
Generally, the AU-specific feature vectors learned by the GCN network are
denoted by:

$\displaystyle V^{FGG}=f_{FGG}\left(V\right)$ (2)

Finally, we calculate the similarity between the $v^{FGG}_{i}$ and $s_{i}$ to
get the probability of each AU activation using the node features from the
GCN. The similarity function is defined by:

$\displaystyle\hat{y}_{AU,i}=\frac{ReLU\left(v^{FGG}_{i}\right)^{T}ReLU\left(s_{i}\right)}{{\lVert
ReLU\left(v^{FGG}_{i}\right)\rVert}_{2}{\lVert
ReLU\left(s_{i}\right)\rVert}_{2}}$ (3)

where $s_{i}$ is trainable vector having same dimension as $v^{FGG}_{i}$. The
operation of ANFL is illustrated in Fig 2. More detail about ANFL is in [14].

Figure 2: AUs Relationship-aware Node Feature Learning module

###  3.2 FER and VA estimation heads

To estimate FER and VA, we simply put the image feature X into two fully
connected layers in parallel. We used Batch Normalization and Tanh activation
function to produce the valance-arousal value $\hat{y}_{VA}$. Meanwhile, FER
head generates unweighted logit prediction $\tilde{y}_{EX}$ without any
activation function. With trainable parameters $W_{VA},b_{VA},W_{EX},b_{EX}$,
the formulas of $\hat{y}_{VA}$ and $\tilde{y}_{EX}$ are given below:

$\displaystyle\hat{y}_{VA}=\tanh\left(W_{VA}X+b_{VA}\right)\;$ (4)
$\displaystyle\tilde{y}_{EX}=W_{EX}X+b_{EX}$ (5)

Inspired by the additive attention of Bahdanau [1], we devised a multi-task
cross attention module to discover the relationship between AU prediction and
facial expression. Given an AU prediction $\hat{y}_{AU}$ (query) and FER
unweighted prediction $\tilde{y}_{EX}$ (key), the attention module generates
the attention scores as the formula below:

$\displaystyle
h=W_{v}*\tanh\left(W_{q}\hat{y}_{AU}+W_{k}\tilde{y}_{EX}\right)$ (6)

where $W_{q},W_{k},W_{v}$ are trainable parameters.

The attention weights $a$ are computed as the softmax output of the attention
scores $h$. This process is formulated as:

$\displaystyle a=Softmax(h)$ (7)

Finally, the weighted FER prediction is the element-wise product of attention
weight $a$ and the FER unweighted prediction $\tilde{y}_{EX}$ (value):

$\displaystyle\hat{y}_{EX}=a*\tilde{y}_{EX}$ (8)

###  3.3 Loss function

We used the weighted asymmetric loss proposed by Luo [14] for AU detection
task training process:

$\displaystyle\mathcal{L}_{AU}=-\frac{1}{N}\sum_{i=1}^{N}w_{i}\left[y_{AU,i}\log\left(\hat{y}_{AU,i}\right)+\left(1-y_{AU,i}\right)\hat{y}_{AU,i}\log\left(1-\hat{y}_{AU,i}\right)\right]$
(9)

Each binary entropy loss of an AU is multiplied with a coefficient $w_{i}$.
These factors are used to balance the contribution of each AU in the loss
function because their appearance frequencies of AUs are different from each
other as in Fig 3 . The factor $w_{i}$ is computed from occurrence rate
$r_{i}$ of AU $i^{th}$ in the dataset:

$\displaystyle w_{i}=N\frac{\frac{1}{r_{i}}}{\sum_{j=1}^{N}\frac{1}{r_{j}}}$
(10)

As the ratio of 8 expressions are imbalance, the loss we use for FER task is
weighted cross entropy function, which is given by:

$\displaystyle\mathcal{L}_{EX}=-\sum_{i}^{C}P_{i}y_{EX,i}\log\left({\rho}_{i}\left(\hat{y}_{EX}\right)\right)$
(11)

where ${\rho}_{i}\left(\hat{y}_{EX}\right)$ represents the softmax function,
$P_{i}$ is the refactor weight calculated from training set and $C$ is the
number of facial expressions.

Figure 3: Distribution of facial attributes in s-Aff-Wild2 database

For VA estimation task, the loss function is obtained as the sum of individual
CCC loss of valence and arousal. The formula is given by:

$\displaystyle\mathcal{L}_{VA}=1-CCC^{V}+1-CCC^{A}$ (12)

$CCC^{i}$ is the concordance correlation coefficient (CCC) and $i$ could be
$V$ (valence) or $A$ (arousal). CCC is a metric measures the relation of two
distributions, denoted by:

$\displaystyle
CCC=\frac{2s_{xy}}{s^{2}_{x}+s^{2}_{y}+\left(\overline{x}-\overline{y}\right)^{2}}$
(13)

where $\overline{x}$ and $\overline{y}$ are the mean values of ground truth
and predicted values, respectively, $s_{x}$ and $s_{x}$ are corresponding
variances and $s_{xy}$ is the covariance value.

##  4 Experiment

###  4.1 Dataset

The experiment is conducted on the s-Aff-Wild2 dataset [5], which is a static
version of Aff-Wild [20] and Aff-Wild2 [11] databases. s-Aff-Wild2 is a
collection of 221,928 images annotated with 12 action units, 6 basic
expressions, and 2 continuous emotion labels in valence and arousal
dimensions. In the s-Aff-Wild2 dataset, some samples may lack annotations for
one of the three mentioned labels. Such missing labels may cause an imbalance
in the number of valid annotations among the batches split by the data
sampler. For simplicity, instead of implementing a dedicated data sampler, we
decided to train three tasks separately.

The organizers provided the facial cropped images extracted from the videos
and corresponding cropped-aligned images. We only used the cropped-aligned
version to exploit the facial features. We resized all images from
$112\times$112 to $224\times$224 pixel, normalized, applied random cropping
and random horizontal flipping before the feature extraction step.

###  4.2 Evaluation metrics

Following [4], the evaluation metric of the MTL challenge $\mathcal{P}_{MTL}$
is the summation of three uni-task performance measure:

$\displaystyle\mathcal{P}_{MTL}=\mathcal{P}_{AU}+\mathcal{P}_{EX}+\mathcal{P}_{VA}$
(14)

where $\mathcal{P}_{AU}$ is the average F1 score of the 12 AUs in AU detection
task, $\mathcal{P}_{EX}$ is the average F1 score of 8 expression categories in
FER task and $\mathcal{P}_{VA}$ is the average of CCC indexes of valance and
arousal in VA estimation task. The performance metrics of individual tasks can
be formulated as:

$\displaystyle\mathcal{P}_{AU}=\frac{\sum_{i}^{N}{F}^{AU,i}_{1}}{12}$ (15)
$\displaystyle\mathcal{P}_{EX}=\frac{\sum_{i}^{C}{F}^{exp,i}_{1}}{8}$ (16)
$\displaystyle\mathcal{P}_{VA}=\frac{CCC_{V}+CCC_{A}}{2}$ (17)

where $CCC_{V},CCC_{A}$ are the CCC of valence and arousal, respectively;
${F}^{exp,i}_{1}$ is the F1 score of the emotion label and ${F}^{AU,i}_{1}$ is
the F1 score of the action unit.

###  4.3 Experiment setting

We implemented our solution using Pytorch framework and conducted the
experiments on NVIDIA RTX 2080 Ti GPU. Stochastic Gradient Descent was applied
following with Sharpness-aware Minimization Optimizer [3] to minimize the loss
function. The model was trained with an initial learning rate of $10^{-3}$ and
the cosine decay learning rate scheduler was also used. Because the CCC lost
function requires a sufficient sequence of predictions [9] so we chose the
batch size of 256 for all tasks’ training process. The weight decay was
applied to prevent overfitting.

The pre-trained EffecientNet in [16] can capture facial features efficiently.
Therefore, we decide not to train it in our experiment. Firstly, we train the
ANFL module with AU detection tasks so that the model can learn the knowledge
of AUs activation. Subsequently, we continue training the model with the FER
task to take advantage of AU detection results to support the prediction of
emotion. The VA estimation head can be trained independently.

###  4.4 Result

We attain the evaluation measure of $128.8\%$ compared to $30.0\%$ of the
baseline model [4]. A model which uses pre-trained EfficientNet and three
fully connected layers accomplishes the performance score of $121.5\%$. By
utilizing the multi-task cross attention and GCN, we improve the prediction of
three tasks as shown in Table 1. Our best model obtains the score of
$111.35\%$ on the test set of the s-Aff-Wild2 database. All metrics in this
section are in percentage (%).

Table 1: The uni-task and multi-task performance of our model and other options. The evaluation metrics are performed on the validation set of the s-Aff-Wild2 dataset Model | $\mathcal{P}_{AU}$ | $\mathcal{P}_{EX}$ | $\mathcal{P}_{VA}$ | $\mathcal{P}_{MTL}$
---|---|---|---|---
Baseline | - | - | - | 30.0
FC-layer heads | 41.4 | 32.2 | 47.9 | 121.5
FC-layer heads (BatchNorm VA head) | 45.6 | 32.2 | 47.9 | 125.7
Proposed method (w/o cross attention) | 45.6 | 32.2 | 49.9 | 127.7
Proposed method (cross attention) | 45.6 | 33.3 | 49.9 | 128.8

By discovering the association of AUs, the model increases the average F1
score to $49.9$, better than the measure of $47.9$ when using a fully
connected layer followed by a sigmoid activation. The SAM optimizer improves
model generalization and yields better performance. The detailed results are
listed in Table 2.

Table 2: The comparison F1 score of each AU prediction on validation set between FC-Sig (a fully connected layer followed by a sigmoid function), ANFL, and sANFL (ANFL with SAM optimizer) | AU# |
---|---|---
Model | 1 | 2 | 4 | 6 | 7 | 10 | 12 | 15 | 23 | 24 | 25 | 26 | $\mathcal{P}_{AU}$
FC-Sig | 53.1 | 42.2 | 56.4 | 56.3 | 73.5 | 72.0 | 63.6 | 9.9 | 14.8 | 9.2 | 87.6 | 35.5 | 47.9
ANFL | 51.2 | 36.0 | 55.8 | 56.9 | 72.9 | 70.6 | 63.7 | 18.8 | 15.9 | 10.3 | 87.6 | 32.6 | 47.7
sANFL | 56.6 | 42.9 | 61.1 | 55.7 | 73.7 | 71.1 | 66.5 | 18.5 | 15.7 | 13.0 | 87.8 | 36.4 | 49.9

In the VA estimation task, our assumption is that the batch normalization can
boost the accuracy of the prediction. The batch normalization layer can change
the mean and variance of a batch, respectively, to the new values $\beta$ and
$\gamma$, which are learnable parameters. By using CCC for the loss function
as in section 3.3, the network can learn to adapt the parameters
$\beta,\gamma$ to the mean and variance of ground truth distribution. Thus,
the performance of the VA estimation task can be increased. We conducted the
experiments with and without the batch normalization layer to test its
operation. As the result, the batch normalization layer can improve
$\mathcal{P}_{VA}$ by more than 4%. The CCC indexes of valence, arousal, and
their average value are described in Table 3.

Table 3: The VA evaluation metrics on validation set of the model with and without Batch Normalization Model | ${CCC}^{V}$ | ${CCC}^{A}$ | $\mathcal{P}_{VA}$
---|---|---|---
FC+Tanh Activation | 43.9 | 38.8 | 41.4
FC+BatchNorm+Tanh Activation | 47.5 | 43.6 | 45.6

In the FER network, there is a noticeable improvement when we use cross
attention. We obtain the average F1 score of 33.3 which is considerably higher
than the case of not using multi-task cross attention. To evaluate the
contribution of the attention module, we compared the F1 scores for individual
emotion labels by excluding the attention mechanism. As shown in Table 4, the
use of attention increased the performance for individual labels except for
happy and surprise.

Table 4: The F1 scores of emotion labels on validation set of the model with and without cross-attention Model | Neu | Ang | Dis | Fea | Hap | Sad | Sur | Oth | $\mathcal{P}_{EX}$
---|---|---|---|---|---|---|---|---|---
Without attention | 35.7 | 24.0 | 45.4 | 23.3 | 43.9 | 35.4 | 22.8 | 27.3 | 32.2
With attention | 36.3 | 24.6 | 49.9 | 25.2 | 41.8 | 36.9 | 22.0 | 29.7 | 33.3

Although there are enhancements in multi-task inference, our model can be
further improved and tuned for individual tasks. The operation of the AU graph
is overfitting to the training set without the support of SAM optimizer. It is
less efficient than the FC-Sig module on the validation set when we remove
SAM. Regarding the FER task, the prediction of happy label, which is a common
emotion, is not improved when the attention mechanism is applied.

##  5 Conclusion

In this work, we introduced the attention-based approach to the multi-task
learning problem. The idea is to exploit the output of one task to enhance the
performance of another task. In particular, our model attends to the AU
detection task’s result to achieve better output in the FER task. Our result
is outstanding compared to the baseline and the cross attention module
improves the evaluation metric on the FER task. The experiment demonstrates
that facial AUs have a strong relationship with facial expression and this
relation can be leveraged to recognize human emotion more efficiently.
Additionally, we take advantage of the GCN and the batch normalization to
accomplish the AU detection and VA estimation task, respectively, with
considerable advancement.

However, in our model, the progress to generate valence and arousal is
completely independent of other tasks. In our architecture, except for the
image feature, there is no common knowledge between VA estimation and
remaining heads. The relation between valence-arousal and other facial
attributes is not exploited in this paper. In the future, we plan to study the
influence between valence-arousal and other facial behavior tasks.

##  6 Acknowledgement

This work was supported by a National Research Foundation of Korea (NRF) grant
funded by the Korean government (MSIT). (NRF-2020R1A4A1019191).

## References

  * [1] Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
  * [2] Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: A dataset for recognising faces across pose and age. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). pp. 67–74. IEEE (2018)
  * [3] Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=6Tm1mposlrM
  * [4] Kollias, D.: Abaw: Learning from synthetic data & multi-task learning challenges. arXiv preprint arXiv:2207.01138 (2022)
  * [5] Kollias, D.: Abaw: Valence-arousal estimation, expression recognition, action unit detection & multi-task learning challenges. arXiv preprint arXiv:2202.10659 (2022)
  * [6] Kollias, D., Cheng, S., Pantic, M., Zafeiriou, S.: Photorealistic facial synthesis in the dimensional affect space. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 0–0 (2018)
  * [7] Kollias, D., Cheng, S., Ververas, E., Kotsia, I., Zafeiriou, S.: Deep neural network augmentation: Generating faces for affect analysis. International Journal of Computer Vision 128(5), 1455–1484 (2020)
  * [8] Kollias, D., Nicolaou, M.A., Kotsia, I., Zhao, G., Zafeiriou, S.: Recognition of affect in the wild using deep neural networks. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. pp. 1972–1979. IEEE (2017)
  * [9] Kollias, D., Sharmanska, V., Zafeiriou, S.: Distribution matching for heterogeneous multi-task learning: a large-scale face study. arXiv preprint arXiv:2105.03790 (2021)
  * [10] Kollias, D., Tzirakis, P., Nicolaou, M.A., Papaioannou, A., Zhao, G., Schuller, B., Kotsia, I., Zafeiriou, S.: Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. International Journal of Computer Vision pp. 1–23 (2019)
  * [11] Kollias, D., Zafeiriou, S.: Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface. arXiv preprint arXiv:1910.04855 (2019)
  * [12] Kollias, D., Zafeiriou, S.: Va-stargan: Continuous affect generation. In: International Conference on Advanced Concepts for Intelligent Vision Systems. pp. 227–238. Springer (2020)
  * [13] Kollias, D., Zafeiriou, S.: Affect analysis in-the-wild: Valence-arousal, expressions, action units and a unified framework. arXiv preprint arXiv:2103.15792 (2021)
  * [14] Luo, C., Song, S., Xie, W., Shen, L., Gunes, H.: Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition. arXiv preprint arXiv:2205.01782 (2022)
  * [15] Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing 10(1), 18–31 (2017)
  * [16] Savchenko, A.V.: Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. In: Proceedings of the 19th International Symposium on Intelligent Systems and Informatics (SISY). pp. 119–124. IEEE (2021), https://arxiv.org/abs/2103.17107
  * [17] Savchenko, A.V.: Frame-level prediction of facial expressions, valence, arousal and action units for mobile devices. arXiv preprint arXiv:2203.13436 (2022)
  * [18] Sharma, G., Ghosh, S., Dhall, A.: Automatic group level affect and cohesion prediction in videos. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). pp. 161–167. IEEE (2019)
  * [19] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)
  * [20] Zafeiriou, S., Kollias, D., Nicolaou, M.A., Papaioannou, A., Zhao, G., Kotsia, I.: Aff-wild: Valence and arousal ‘in-the-wild’challenge. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. pp. 1980–1987. IEEE (2017)