# A Guide to Employ Hyperspectral Imaging for Assessing Wheat Quality at
Different Stages of Supply Chain in Australia: A Review

Priyabrata Karmakar, Shyh Wei Teng. Manzur Murshed, Paul Pang, Cuong Van Bui
Priyabrata Karmakar, Shyh Wei Teng. Manzur Murshed, Paul Pang, Cuong Van Bui
are with the Institute of Innovation, Science and Sustainability, Federation
University Australia (email:p.karmakar,shyh.wei.teng, manzur.murshed, p.pang,
v.bui@federation.edu.au).This work was supported by Australian Government
Traceability Grants Program Round 2.

###### Abstract

Wheat is one of the major staple crops across the globe. Therefore, it is
mandatory to measure, maintain and improve the wheat quality for human
consumption. Traditional wheat quality measurement methods are mostly
invasive, destructive and limited to small samples of wheat. In a typical
supply chain of wheat, there are many receival points where bulk wheat
arrives, gets stored and forwarded as per the requirements. In this receival
points, the application of traditional quality measurement methods is
difficult and often very expensive. Therefore, there is a need for non-
invasive, non-destructive real-time methods for wheat quality assessments. One
such method that fulfils the above-mentioned criteria is hyperspectral imaging
(HSI) for food quality measurement and it can also be applied to bulk samples.
In this paper, we have investigated how HSI has been used in the literature
for assessing stored wheat quality. So that the required information to
implement real-time digital quality assessment methods at the different stages
of Australian supply chain can be made available in a single and compact
document.

###### Index Terms:

Wheat supply chain, quality, safety, hyperspectral imaging.

##  I Introduction

Wheat is the most produced crop in Australia. On average 25 million tonnes of
wheat is produced in Australia per annum and it is one of the most important
commodity in the global trade scenario [1]. Approximately 75% of the
production can be exported to overseas and the remaining is used for domestic
market [2].

There are various wheat classes in Australia based on the quality parameters
and the suitability for end-use products: Australian Prime Hard (APH),
Australian Hard (AH), Australian Premium White (APW), Australian Noodle Wheat
(ANW), Australian Standard White (ASW), Australian Premium Durum (ADR), and
Australian Soft (ASFT) [3]. Each class may have different grades which is
allocated by Grain Trade Australia (GTA) based on the physical grain quality.
Table A Guide to Employ Hyperspectral Imaging for Assessing Wheat Quality at
Different Stages of Supply Chain in Australia: A Review shows how different
classes and grades of Australian wheat differs in terms of quality. When a
consignment reaches to a receival point in a supply chain, sample of grains
are taken out for assessing the physical quality such that, 3 litres of
samples are drawn for deliveries up to 10 tonnes with an additional litre for
each additional 1o tonnes. Assessment of wheat samples consist of testing for
moisture, protein, test weight, screenings, un-millable material, infected
grains, foreign seed contaminants and other contaminants [3].

Traditional methods of testing wheat quality and safety parameters consist of
chemical, biological analysis and human visual inspection, respectively.
However, these methods are invasive, destructive, slow and expensive.
Therefore, the testing paradigm is shifted more towards the faster, non-
invasive, non-destructive and real-time/online approaches.

One of most popular non-invasive and digital approach for quality and safety
assessments in agriculture and food industry is hyperspectral imaging (HSI).
At the beginning, HSI was used as a tool in remote sensing applications but
gradually it has been successfully applied in different applications of
agriculture and food industry. Some examples are: food quality and safety
analysis [4, 5], detection of damages in fruits and vegetables [6, 7],
detection of contaminants in food products [8], quality analysis of dairy [9]
and meat [10] products. HSI has also been widely used cereal grain quality and
safety analysis. In the literature, there are many research and survey
articles can be found on the usage of HSI in food quality and safety analysis
including quality assessments of wheat and other cereal grains. However, the
literature still lacks a thorough analysis how HSI can be used to assess
stored wheat quality at the different stages of supply chain in the Australian
context. Therefore, we have written this survey paper where we have discussed
how HSI can be used to assess post-harvest stored wheat quality as per GTA
standards [3] provided in Table A Guide to Employ Hyperspectral Imaging for
Assessing Wheat Quality at Different Stages of Supply Chain in Australia: A
Review.

The rest of paper is organised as follows. Section II provides an introduction
to HSI. Section III discusses the applications of HSI for wheat protein
prediction followed by wheat moisture analysis discussion in Section IV.
Detection of different types of defects in wheat grain using HSI is discussed
in Section V. Section VI illustrates the detection of contaminants in stored
wheat using HSI. Finally, Section VII concludes the paper.

##  II Introduction to Hyperspectral Imaging

RGB colour imaging is a non invasive and non destructive approach. It deals
with three spectral measurements (red, green and blue) within the wavelength
range (400—700 nm) [11] of electromagnetic spectrum to match with the
trichromatic nature of the human eyes. Different types of features like
morphological (shape and size), colour, texture can be extracted from RGB
images and they have been successfully used in various applications of
agriculture industry. For example, classification of cereal grains using
morphological, colour and texture features [12, 13], classification of
different grain types and different dockage categories [14, 15, 16] and
identification of damaged kernels in bulk wheat [17] using morphological and
colour features, identification of moisture level in wheat kernels using
morphological features [18, 19]. However, RGB image features have some
limitations. The effectiveness of these features to analyse the chemical
composition of objects is limited. They are also unable to detect the minor
constituents and tiny contaminants. In agriculture and food industry, it is
required to build a system that can analyse not only the physical properties
but the chemical properties as well. In addition, the system should be
effective to identify and separate tiny elements.

In early 2000’s, X-ray imaging [20, 21] and near-infrared spectroscopy (NIRS)
[22] have shown potential for online, non-invasive and non-destructive
applications. However, X-ray imaging present potential health risks as wheat
kernels are exposed to radiation and it is an expensive process as well. NIRS
analyse the spectral signatures of the objects and able to detect changes in
chemical composition. However, it does not provide any spatial information.
Therefore, it is unable to provide any location specific information.

On the other hand, hyperspectral imaging (HSI) combines NIR spectroscopy and
digital imaging to provide spatial as well as spectral information of the
targets. HSI cameras can operate from the UV range to the near-infrared range
and the captured data contains information belonging to a wide range (200 nm-
1300 nm) [23] of electromagnetic (EM) spectrum. A visual representation of EM
spectrum from ultraviolet to infrared range is given in Figure 1.

HSI captured data corresponds to three-dimensional (3D) hypercubes consist of
two spatial dimensions and one spectral dimension [24]. It captures multiple
images at different wavelengths for the same spatial area. HSI provides a
large amount of data that helps analysing inherent properties of the target.
When biological elements are exposed to the EM energy, they reflect, scatter
and absorb in a unique pattern at different wavelengths due to their chemical
composition and inherent physical structure. This pattern is called spectral
signature or spectrum. As it is unique to each element, the spectral
signatures can be used to identify or distinguish between different elements
[25].

Figure 1: Electromagnetic spectrum wavelength (nm) [26].

Hyperspectral imaging generally measures the intensity of the absorbed or
emitted radiation over a range of a spectral band along the EM spectrum. The
goal of the HSI is to understand the characteristics of the target object at
the molecular level when it is exposed to EM signals. As HSI deals at the
molecular level, it can identify the minor changes in the chemical composition
of the target objects [27].

Data collected using a hyperspectral imaging system are the absorbed or the
emitted radiation values and are stored in the form of a hypercube. The
hypercube is a complex data unit, which contains abundant information about
the physical and chemical properties of the sample. Hypercubes can be viewed
as the stacks of 2D images each captured at a particular wavelength. Each
pixel in a hypercube contains the entire spectrum belonging to the imaging
bands. These spectra are analysed to obtain the information about the
composition of the corresponding pixels. A sample hypercube along with a pair
of spectra is provided in Figure 2.

Figure 2: A sample Hypercube showing spatial ($x$,$y$) and spectral
($\lambda$) dimensions (left) and spectra of two separate pixels (right).
Adopted from [28].

After HSI data has been captured, it is pre-processed to extract analytical
information and remove scattering effects. At the next step, chemometric
analysis to be performed on the pre-processed data for classifying,
identifying the elements under consideration or predicting the chemical
composition (e.g., protien , moisture, DON level) of elements [29]. Examples
of chemometric analysis are principal component analysis, partial least
squares, multi-linear regression, linear discriminant analysis, Fischer
discriminant analysis and artificial neutral networks [30]. In this paper, we
have reviewed literature belonging to HSI applications for quality assessments
of stored wheat. So that an overall idea can be established on how to assess
wheat quality at the different stages of supply chain in the Australian
context. A list of paper which are reviewed is provided in Table I. The
detailed discussion is provided in the later sections.

TABLE I: List of literature  Application |  Related literature
---|---
Protein prediction |  [31],[32],[33],[34],[35],[36],[37], [38], [39], [40], [41], [42], [43], [44]
Moisture prediction |  [45], [38], [46], [47]
Defect detection |  [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [54], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74] [75], [76], [77], [78], [79]
Contaminant detection |  [80], [81], [82], [83], [84]

##  III HSI for protein prediction

Protein is one of the most important wheat quality parameter due to its impact
in baked products. Therefore, wheat price is dependent on protein content.
Based on this, bonus payments or discount offers are being made on top of the
base rate for each grade [85]. Before the application of HSI, Near-infrared
spectroscopy (NIRS) was used as the digital method to investigate the wheat
protein content. Initially, NIRS was possible to be applied only on ground
wheat or flour[31]. As it is destructive approach, investigations on non-
destructive NIRS application on bulk of whole whole wheat were initiated in
[32, 33]. Among individual wheat kernels, protein content varies [36].
However, analysis of bulk wheat samples, these variability cannot be detected
[34]. Therefore, NIRS analysis has been applied to individual kernels as well
[34, 35]. It has been also shown that protien prediction of bulk wheat can be
improved by averaging the individual kernel measurement of the corresponding
bulk [37]. Partial least squares (PLS) regression was applied for protein
prediction on the NIRS data of single wheat kernels in [38].

Development of an HSI calibration was proposed in [39] to measure and
visualise protein distribution within single wheat kernels by assessing the
uniformity present in the wheat samples. A deep-learning based regression
model has been proposed in [40] for protein prediction in bulk grain by
processing the entire hypercube (spatial and spectral data) using a
modification of ResNet-18 [41], a popular convolutional neural network
architecture. A comparison of two regression models: Partial Least Squares
Regression (PLSR) and Principal Components Regression (PCR) has been performed
for protein prediction using NIR hyperspectral images of Canadian wheat in
[42] and experimental results show PLSR outperform PCR. Similar observation
was obtained in [43] to detect protein content using HSI of 11 varieties of
wheat. However, in this case, protein prediction accuracy was improved by
applying radial basis function (RBF) neural network.

Traditionally, protein content calculation in a wheat sample is dependent on
its nitrogen (N) content. As per GTA standards, the conversion factor to
obtain the wheat protein from nitrogen is 5.7. i.e., $Protein=Nitrogen\times
5.7$. However, in practice, this conversion factor ranges from $5.7-6.25$ as
different testing laboratories follow their own conversion factors [86, 87].
As N is an indicator of wheat protein content, N prediction like in [44] can
be used for calculating protein prediction where a SWIR imaging and PLS
regression have been performed to predict the N concentrations in wheat
samples and it was found that most important spectral bands are 1451–1600 nm,
1901–2050 nm and 2051–2200 nm.

##  IV HSI for moisture prediction

In Australia, grains are traditionally harvested at a lower moisture content
(mostly under 13%) after it has been sun dried to avoid the extensive drying
process post harvest [85]. Although post maturity sun drying can decrease the
quality and early harvesting with higher moisture can increase the yield [88],
the lower the moisture content, the higher advantage for buyers as higher
moisture wheat are prone to insect infestation, mould growth, reduced
germination [89].

Therefore, precise identification of moisture content in wheat is crucial.
Higher level of moisture can cause spoilage and sprouting in wheat kernels. At
a supply chain receival point, wheat samples may arrive from different sources
of variable moisture levels and kept in storage together prior to further
processing. Therefore, before mixing the wheat from various sources, it is
necessary to dry high-moisture wheats to an optimum level to prevent spoiling
the entire storage bin.

Similar to protein analysis, NIRS was also used for moisture prediction in
wheat as non-invasive digital method. Initially NIRS was used for moisture
prediction in ground wheat [45] but later on it was also used for whole wheat
kernels in [38] where PLS regression was performed on NIRS wheat data for
moisture content analysis after removing the influence of the protein content
to avoid the correlation between moisture and protein.

HSI was used in [46] to classify five different moisture levels in western
Canadian wheat classes in the wavelength range of 960-1,700 nm. Also each
wheat class is identified at five different moisture levels (12, 14, 16, 18
and 20%) at four effective wavelengths of 1,060, 1,090, 1,340, and 1,450 nm.
This work has been extended to classify moisture-specific wheat classes in
[47].

##  V HSI for Grain defect detection

The percentage of defective kernels in a bulk of wheat is a determining factor
to classify wheat in to the major milling grades. In Australia, the maximum
allowable percentage of defective grains due to different factors is
standardised by Grain Trade Australia (GTA) [3]. Traditionally, defects were
detected by visual inspection by examining the kernels. However, these methods
are time consuming, labour intensive and expensive. Moreover, often visual
identification of defective grains are challenging as they may look similar to
the sound kernels.

###  V-A Sprouting

One of the main cause for grain defect is sprouting. The baking quality of
flour produced from sprouted wheat is poor. Therefore, sprouting is one of the
main determining factor for assessing wheat in different grades. Sprouting
occurs as a result of germination due to rain damage by absorbing water at the
pre-harvest stage. Rain damage causes the development of the $\alpha$-amylase
and other germinative enzymes which negatively impact the flour produced from
the sprouted wheat. In addition, sprouted wheat kernels are prone to
infestation by insects [90]. In Australia, there is zero tolerance of sprouted
wheat for human consumption [3]. A sample image showing different stages of
sprouting in wheat is provided in Figure 3.

Figure 3: Different sprouting stages in a wheat kernel. Adopted from [90].

Traditional pre-harvest rain damage detection methods like Hagberg Falling
Number test [90] is destructive and time-consuming. Therefore, non-destructive
methods were started to be explored. X-rays used to classify sprouted and
sound wheat kernels in [21] but X-rays present potential health risks as wheat
kernels are exposed to radiation.

Similar to other parameters of assessing wheat quality, HSI also has been
explored to detect rain-damage in wheat kernels. It was demonstrated in [48]
that NIR spectra can be used for damage detection due to sprouting but, the
effectiveness was compromised as only three wavelengths were considered in
1445–2345 nm range of NIR spectra. An equally spaced 60 wavelengths in the
range 1000-1600 nm has been used for NIR HSI for classifying sprouted and
healthy wheat kernels. From the results, three specific wavelengths 1101.7,
1132.2, and 1305.1 nm were identified as significant for the analysis. A
visible near-infrared (VNIR) HSI in the range 400-1000 nm was employed to
classify sprouted, severely sprouted and healthy kernels. Average spectra of
sprouted and severely sprouted kernels show higher reflectance responses
compared to the healthy kernels in the wavelength region above 720 nm.
Principle component analysis (PCA) was used to pick four suitable wavelengths.
It has also been shown that by taking the ratio of reflectance between 878 nm
and 728 nm, healthy kernels can be distinguished from the sprouted ones.[49].

In [50], authors claimed that when HSI analysis performed at VNIR range, at
675 nm, a significant difference for spectral reflectivity between sprouting
and healthy kernels was noticed. It was also shown that the imaging spectra
can differentiate different levels of wheat preharvest sprouting. To
discriminate between sprouted and sound kernels using NIR HSI, it was observed
that, there is a visible difference between them in two spectral bands which
are 844-1140 nm and 1386-1700 nm, respectively. In the first band (844-1140
nm), sound kernels have a higher reflectance 10-20% than the sprouted ones. In
contrast, the reflectance of sprouted kernels is 20%-60% larger than the sound
kernels in second band(1386-1700 nm) [51]. It is difficult to identify
slightly sprouted kernels by visual inspection and it can deteriorate the
quality of sound kernels if mixed together. Therefore, in [52] NIR HSI has
been used in 866.4 nm - 1701 nm wavelength range to differentiate between
slightly sprouted and sound kernels. The authors have also investigated the
effect of imaging of each kernel from both sides and found that the result is
better when considering reverse (dorsal) side than the front (ventral) side.
Wheat kernels undergo changes in chemical composition, enzymatic activities,
starch and protein contents when affected by rain damage. A portable NIR
device which works in the wavelength range of 950–1650 nm has been employed to
monitor these changes during the germination process to determine the level of
sprouting [53].

###  V-B Black point

Another major grain defect is black point or black tip that begins in the germ
and extends around one or both cheeks and into the crease. It is caused by
moisture and damp conditions during the ripening of wheat. Black point
affected wheat considered to be not fit for human consumption and therefore,
needs to be removed [90]. A sample image showing black point affected kernels
is provided in 4.

Figure 4: (left): Sound kernel; (right): Black point affected kernel. Adopted
from [90].

Classification of black point affected kernels and sound kernels has been
conducted in [54] using HSI. The authors have shown that the results have
minimal effect if shorter wavelength ranges ( 420-1000 nm or 420-700 nm) have
been used instead larger one (420-2500 nm). Therefore, higher classification
accuracies can be achieved considering a few wavelengths and the process can
be made efficient and less-expensive. In [55], a single fluorescence
wavelength (531 nm) was selected for spectral image processing to
differentiate between sound and black point-damaged kernels. Authors have
concluded that the effectiveness of detection of black points are higher when
wheat kernels are imaged from the dorsal side. This is because, the damaged
region is comparatively smaller in the ventral side. Also the crease in the
ventral side may create a shadowing effect which can cause confusion with the
black point.

Individual damage or defect detection in wheat kernels may be time consuming
and expensive where a bulk of wheat samples are needed to be analysed faster.
Therefore, in [56], an HSI approach in 865-1711 nm wavelength range is
proposed to identify damaged (including black point) kernels from a bulk of
wheat. At first, effective wavelengths were obtained separately by the
application of PCA and successive projection algorithm (SPA). After that,
Partial least square (PLS) and least square-support vector machine (LS-SVM)
classifiers were built using both full spectral and effective wavelength data.
It has been observed that LS-SVM outperforms PLS and results with full
spectral and effective wavelength data were comparable.

###  V-C Fungal infection

Fungal infection causes severe damage in wheat kernels. Fusarium head blight
(FHB) or fusarium damage is another major defect in wheat kernels caused
mainly by a fungus name Fusarium graminearum [91]. Other species such as
Fusarium culmorum, and Fusarium poae can also cause FHB damage [57]. Fusarium
infection causes the production of mycotoxin deoxynivalenol (DON), a toxic for
humans and livestocks. Also, it reduces the kernel weight, affects flour
colour and baking performance. Therefore, infected kernels need to be removed.
A sample containing sound and FHB damaged kernels is provided in Figure 5.

Figure 5: A sample of wheat containing sound and FHB damaged kernels. FHB
damaged kernels are shrivelled and pale compared to the healthy ones. Adopted
from [92].

Traditional approach of detecting fungal infection by visual inspectors is not
suitable for rapid online testing. Thus, effective digital solutions are
required. NIR spectroscopy has been successfully used to separate Fusarium-
infected and sound kernels under controlled laboratory conditions [58] but
under commercial operating conditions, the NIR spectroscopy performance is not
satisfactory [59]. Imaging in short-wave infrared range (1000-2500 nm) has
also been used FHB detection [60] but the cameras operating in the short-wave
infrared range are very expensive and cannot be afford by many. To overcome
this situation, it has been shown in [54] that the HSI using 420–1000 nm
wavelength range provides comparable fungal-infection detection.

In [61], mildly and severely fusarium-damaged wheat kernels were distinguished
from the sound wheat kernels using HSI in the VNIR wavelength range. Spectral
images pre-processed by PCA to reduce the data to 10 scores upon which linear
discriminant analysis (LDA) models were developed to classify sound and
fusarium-damaged kernels. The damaged kernels were further classified into
mild and severe. It has also been shown that optimally selected six
wavelengths (484 nm, 567 nm, 684 nm, 817 nm, 900 nm and 950 nm) can exhibit
comparable results that with the full VNIR spectrum. Different degrees of FHB-
damage (e.g., mild, moderate, severe) in wheat kernels is identified in [62]
using HSI and deep neural networks. At first, five effective wavelengths (EW)
were selected from preprocessed raw HSI data. After that the combination of
reflectance images (RI) corresponding to the EWs as seen in Figure 6 were
analysed using LeNet-5 and residual attention convolution neural network
(RACNN) to identify the degrees of FHB-damage in wheat kernels. It was
concluded that the optimal performance was achieved for the combination of RIs
at 940 nm and 678 nm.

Figure 6: Reflectance images at five EWs of sound and FHB damaged kernels.
Class 0 to healthy and 1-3 corresponds to mild, moderate and severe damaged
kernels. Adopted from [62].

Another work on identifying the degree of FHB-damage in wheat kernels using
HSI and deep neural networks is proposed in [63]. The deep learning network is
neural architecture search (NAS)-based. HSI has been performed in 400-1000 nm
range. Effective wavelengths (EWs) were selected for reflectance spectroscopy
of HSI and corresponding monochromatic images were combined with an
architecture self-search deep network (ASSDN) to develop the self generated
and self-optimised determination model for the degree of FHB damage.

An HSI approach in NIR range (1000-1700 nm) has been considered in [64] to
separate fusarium-damaged kernels from the mixture of yellow berry kernels
(acceptable amount) and vitreous (sound) kernels. In addition, yellow berry
kernels were also separated from vitreous kernels when top quality wheat is
required. After compressing the data using PCA, partial least squares
discriminant analysis (PLS-DA) was applied to classify vitreous, yellow berry
and fusarium-damaged kernels. In addition, interval PLS-DA (iPLS-DA) was used
for the same classification in three shorter wavelength ranges. It has been
shown that classification results obtained using the entire wavelength range
and the shorter wavelength ranges are comparable. NIR HSI method has also been
used in [65] for processing single wheat kernel to classify them according to
fusarium damage, followed by determination of DON level in infected kernels
and finally classify infected kernels based on DON level. For HSI, single
wheat kernels were scanned both crease-up and crease-down position. After
spectral preprocessing the raw hyperspectral images, the data corresponding to
the optimally selected wavelengths used to train and validate machine learning
models (LDA, Naive Bayes, KNN) to achieve the classification and regression
predictions. Another NIR HSI method in the 900-1700 nm wavelength range has
been proposed for detection of fusarium damage and DON content in [66].

An online rapid detection of FHB-infected wheat kernels followed by the
determination of DON level in the infected kernels using HSI was proposed in
[67]. Imaging has been performed in 400–1000 nm. Two spectral preprocessing
methods: Standard normal variate (SNV) transformation and multiplicative
scatter correction (MSC) combined independently with two wavelength selection
methods: successive projections algorithm (SPA) and random frog (RF),
generated four different combinations of optical wavelength. After the pre-
processing, optimally selected data were used to build classification models
using SVM and PLS-DA classifiers. The combination of MSC–SPA–SVM resulted in
the highest classification accuracy of 97.92% on the test set. After that, the
DON levels in the FHB-infected kernels were also determined from the predicted
values of classifiers.

In [68], grayscale image features were fused with spectral data to identify
the degree of FHB-damage in wheat kernels. After pre-processing the raw HSI
data, optimal wavelengths were selected by applying successive projections
algorithm with random forest (SPA-RF) method. Image features were extracted
from the images corresponding to the optical wavelengths and combined with
spectral features. This fused features were then used to build classification
models to identify the sound or different degrees of damaged wheat kernels.
During the analysis of reflectance spectra of HSI data, it was noticed in [69]
that the more severe the FHB infection is, the greater the reflection is. This
is because the toxin released by the FHB destroys the cell wall of the wheat
to form a hole like structure between the seed coat and aleurone layer.
Therefore, it causes the increase in reflectance.

Formation of mycotoxins by fusarium in post harvest dry stored grain is
limited. In contrast, damage in stored grains are more prone to fungi like
Aspergillus and Penicillium [70] and produces a toxin named Ochratoxin A. The
damage due to these fungi and the resulting toxins were detected using an NIR
HSI approach in [71]. After preprocessing the HSI data, six statistical and
ten histogram features were extracted from the data corresponding to the
selected effective wavelengths. The extracted features were then used for
developing linear, quadratic and Mahalanobis discriminant classifiers. It has
been found that wavelengths 1280, 1300, and 1350 nm were crucial to detect
fungal damage and to detect Ochratoxin A. Among the classifiers, quadratic
discriminant classifier outperform the other two. Similar work of detecting
the damage due to Aspergillus and Penicillium was proposed in [72] where HSI
in 1000-1600 nm range was performed. After that the PCA-applied
dimensionality-reduced spectral data was used to build linear and quadratic
discriminant classifiers to distinguish between damaged and sound kernels.

A comparison of VNIR and SWNIR imaging have been performed for detecting DON
levels in fusarium damaged wheat kernels in [73]. The changes occurred in the
protein, carbohydrate and lipid contents of the damaged kernels due to
mycotoxins (DON) cannot be analysed visually. Therefore, to investigate the
effectiveness of damage detection by using the spectral characteristics, this
comparison has been performed. Data captured at the VNIR and SWNIR wavelength
ranges were preprocessed by either multiplicative scatter correction (MSC) or
standard normal variate (SNV) followed by the application of genetic algorithm
(GA) to select the effective wavelengths. Support vector machine (SVM) and
sparse autoencoder(SAE) network were used to build the classifiers and it was
found that the detection of DON in damaged kernels is more effective in VNIR
rather than SWNIR. Although the performances of SVM and SAE were comparable,
it was found that the combination of MSC-GA-SAE produced the highest accuracy.

In practical experiment scenario, it often happens that HSI samples available
to build machine learning models to identify sound and fungal-infected wheat
kernels is limited. Especially, the samples of damaged/infected kernels are
found lesser compared to the samples of sound kernels. This data imbalance can
cause compromising results generated from machine learning models. To overcome
this issue, in [74] deep convolutional generative adversarial network
(DCGAN)-based data augmentation has been used to increase the training data
from the raw NIR HSI data. Experimental results show that by expanding the
training samples, classification accuracies of different machine learning
models has been increased significantly.

###  V-D Insect-damage

Damage caused by insects such as rice weevil (Sitophilus oryzae), lesser grain
borer (Rhyzopertha dominica) [93] is a major concern in wheat storage
locations. Wheat grains result in weight and nutrient loss, reduced
germination ability and prone to fungal infection. Therefore, quality of the
wheat grains deteriorate significantly and causes reduced market value. In
addition, presence of insect fragments degrade the milling quality [94]. An
example of insect-damaged wheat kernel is shown in Figure 7.

Figure 7: Insect-damaged wheat kernel. Adopted from [95].

Mainly two types of insects damage wheat kernels, they are primary internal
feeders and secondary feeders. Former types are difficult to detect, they
develop inside the wheat kernels and get out after growing up, leaving holes
in the kernels. At the start of life cycle, female insects create a tiny hole
in the healthy kernels and deposit eggs there. The kernel cavity is covered by
a gelatinous secretion. The eggs hatch inside the kernel, feed and the grown
up insects emerges out of the wheat kernels. Often kernels infested by primary
feeders contain immature stages of insects (eggs, larvae, and pupae) but they
look apparently same as sound kernels. Later types survive on already damaged
and broken kernels [96]. A sample image of larvae and adult rice weevil is
shown in Figure 8.

Figure 8: (a) Larvae and (b) Adult stages of Rice weevil (Sitophilus oryzae).
Adopted from [97].

Respiratory and metabolic activities of insects generate heat and moisture
which can cause local hotspots in the grain storage and can result in fungal
infection [98, 99]. Traditional methods can only detect visual damages or
traces of insects but unable to detect internal insect infestation. HSI can
address this issue even with the spatial location of the infestation. When
wheat kernels are infested, there are significant changes occur in their
moisture and starch content [100]. These changes can be easily detected by HSI
unlike the traditional methods.

In [75], it has been found that images of internally infested and sound wheat
kernels captured at two different wavelengths (1202 and 1300 nm) show evident
differences. Internally infested kernels appear brighter. Whereas, the sound
kernels appear darker as can be seen in Figure 9. Although HSI analysis in
[75] effective to distinguish insect-damaged kernels, a machine learning model
build with training data is necessary to process unknown samples. Therefore,
supervised classification models using long-wave NIR HSI has been developed in
[76] to classify sound and infested kernels using the training samples as
images captured in the range of 1000-1600 nm. After preprocessing and
dimensionality reduction using multivariate analysis, two wavelengths (1101.69
and 1305.05 nm) were shortlisted. Six statistical features and 10 histogram
features from the images corresponding to the shortlisted wavelengths were
extracted to build statistical discriminant classifiers. As working in the
long-wave NIR region is expensive, the same authors have investigated to build
the classifiers in short-wave NIR range (700 - 1100 nm) in [77].

Researchers from Murdoch University, Australia have developed a system name
Capsulanet using VNIR HSI to identify different types of insects in the stored
grain setting [78]. More than 2000 images of adults, larvae and larvae skins
of different types of insects have been captured to analyse the reflected
energies at different wavelengths. This analysis helps to determine the
physical and chemical characteristics of the insects and eventually detecting
them [79].

Figure 9: C: sound kernels; I: internally infested kernels imaged at 1202 nm
and 1300 nm, respectively. Adopted from [75].

##  VI HSI for Contaminants detection

The presence of unwanted materials is a major determining factor for assessing
wheat quality. This contaminants include un-millable materials such as,
broken, small and cracked kernels. In addition, foreign seeds and other
contaminants are also unwanted. Examples of foreign seed contaminants are
poppy, coriander, peanut, soybean, sunflower, chickpea, broad bean etc. As per
GTA standard [3], they fall into 11 categories. Examples of other contaminants
are ryegrass ergot, insects, snails, loose smut, sand etc. Some contaminants
are often lighter than the sound wheat kernels and contain higher moisture. A
sample of various types of contaminants coexist with wheat is shown in Figure
10. Bulk wheat are stored in silos and the lighter contaminants gather along
the wall of silos. This makes the high-moisture, lighter contaminants more
prone towards mould growth and eventually damage the wheat of the entire silo.
Therefore, accurate identification followed by removal of these unwanted
materials is necessary for long-term safe storage.

Figure 10: Images of (a) two wheat kernel, a straw and one maize kernel (b)
two wheat kernels, six rapeseeds and two insects (c) two wheat kernels, two
chaffy barley kernels and a piece of painting. Adopted from [80].

In addition, any appalling item like animal excreta contributed by the the
living animals like insects, snails, rodents contaminate wheat is considered
as filth [101]. Visual presence of filth reduces the wheat quality and its
marketability. Therefore, it is required to identify and remove these animals
from the bulk wheat storage.

Traditional methods like visual inspection to detect contaminants has many
drawbacks and therefore, to facilitate fast and online detection, HSI has also
been used for contaminant detection. In [80], NIR HSI in the wavelength range
of 1100-2400 nm has been used to detect insects, irrelevant cereal grains,
botanical impurities in the bulk grain. Another NIR HSI in the wavelength
range of 1000-1600 nm has been used to classify seven foreign material types
(barley, canola, maize, flaxseed, oats, rye, and soybean), six dockage types
(broken wheat kernels, buckwheat, chaff, wheat spikelets, stones, and wild
oats), and two animal excreta types (deer and rabbit droppings). The raw NIR
spectra were pre-processed by five different spectral processing methods
followed by classifying the pre-processed data by three classification models.
Each classification model has been used in two approaches. First one is two-
class classification: classification of each contaminant type from the sound
wheat kernels. Another approach is multi-class classification: classification
of all contaminant types from sound wheat kernel [81]. Among the different
combinations, highest classification accuracies were observed when spectral
data were pre-processed using the SNV technique and classified using the k-NN
classifier [82].

Two regression models: principal component regression (PCR) and partial least-
squares regression (PLSR) were developed to predict the percentage level (0,
3, 6, 9, 12, and 15%) of broken wheat kernels. Hyper spectral images in
1000–1600 nm wavelength range at a 10 nm interval of bulk wheat samples at
different broken level were captured and the corresponding reflectance spectra
were used to train the regression models. In addition, classification models
were developed using linear discriminant analysis (LDA) and quadratic
discriminant analysis (QDA) to successfully classify sound kernels from
different levels of broken kernels in bulk wheat samples [83].

Cereal grains like wheat are often affected by fungal infection. One such
fugal species is Claviceps purpurea, it produces ergot bodies. Ergot bodies
possess alkaloid content and it is toxic for both human and animals [102]. In
Australia, some of the major contaminants are ryegrass ergot and cereal ergot
[103]. The toxicity of ergot bodies are followed in the wheat flour as well as
in the baked products. Therefore, these bodies needs to be identified and
removed. Often these bodies are almost the same size and shape as a wheat
kernel and replace healthy wheat kernels [104]. It makes the ergot removal
even difficult. A sample images in Figure 11 shows the visual comparison of
sound wheat kernels and ergot bodies. Different countries have their tolerance
levels for ergot bodies. In Australia, the maximum allowable ergot bodies in
the major milling grades is assigned by GTA [3].

Figure 11: Sound wheat kernels (left) are compared to ergot bodies (right).
Adopted from [104].

An NIR HSI method has been proposed in [84] to detect and quantify ergot
bodies. Images were collected using an NIR hyperspectral line scan combined
with a conveyor belt at 209 wavelengths in the range of 1100–2400 nm. For
training and evaluation of regression models (e.g., partial least square
discriminant analysis, PLSDA), SVM, several samples of wheat with different
ergot contamination level were prepared. Evaluated results exhibited a
correlation higher than 0.99 between the predicted values and the ground-truth
values.

##  VII conclusion

In this paper we have reviewed literature related to post-harvest wheat
quality assessment. In a supply chain, wheat trucks arrives as well as leaves
the receival points where wheat from different sources may get mixed and
stored for a limited time. Therefore, it is required to constantly assess the
wheat in this receival points to avoid any degradation or damage to the wheat
quality. Traditional assessment methods are invasive, destructive, time-
consuming and often expensive. Therefore, non-invasive, non-destructive, real-
time or digital methods gained popularity as efficient and effective
alternatives. Wheat quality assessment using HSI is one of the most popular
alternative. In this paper, we have discussed how different wheat quality
parameters from protein and moisture prediction to defect and contaminant
detection have been performed in the literature using HSI. Therefore, this
article can be regarded as a reference while implementing digital methods for
assessing wheat quality at the different stages of Australian supply chain.

## References

  * [1] “Australian wheat: Australian export grains innovationn centre,” https://www.aegic.org.au/australian-grains/wheat/.
  * [2] “Abares, 2015. australian crop report. australian bureau of agricultural and resource economics and sciences, canberra, australia. no. 174.” https://www.agriculture.gov.au/abares/publications/.
  * [3] “Grain trade australia. wheat trading standards,” https://www.graintrade.org.au/sites/default/files/Standards/GTA%20Wheat%20Statement%20of%20Standards%20202122%20All%20Grades.pdf.
  * [4] Y. Liu, H. Pu, and D.-W. Sun, “Hyperspectral imaging technique for evaluating food quality and safety during various processes: A review of recent applications,” _Trends in food science & technology_, vol. 69, pp. 25–35, 2017.
  * [5] D. Wu and D.-W. Sun, “Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment: A review—part ii: Applications,” _Innovative Food Science & Emerging Technologies_, vol. 19, pp. 15–28, 2013.
  * [6] D. P. Ariana, R. Lu, and D. E. Guyer, “Near-infrared hyperspectral reflectance imaging for detection of bruises on pickling cucumbers,” _Computers and electronics in agriculture_ , vol. 53, no. 1, pp. 60–70, 2006.
  * [7] X. Zhu and G. Li, “Rapid detection and visualization of slight bruise on apples using hyperspectral imaging,” _International journal of food properties_ , vol. 22, no. 1, pp. 1709–1719, 2019.
  * [8] H.-J. He and D.-W. Sun, “Hyperspectral imaging technology for rapid detection of various microbial contaminants in agricultural and food products,” _Trends in Food Science & Technology_, vol. 46, no. 1, pp. 99–109, 2015\.
  * [9] R. Calvini, S. Michelini, V. Pizzamiglio, G. Foca, and A. Ulrici, “Exploring the potential of nir hyperspectral imaging for automated quantification of rind amount in grated parmigiano reggiano cheese,” _Food Control_ , vol. 112, p. 107111, 2020.
  * [10] C.-H. Feng and Y. Makino, “Colour analysis in sausages stuffed in modified casings with different storage days using hyperspectral imaging–a feasibility study,” _Food Control_ , vol. 111, p. 107047, 2020.
  * [11] W. Harris and C. Freudenrich, “How light works,” _HowStuffWorks. com_ , 2000\.
  * [12] S. Majumdar and D. S. Jayas, “Classification of cereal grains using machine vision: Iv. combined morphology, color, and texture models,” _Transactions of the ASAE_ , vol. 43, no. 6, p. 1689, 2000.
  * [13] R. Choudhary, J. Paliwal, and D. Jayas, “Classification of cereal grains using wavelet, morphological, colour, and textural features of non-touching kernel images,” _Biosystems engineering_ , vol. 99, no. 3, pp. 330–337, 2008.
  * [14] J. Paliwal, N. Visen, and D. Jayas, “Evaluation of neural network architectures for cereal grain classification using morphological features,” _Journal of agricultural engineering research_ , vol. 79, no. 4, pp. 361–370, 2001.
  * [15] J. Paliwal, N. Visen, D. Jayas, and N. White, “Cereal grain and dockage identification using machine vision,” _Biosystems engineering_ , vol. 85, no. 1, pp. 51–57, 2003.
  * [16] N. S. Visen, J. Paliwal, D. Jayas, and N. White, “Image analysis of bulk grain samples using neural networks,” in _2003 ASAE Annual Meeting_. American Society of Agricultural and Biological Engineers, 2003, p. 1.
  * [17] X. Luo, D. Jayas, and S. Symons, “Identification of damaged kernels in wheat using a colour machine vision system,” _Journal of cereal science_ , vol. 30, no. 1, pp. 49–59, 1999.
  * [18] A. Tahir, S. Neethirajan, D. Jayas, M. Shahin, S. Symons, and N. White, “Evaluation of the effect of moisture content on cereal grains by digital image analysis,” _Food Research International_ , vol. 40, no. 9, pp. 1140–1145, 2007.
  * [19] G. Ramalingam, S. Neethirajan, D. S. Jayas, and N. D. White, “Characterization of the influence of moisture content on the morphological features of single wheat kernels using machine vision,” _Applied Engineering in Agriculture_ , vol. 27, no. 3, pp. 403–409, 2011.
  * [20] C. Karunakaran, D. Jayas, and N. White, “X-ray image analysis to detect infestations caused by insects in grain,” _Cereal Chemistry_ , vol. 80, no. 5, pp. 553–557, 2003.
  * [21] S. Neethirajan, D. Jayas, and N. White, “Detection of sprouted wheat kernels using soft x-ray image analysis,” _Journal of Food Engineering_ , vol. 81, no. 3, pp. 509–513, 2007.
  * [22] E. B. Maghirang, F. Dowell, J. Baker, and J. E. Throne, “Automated detection of single wheat kernels containing live or dead insects using near–infrared reflectance spectroscopy,” _Transactions of the ASAE_ , vol. 46, no. 4, p. 1277, 2003.
  * [23] S. Sudharsan, R. Hemalatha, and S. Radha, “A survey on hyperspectral imaging for mineral exploration using machine learning algorithms,” in _2019 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET)_. IEEE, 2019, pp. 206–212.
  * [24] A. A. Gowen, C. P. O’Donnell, P. J. Cullen, G. Downey, and J. M. Frias, “Hyperspectral imaging–an emerging process analytical tool for food quality and safety control,” _Trends in food science & technology_, vol. 18, no. 12, pp. 590–598, 2007.
  * [25] G. Elmasry, M. Kamruzzaman, D.-W. Sun, and P. Allen, “Principles and applications of hyperspectral imaging in quality evaluation of agro-food products: a review,” _Critical reviews in food science and nutrition_ , vol. 52, no. 11, pp. 999–1023, 2012.
  * [26] “Introduction to hyperspectral imaging,” https://resonon.com/library.
  * [27] S. Srivastava and H. N. Mishra, “Detection of insect damaged rice grains using visible and near infrared hyperspectral imaging technique,” _Chemometrics and Intelligent Laboratory Systems_ , vol. 221, p. 104489, 2022\.
  * [28] M. A. Shahin and S. J. Symons, “Detection of hard vitreous and starchy kernels in amber durum wheat samples using hyperspectral imaging (grl number m306),” _NIR news_ , vol. 19, no. 5, pp. 16–18, 2008.
  * [29] A. Del Fiore, M. Reverberi, A. Ricelli, F. Pinzari, S. Serranti, A. Fabbri, G. Bonifazi, and C. Fanelli, “Early detection of toxigenic fungi on maize by hyperspectral imaging analysis,” _International journal of food microbiology_ , vol. 144, no. 1, pp. 64–71, 2010.
  * [30] I. Orina, M. Manley, and P. J. Williams, “Non-destructive techniques for the detection of fungal infection in cereal grains,” _Food Research International_ , vol. 100, pp. 74–86, 2017.
  * [31] B. G. Osborne, “Investigations into the use of near infrared reflectance spectroscopy for the quality assessment of wheat with respect to its potential for bread baking,” _Journal of the Science of Food and Agriculture_ , vol. 35, no. 1, pp. 106–110, 1984.
  * [32] P. Williams, “Near-infrared technology in the agricultural and food industries,” Tech. Rep., 1990.
  * [33] P. C. Williams and D. Sobering, “Comparison of commercial near infrared transmittance and reflectance instruments for analysis of whole grains and seeds,” _Journal of Near Infrared Spectroscopy_ , vol. 1, no. 1, pp. 25–32, 1993.
  * [34] T. Bramble, F. Dowell, and T. Herrman, “Single-kernel near-infrared protein prediction and the role of kernel weight in hard red winter wheat,” _Applied Engineering in Agriculture_ , vol. 22, no. 6, pp. 945–949, 2006\.
  * [35] S. R. Delwiche, “Single wheat kernel analysis by near-infrared transmittance: Protein content,” 1995.
  * [36] S. Delwiche, “Protein content of single kernels of wheat by near-infrared reflectance spectroscopy,” _Journal of Cereal Science_ , vol. 27, no. 3, pp. 241–254, 1998.
  * [37] S. R. Delwiche and W. R. Hruschka, “Protein content of bulk wheat from near-infrared reflectance of individual kernels,” _Cereal Chemistry_ , vol. 77, no. 1, pp. 86–88, 2000.
  * [38] F. Dowell, E. Maghirang, F. Xie, G. Lookhart, R. Pierce, B. Seabourn, S. Bean, J. Wilson, and O. Chung, “Predicting wheat quality characteristics and functionality using near-infrared spectroscopy,” _Cereal chemistry_ , vol. 83, no. 5, pp. 529–536, 2006.
  * [39] N. Caporaso, M. B. Whitworth, and I. D. Fisk, “Protein content prediction in single wheat kernels using hyperspectral imaging,” _Food chemistry_ , vol. 240, pp. 32–42, 2018.
  * [40] O.-C. G. Engstrøm, E. S. Dreier, and K. S. Pedersen, “Predicting protein content in grain using hyperspectral deep learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1372–1380.
  * [41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [42] S. Mahesh, D. Jayas, J. Paliwal, and N. White, “Comparison of partial least squares regression (plsr) and principal components regression (pcr) methods for protein and hardness predictions using the near-infrared (nir) hyperspectral images of bulk samples of canadian wheat,” _Food and bioprocess technology_ , vol. 8, no. 1, pp. 31–40, 2015.
  * [43] Y. Shuqin, H. Dongjian, and N. Jifeng, “Predicting wheat kernels’ protein content by near infrared hyperspectral imaging,” _International Journal of Agricultural and Biological Engineering_ , vol. 9, no. 2, pp. 163–170, 2016\.
  * [44] I. Tahmasbian, N. K. Morgan, S. Hosseini Bai, M. W. Dunlop, and A. F. Moss, “Comparison of hyperspectral imaging and near-infrared spectroscopy to determine nitrogen and carbon concentrations in wheat,” _Remote Sensing_ , vol. 13, no. 6, p. 1128, 2021.
  * [45] W. Wang, J. Paliwal, and D. Jayas, “Determination of moisture content of ground wheat using near-infrared spectroscopy,” in _ASAE/CSAE Meeting presentation, paper no. MB04–200_ , 2004.
  * [46] S. Mahesh, D. Jayas, J. Paliwal, and N. White, “Identification of wheat classes at different moisture levels using near-infrared hyperspectral images of bulk samples,” _Sensing and instrumentation for food quality and safety_ , vol. 5, no. 1, pp. 1–9, 2011.
  * [47] S. Mahesh, D. S. Jayas, J. Paliwal, and N. D. White, “Comparing two statistical discriminant models with a back-propagation neural network model for pairwise classification of location and crop year specific wheat classes at three selected moisture contents using nir hyperspectral images,” _Transactions of the ASABE_ , vol. 57, no. 1, pp. 63–74, 2014.
  * [48] K. Shashikumar, “Predicting wheat sprout damage by near-infrared reflectance analysis,” _Cereal foods world_ , vol. 38, no. 5, pp. 364–366, 1993.
  * [49] J. Xing, S. Symons, M. Shahin, and D. Hatcher, “Detection of sprout damage in canada western red spring wheat with multiple wavebands using visible/near-infrared hyperspectral imaging,” _Biosystems Engineering_ , vol. 106, no. 2, pp. 188–194, 2010.
  * [50] Q. Wu, J. H. Wang, D. Zhu, C. Wang, and Z. Ma, “Recognition of wheat preharvest sprouting based on hyperspectral imaging,” _Optical Engineering_ , vol. 51, no. 11, p. 111710, 2012.
  * [51] J. G. Barbedo, E. M. Guarienti, and C. S. Tibola, “Detection of sprout damage in wheat kernels using nir hyperspectral imaging,” _biosystems engineering_ , vol. 175, pp. 124–132, 2018.
  * [52] L. Zhang, H. Sun, Z. Rao, and H. Ji, “Non-destructive identification of slightly sprouted wheat kernels using hyperspectral data on both sides of wheat kernels,” _Biosystems Engineering_ , vol. 200, pp. 188–199, 2020.
  * [53] S. Grassi, G. Cardone, D. Bigagnoli, and A. Marti, “Monitoring the sprouting process of wheat by non-conventional approaches,” _Journal of Cereal Science_ , vol. 83, pp. 180–187, 2018.
  * [54] M. Berman, P. Connor, L. Whitbourn, D. Coward, B. Osborne, and M. Southan, “Classification of sound and stained wheat grains using visible and near infrared hyperspectral image analysis,” _Journal of Near Infrared Spectroscopy_ , vol. 15, no. 6, pp. 351–358, 2007.
  * [55] S. R. Delwiche, I.-C. Yang, and M. S. Kim, “Hyperspectral imaging for detection of black tip damage in wheat kernels,” in _Sensing for Agriculture and Food Quality and Safety_ , vol. 7315. International Society for Optics and Photonics, 2009, p. 73150K.
  * [56] Y. Shao, C. Gao, G. Xuan, X. Gao, Y. Chen, and Z. Hu, “Determination of damaged wheat kernels with hyperspectral imaging analysis,” _International Journal of Agricultural and Biological Engineering_ , vol. 13, no. 5, pp. 194–198, 2020.
  * [57] E. Alisaac, J. Behmann, A. Rathgeb, P. Karlovsky, H.-W. Dehne, and A.-K. Mahlein, “Assessment of fusarium infection and mycotoxin contamination of wheat kernels and flour using hyperspectral imaging,” _Toxins_ , vol. 11, no. 10, p. 556, 2019.
  * [58] S. R. Delwiche and G. A. Hareland, “Detection of scab-damaged hard red spring wheat kernels by near-infrared reflectance,” _Cereal chemistry_ , vol. 81, no. 5, pp. 643–649, 2004.
  * [59] S. R. Delwiche, “High-speed bichromatic inspection of wheat kernels for mold and color class using high-power pulsed leds,” _Sensing and Instrumentation for Food Quality and Safety_ , vol. 2, no. 2, pp. 103–110, 2008\.
  * [60] G. Polder, G. Van Der Heijden, C. Waalwijk, and I. Young, “Detection of fusarium in single wheat kernels using spectral imaging,” _Seed Science and Technology_ , vol. 33, no. 3, pp. 655–668, 2005.
  * [61] M. A. Shahin and S. J. Symons, “Detection of fusarium damaged kernels in canada western red spring wheat using visible/near-infrared hyperspectral imaging and principal component analysis,” _Computers and electronics in agriculture_ , vol. 75, no. 1, pp. 107–112, 2011.
  * [62] S. Weng, K. Han, Z. Chu, G. Zhu, C. Liu, Z. Zhu, Z. Zhang, L. Zheng, and L. Huang, “Reflectance images of effective wavelengths from hyperspectral imaging for identification of fusarium head blight-infected wheat kernels combined with a residual attention convolution neural network,” _Computers and Electronics in Agriculture_ , vol. 190, p. 106483, 2021.
  * [63] L. Yipeng, L. Wenbing, H. Kaixuan, T. Wentao, Z. Ling, W. Shizhuang, and H. Lingsheng, “Determination of wheat kernels damaged by fusarium head blight using monochromatic images of effective wavelengths from hyperspectral imaging coupled with an architecture self-search deep network,” _Food Control_ , p. 108819, 2022.
  * [64] S. Serranti, D. Cesare, and G. Bonifazi, “The development of a hyperspectral imaging method for the detection of fusarium-damaged, yellow berry and vitreous italian durum wheat kernels,” _Biosystems engineering_ , vol. 115, no. 1, pp. 20–30, 2013.
  * [65] A. Femenias, M. B. Bainotti, F. Gatius, A. J. Ramos, and S. Marín, “Standardization of near infrared hyperspectral imaging for wheat single kernel sorting according to deoxynivalenol level,” _Food Research International_ , vol. 139, p. 109925, 2021.
  * [66] M. Nadimi, J. Brown, J. Morrison, and J. Paliwal, “Examination of wheat kernels for the presence of fusarium damage and mycotoxins using near-infrared hyperspectral imaging,” _Measurement: Food_ , vol. 4, p. 100011, 2021.
  * [67] K. Liang, Q. Liu, J. Xu, Y. Wang, C. Okinda, and M. Shena, “Determination and visualization of different levels of deoxynivalenol in bulk wheat kernels by hyperspectral imaging,” _Journal of Applied Spectroscopy_ , vol. 85, no. 5, pp. 953–961, 2018.
  * [68] D. Zhang, G. Chen, H. Zhang, N. Jin, C. Gu, S. Weng, Q. Wang, and Y. Chen, “Integration of spectroscopy and image for identifying fusarium damage in wheat kernels,” _Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy_ , vol. 236, p. 118344, 2020.
  * [69] T. Zhao, M. Chen, X. Jiang, F. Shen, X. He, Y. Fang, Q. Liu, and Q. Hu, “Integration of spectra and image features of vis/nir hyperspectral imaging for prediction of deoxynivalenol contamination in whole wheat flour,” _Infrared Physics & Technology_, vol. 109, p. 103426, 2020.
  * [70] A. D. Hocking, “Microbiological facts and fictions in grain storage,” in _Proceedings of the Australian postharvest technical conference_. CSIRO, 2003, pp. 55–58.
  * [71] T. Senthilkumar, D. Jayas, N. White, P. Fields, and T. Gräfenhan, “Detection of fungal infection and ochratoxin a contamination in stored wheat using near-infrared hyperspectral imaging,” _Journal of Stored Products Research_ , vol. 65, pp. 30–39, 2016.
  * [72] C. Singh, D. Jayas, J. Paliwal, and N. White, “Fungal detection in wheat using near-infrared hyperspectral imaging,” _Transactions of the ASABE_ , vol. 50, no. 6, pp. 2171–2176, 2007.
  * [73] K. Liang, J. Huang, R. He, Q. Wang, Y. Chai, and M. Shen, “Comparison of vis-nir and swir hyperspectral imaging for the non-destructive detection of don levels in fusarium head blight wheat kernels and wheat flour,” _Infrared Physics & Technology_, vol. 106, p. 103281, 2020.
  * [74] H. Li, L. Zhang, H. Sun, Z. Rao, and H. Ji, “Discrimination of unsound wheat kernels based on deep convolutional generative adversarial network and near-infrared hyperspectral imaging technology,” _Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy_ , vol. 268, p. 120722, 2022.
  * [75] C. Ridgway and J. Chambers, “Detection of insects inside wheat kernels by nir imaging,” _Journal of Near Infrared Spectroscopy_ , vol. 6, no. 1, pp. 115–119, 1998.
  * [76] C. B. Singh, D. S. Jayas, J. Paliwal, and N. D. White, “Detection of sprouted and midge-damaged wheat kernels using near-infrared hyperspectral imaging,” _Cereal Chemistry_ , vol. 86, no. 3, pp. 256–260, 2009.
  * [77] ——, “Identification of insect-damaged wheat kernels using short-wave near-infrared hyperspectral and digital colour imaging,” _Computers and electronics in agriculture_ , vol. 73, no. 2, pp. 118–125, 2010.
  * [78] “New app shields farmers from ‘most unwanted pest,” https://campusnews.murdoch.edu.au/science/new-app-targets-most-unwanted-pest/.
  * [79] “Machine learning puts high-risk pests in the frame,” https://grdc.com.au/resources-and-publications/groundcover/groundcover-july-august-2018/machine-learning-puts-high-risk-pests-in-the-frame.
  * [80] J. F. Pierna, P. Vermeulen, O. Amand, A. Tossens, P. Dardenne, and V. Baeten, “Nir hyperspectral imaging spectroscopy and chemometrics for the detection of undesirable substances in food and feed,” _Chemometrics and Intelligent Laboratory Systems_ , vol. 117, pp. 233–239, 2012.
  * [81] L. Ravikanth, C. B. Singh, D. S. Jayas, and N. D. White, “Classification of contaminants from wheat using near-infrared hyperspectral imaging,” _Biosystems Engineering_ , vol. 135, pp. 73–86, 2015.
  * [82] ——, “Performance evaluation of a model for the classification of contaminants from wheat using near-infrared hyperspectral imaging,” _Biosystems engineering_ , vol. 147, pp. 248–258, 2016.
  * [83] L. Ravikanth, V. Chelladurai, D. S. Jayas, and N. D. White, “Detection of broken kernels content in bulk wheat samples using near-infrared hyperspectral imaging,” _Agricultural Research_ , vol. 5, no. 3, pp. 285–292, 2016.
  * [84] P. Vermeulen, J. F. Pierna, H. v. Egmond, P. Dardenne, and V. Baeten, “Online detection and quantification of ergot bodies in cereals using near infrared hyperspectral imaging,” _Food Additives & Contaminants: Part A_, vol. 29, no. 2, pp. 232–240, 2012.
  * [85] J. Taylor, B. Whelan, L. Thylén, M. Gilbertsson, and J. Hassall, “Monitoring wheat protein content on-harvester: Australian experiences,” _Precision agriculture_ , vol. 5, pp. 369–375, 2005.
  * [86] “Grain protein and energy levels,” https://www.daf.qld.gov.au/business-priorities/agriculture/animals/pigs/feed-nutrition/nutrients-diets/grain-protein.
  * [87] F. Mariotti, D. Tomé, and P. P. Mirand, “Converting nitrogen into protein—beyond 6.25 and jones’ factors,” _Critical reviews in food science and nutrition_ , vol. 48, no. 2, pp. 177–184, 2008.
  * [88] J. Banks, “High moisture levels increase yield and quality,” _Farming Ahead_ , vol. 94, pp. 39–40, 1999.
  * [89] N. Nanje Gowda and K. Alagusundaram, “Use of thermal imaging to improve the food grains quality during storage,” _International Journal of Current Agricultural Research_ , vol. 1, no. 7, pp. 34–41, 2013.
  * [90] “Understanding australian wheat quality,” https://www.grdc.com.au/__data/assets/pdf_file/0011/202421/grdcwheatquality.pdf.
  * [91] R. S. Goswami and H. C. Kistler, “Heading for disaster: Fusarium graminearum on cereal crops,” _Molecular plant pathology_ , vol. 5, no. 6, pp. 515–525, 2004.
  * [92] “Diagnosing fusarium head blight in cereals,” https://www.agric.wa.gov.au/mycrop/diagnosing-fusarium-head-blight-cereals.
  * [93] “Stored grain insect resistance monitoring and management,” https://www.agric.wa.gov.au/exporting-western-australia/stored-grain-insect-resistance-monitoring-and-management.
  * [94] J. Perez-Mendoza, J. E. Throne, F. E. Dowell, and J. E. Baker, “Detection of insect fragments in wheat flour by near-infrared spectroscopy,” _Journal of stored products research_ , vol. 39, no. 3, pp. 305–312, 2003\.
  * [95] “Visual recognition standards guide,” https://www.graintrade.org.au/sites/default/files/file/Commodity%20Standards/2015_2016/Visual%20Recognition%20Standards%20Guide%202015-2016.pdf.
  * [96] T. Pearson, D. Brabec, and C. Schwartz, “Automated detection of internal insect infestations in whole wheat kernels using a perten skcs 4100,” _Applied Engineering in Agriculture_ , vol. 19, no. 6, p. 727, 2003.
  * [97] “Pests and contaminants of grain and plant products,” https://www.awe.gov.au/sites/default/files/documents/pests-and-contaminants-of-grain-and-plant-products.pdf.
  * [98] A. Manickavasagan, D. Jayas, N. White, and F. Jian, “Thermal imaging of a stored grain silo to detect a hot spot,” _Applied Engineering in Agriculture_ , vol. 22, no. 6, pp. 891–897, 2006.
  * [99] “Sound storage eliminates post-harvest blues,” http://storedgrain.com.au/wp-content/uploads/2013/06/Sound-storage-eliminates-post-harvest-blues1.pdf/.
  * [100] O. A. El-Sebai, R. Sanderson, M. P. Bleiweiss, and N. Schmidt, “Detection of sitotroga cerealella (olivier) infestation of wheat kernels using hyperspectral reflectance,” _Journal of Entomological Science_ , vol. 41, no. 2, pp. 155–164, 2006.
  * [101] H. Dogan and B. Subramanyam, “Analysis for extraneous matter,” in _Food analysis_. Springer, 2010, pp. 351–365.
  * [102] S. Agriopoulou, “Ergot alkaloids mycotoxins in cereals and cereal-derived food products: Characteristics, toxicity, prevalence, and control strategies,” _Agronomy_ , vol. 11, no. 5, p. 931, 2021.
  * [103] B. J. Blaney, J. B. Molloy, and I. J. Brock, “Alkaloids in australian rye ergot (claviceps purpurea) sclerotia: implications for food and stockfeed regulations,” _Animal Production Science_ , vol. 49, no. 11, pp. 975–982, 2009.
  * [104] “Ergot’s effect on wheat grades,” https://grainscanada.gc.ca/en/grain-quality/grain-grading/grading-factors/ergot.html.

TABLE II: Wheat quality standards from GTA [3] Class |  Australian Prime Hard (APH) |  Australian Hard (AH) |  Australian Premium White (APW) |  Australian Standard White (ASW) |  Australian Noodle Wheat (ANW) |  Australian Soft (ASFT) |  Australian Durum (ADR) |  Australian General Purpose |  Australian Feed
---|---|---|---|---|---|---|---|---|---
Grade | APH1 | APH2 | H1 | H2 | APW1 | APW2 | ASW1 | ANW1 | SFTl | ADR1 | AGP1 | FED1
Moisture Max (%) | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5 | 12.5
Protein Min (%) (N x 5.7) @ 11% Moisture Basis | 14.0 | 13.0 | 13.0 | 11.5 | 10.5 | 10 | N/A | 9.5 | N/A | 13.0 | N/A | N/A
Test Weight Min (kg/hl) | 76.0 | 76.0 | 76.0 | 76.0 | 76.0 | 76.0 | 76.0 | 76.0 | 76.0 | 76.0 | 68.0 | 62.0
Screenings Max (% by weight) | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 15.0
Unmillable Material Max (% by weight ) | 0.6 | 0.6 | 0.6 | 0.6 | 0.6 | 0.6 | 0.6 | 0.6 | 0.6 | 0.6 | 1.2 | 2.6
Falling Number Min (seconds) | 350 | 350 | 300 | 300 | 300 | 300 | 300 | 300 | 300 | 300 | 200 | N/A


# Semantic Reasoning with Differentiable Graph Transformations

Alberto Cetoli
QBE Europe
London, UK
<EMAIL_ADDRESS>
###### Abstract

This paper introduces a differentiable semantic reasoner, where rules are
presented as a relevant set of graph transformations. These rules can be
written manually or inferred by a set of facts and goals presented as a
training set. While the internal representation uses embeddings in a latent
space, each rule can be expressed as a set of predicates conforming to a
subset of Description Logic.

###### Index Terms:

Semantic Reasoning, Semantic Graphs, Graph Transformations, Differentiable
Computing.

##  I Introduction

Symbolic logic is the most powerful representation for building interpretable
computational systems [1]. In this work we adopt a subset of Description Logic
[2] to represent knowledge and build a semantic reasoner, which derives new
facts by applying a chain of transformations to the original set.

In the restricted context of this paper, knowledge can be expressed in
predicate or graph form, interchangeably. Thus, semantic reasoning can be
understood as a sequence of graph transformations [3], which act on a subset
of the original knowledge base and sequentially apply the matching rules.

In this paper, we show that rule matching can be made differentiable by
representing nodes and edges as embeddings. After building a one-to-one
correspondence between a sequence of rules and a linear algebra expression,
the system can eventually train the embeddings using a convenient loss
function. The rules created in this fashion can then be applied during
inference time.

Our system follows the recent revival of hybrid neuro-symbolic models [1],
combining insights from logic programming with deep learning methods. The main
contribution of this work is to show that reasoning over graphs is a learnable
task. While the system is presented here as a proof of concept, we show that
differential graph transformations can effectively learn new rules by training
nodes, edges, and matching thresholds through backpropagation.

In Sec. II we describe in detail the fundamentals of our reasoner, with
working examples shown in Sec. III. Sec. IV reviews specific connections with
prior works and finally a few remarks in Sec. V conclude the paper 111The
relevant code can be found at https://github.com/fractalego/dgt/.

##  II Problem statement

Figure 1: A matching rule example, as explained in II-A. The facts on the left
are "transformed" into the ones on the right
following the application of the relevant rule. This picture makes explicit
the dual nature of the predicates/graph representation.

The system presented here is a semantic reasoner inspired by the early STRIPS
language [4]. It creates a chain of rules that connects an initial state of
_facts_ to a final state of inferred predicates. Each rule has a set of pre-
and post-conditions, expressed here using a subset of Description Logic (DL).
In the following, we restrict our DL to Assertional Axioms (ABox). Thus, each
fact can be represented as a set of predicates, or - equivalently - as a graph
with matching rules as described below.

###  II-A Rules as graph transformations

We use a predicate form to represent facts, rules, and intermediate states, as
shown in Fig. (1). For example the semantics for "Joe wins the election in the
USA" is captured in the following form

⬇

joe(a), win(a,b), election(b), in(b,c), USA(c)

In the prior example, $joe$, $election$, and $USA$ are nodes of the semantic
graph, whereas $win$ and $in$ are convenient relations to represent the
graph’s edges.

The rules are specified with a MATCH/CREATE pair as below

⬇

MATCH person(a), win(a,b), election(b)

CREATE (a), be(a,b), president(b)

The MATCH statement specifies the pre-condition that triggers the rule, while
the CREATE statement acts as the effect - or post-condition - after applying
the rule. The result of applying this rule is shown in Fig. 1, where a new
state is created from the original fact. Notice that the name $joe$ (which
matches $person$) is propagated forward to the next set of facts.

By applying rules in sequence one builds a inferential chain of MATCH and
CREATE conditions. After each rule the initial facts graph is changed into a
new set of nodes and edges. This chain of graph transformations builds a path
in a convenient semantic space, as shown in Fig. 2. One of this paper’s main
result is to show that there is a one-to-one correspondence between the chain
of matching rules and a chain of linear algebra operations.

###  II-B Nodes and edges as embeddings

Both nodes and edges are represented as embeddings in a latent space. For
convenience, in the current work the vocabulary of possible nodes matches the
Glove 300dim dataset [5], whereas edges are associated random embeddings
linked to the relevant ontology.

###  II-C Nodes and edges matching

A rule is triggered if the pre-conditions graph is a sub-isomorphism of the
facts. Each node and edge of the preconditions has a learnable threshold value
$t$. Two items match if the dot product between their embeddings is greater
than a specific threshold. In the predicate representation, we make explicit
these trainable thresholds by adding the symbol $>$ to the predicate’s name.
In this way, the rule in II-A becomes

⬇

MATCH person>0.6(a), win>0.7(a,b), election>0.6(b)

CREATE (a), be(a,b), president(b)

indicating that - for example - $joe$ and $person$ would only match if their
normalized dot product is greater than $t=0.6$. In the Description Logic
framework this is equivalent to a _individuality assertion_

$joe\approx person\,\iff\,\mathrm{embedding}(joe)\leavevmode\nobreak\
{\Large\cdot}\leavevmode\nobreak\ \,\mathrm{embedding}(person)>t$

Matching facts and preconditions creates a _most general unifier_ (MGU) that
is propagated forward on the inference chain.

###  II-D Creating a trainable path

During training, the final state is a _goal_ of the system, as shown in Fig.
2. The system learns how to create rules given a set of template _empty rules_
, where the embeddings for each node and edge are chosen randomly. These
templates are specified prior to the training using $*$ to indicate a random
embedding, as in the following

⬇

MATCH *(a), *(a,b), *(b)

CREATE (b), *(b,d), *(d)

In the current state of development, the algorithm generates all possible
paths - compatibly with boundary conditions - and then applies to each of them
the training algorithm explained below. A more efficient method will be
pursued in future works.

###  II-E Training of the embeddings and thresholds

At every step of the inference chain the collection of predicates changes
according to the order of transformations. At every step $i$ we employ a
vector $f_{i}$ that signals the truth value of each predicate. For
computational reasons, the dimensions of this vector must be fixed in advance
and set to the maximum size of the predicate set. The first value $f_{0}$ is a
vector of ones, as every predicate in the knowledge base is assumed to be
true.

At the end of the resolution chain there is a "goal" set of predicates,
usually less numerous than the initial set of facts. A vector $g$ indicates
the truth conditions of the goal predicates. This vector - also of size $n$ \-
contains a number of ones equal to the number of goal nodes and is zero
otherwise. The application of a rule can then be described by two matrices:
the similarity matrix $S$ and the rule propagation matrix $R$.

A similarity matrix describes how well a set of facts matches the
preconditions.

$S_{i}=M_{i}\odot\mathrm{Softmax}(P_{i}^{T}\,F_{i}-T_{i})$ (1)

Where $P_{i}$ is the matrix with the preconditions’s nodes as colums, $F_{i}$
is the matrix with the fact nodes as columns at step $i$. $M_{i}$ is the
matrix of the matches, bearing value of $1$ if two nodes match and vanishing
otherwise. For example if the first node of the preconditions matches the
second node of the facts, the matrix will have value $1$ at position (1, 0).

The matrix $T_{i}$ is a bias matrix whose columns are the list of (trainable)
thresholds for each predicate in the pre-conditions
$T_{i}=\left[t^{i}_{1},\leavevmode\nobreak\ t^{i}_{2},\leavevmode\nobreak\
...\leavevmode\nobreak\ t^{i}_{n}\right]$. This bias effectively enforces the
matching thresholds: A negative value as an argument to $\mathrm{Softmax}$
will lead to an exponentially small result after the operation.

All the matrices $M$, $P$, and $F$ are square matrices
$\in\mathcal{R}^{n\,\times\,n}$. Eq. 1 is reminiscent of self-attention [6],
with an added bias matrix $T$ and a mask $M$.

Figure 2: Example of a graph matching path. The facts on the left (represented
as a graph of connected embeddings) are transformed through a chain of pre-
and post-conditions
into the goal on the right. This chain of rules is equivalent to a sequence of
linear algebra operations, where the truth values of each predicated are
propagated forward through a set
of $S$ and $R$ matrices.

A rule propagation matrix $R$ puts into contact the left side of a rule with
the right side. The idea behind $R$ is to keep track of how information
travels inside a single rule. In this work we simplify the propagation matrix
as a fully connected layer with only one trainable parameter. For example, if
the chosen size $n$ is 4, a rule with three pre-conditions and two post-
conditions nodes has an $R$ matrix as

$R_{i}=w\,\left[{\begin{array}[]{cccc}1&1&1&0\\\ 1&1&1&0\\\ 0&0&0&0\\\
0&0&0&0\\\ \end{array}}\right]$ (2)

where $w$ is the "weight" of the rule. Given a first state $f_{0}$, the set of
truth condition after $n$ steps is

$f_{n}=S_{n-1}\,...\,R_{1}\,S_{1}\,R_{0}\,S_{0}\,f_{0}$ (3)

This final state $f_{n}$ is compared against the goal’s truth vector $g$ to
create a loss function.

The training of the relation embeddings follows the same sequence of
operations as for the nodes. A set of truth vectors $f^{r}$ and states $F^{r}$
is acted upon the relation similarity matrix

$S_{i}^{r}=M_{i}^{r}\odot\mathrm{Softmax}({P^{r}_{i}}^{T}\,F^{r}_{i}-t^{r}_{i})\,,$
(4)

and the corresponding rule propagation matrix for relations $R^{r}_{i}$,
leading to the final truth vector for relations

$f^{r}_{n}=S^{r}_{n-1}\,...\,R^{r}_{1}\,S^{r}_{1}\,R^{r}_{0}\,S^{r}_{0}\,f^{r}_{0}\,.$
(5)

Following the example of the nodes, a goal vector for the relations is named
$g^{r}$, containing the desired truth conditions for relations at the end of
the chain.

The system learns the nodes and edges embeddings of the rules, while the
initial facts and the goal are frozen during training. The system also learns
the _matching thresholds_ $t$ and each rule’s _weight_ $w$. Following Eqs. 3
and 5, the final loss function is computed as a binary cross entropy
expression

$\mathcal{L}=g\leavevmode\nobreak\
\mathrm{log}(f_{n})+g^{r}\leavevmode\nobreak\ \mathrm{log}(f_{n}^{r})\,.$ (6)

The system can in principle be trained over a set of multiple facts and goal
pairs, in which case the loss function is the sum of all the pairs’ losses.
For simplicity, in this paper we limit the training to a single pair of facts
and goal.

In order to avoid the Sussman anomaly the same rule can only be used once in
the same path.

##  III Examples and discussion

###  III-A One-rule learning

As a toy example we want the system to learn that if someone is married to a
"first lady", then this person is president. The facts are

⬇

person(a), spouse(a,b), person(b), be(a,c), first-lady(c)

and the goal is

⬇

person(a), profession(a,b), president(b)

Given the empty rule

⬇

MATCH *(a), *(a,b), *(b), *(a,c), *(c)

CREATE (b), *(b,d), *(d)

The system correctly learns the rule that connects the the facts with the
goal.

⬇

MATCH person>0.6(a), first-lady>0.6(b), person>0.6(c),

be>0.63631916(a,b), spouse>0.6338593(a,c)

CREATE (b), president(d), profession(b,d)

While trivial, this is a fundamental test of the capacity of the system to
learn the correct transformation. The matching thresholds have been clipped
and cannot go below $0.6$ in training.

While a successful result is almost guaranteed by choosing a rule that closely
matches the boundary conditions, the system is proven capable of converging
onto the correct embeddings and thresholds using just backpropagation.

###  III-B Chained two-rules learning

While a single-rule transformation can be useful in a few edge cases, the real
power of semantic reasoning comes from combining rules together. In this
section we show - using another toy example - that the system can learn two
rules at the same time. The simplified task is as in the following: to learn
that "if a fruit is round and is delicious, then it is an apple." The facts
are

⬇

fruit(a), be(a,b), round(b), be(a,c), delicious(c)

and the goal is

⬇

fruit(a), be(a,b), apple(b)

The system is given the two template rules to fit

⬇

MATCH *(a), *(a,b), *(b), *(a,c), *(c)

CREATE (b), and(b,c), (c)

MATCH *(a), and(a,b), *(b)

CREATE *(c), *(c,d), *(d)

Notice the "and" relations in the templates. These relations are frozen during
training and constitute another constraint for the system to satisfy. In the
end, our model learns the correct rules

⬇

MATCH fruit>0.6(a), round>0.6(b), delicious>0.6(c),

be>0.6953449(a,b), be>0.6957883(a,c)

CREATE (b), (c), and(b,c)

MATCH round>0.6(a), delicious>0.6(b), and>0.9(a,b)

CREATE fruit(c), apple(d), be(c,d)

which satisfy the goal when chained.

Here we forced the system to apply two rules since no single template would
fit the boundary conditions. Of particular interest is the fact that the
system learned the preconditions of the second rule
$round>0.6(a),\leavevmode\nobreak\ delicious>0.6(b),\leavevmode\nobreak\
and>0.9(a,b)$. This is not a trivial task, given that it started training with
random embeddings and the only information about the correct values is the one
propagated forward from the first rule.

##  IV Related works

Neuro-symbolic reasoning has been an intriguing line of research in the past
decades [7, 8]. Some recent results make use of a Prolog-like resolution tree
as a harness where to train a neural network [9, 10, 11, 12]. Our work is
similar to theirs, but builds upon a STRIPS-like system instead of Prolog. A
different approach employs a Herbrand base for inductive logic programming in
a bottom-up solver [13].

Finally, one can see our method as a sequence of operations that create or
destroy items sequentially. Each (differential) transformation brings forward
a new state of the system made by discrete elements. These types of algorithms
have already been investigated in the Physics community, for example in [14].

##  V Conclusions and future work

In this work we presented a semantic reasoner that leverages on differential
graph transformations for rule learning. The system is build through a one-to-
one correspondence between a chain of rules and a sequence of linear algebra
operations. Given a set of facts, a goal, and a set of rules with random
embeddings, the reasoner can learn new rules that satisfy the constraints. The
rules are then written as a set of predicates with pre- and post-conditions, a
more interpretable representation than embeddings and weights.

The system presented here is limited in speed and - as a consequence - volume
of training data. This is mostly due to our path-creation algorithm, which
generates all possible paths given a set of rules. A more efficient algorithm
would employ a guided approach to path creation, similar to the method in
[12]. A different and possibly novel efficiency gain could be found in a Monte
Carlo method, where the path converges to the correct one through means of a
Metropolis algorithm. This last approach has already found application in the
Computational Physics community and could be useful in our approach as well.

Another open question resides on whether the system is able to generalize,
given a multiple set of facts and goals. This last inquiry will need a faster
algorithm and will be pursued in a future work.

## References

  * [1] A. d’Avila Garcez and L. C. Lamb, “Neurosymbolic AI: The 3rd Wave,” arXiv e-prints, Dec. 2020, p. arXiv:2012.05876.
  * [2] M. Krötzsch, F. Simancik, and I. Horrocks, “A description logic primer,” ArXiv, vol. abs/1201.4089, 2012.
  * [3] H. Ehrig, C. Ermel, U. Golas, and F. Hermann, Graph and Model Transformation, 01 2015.
  * [4] R. E. Fikes and N. J. Nilsson, “Strips: A new approach to the application of theorem proving to problem solving,” Artificial Intelligence, vol. 2, no. 3, 1971, pp. 189–208. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0004370271900105
  * [5] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532–1543. [Online]. Available: http://www.aclweb.org/anthology/D14-1162
  * [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 5998–6008. [Online]. Available: http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
  * [7] A. S. d. Garcez, D. M. Gabbay, and K. B. Broda, Neural-Symbolic Learning System: Foundations and Applications. Berlin, Heidelberg: Springer-Verlag, 2002.
  * [8] A. Garcez, L. Lamb, and D. Gabbay, “Neural-symbolic cognitive reasoning,” in Cognitive Technologies, 2009.
  * [9] T. Rocktäschel and S. Riedel, “End-to-end differentiable proving,” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017\. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/b2ab001909a8a6f04b51920306046ce5-Paper.pdf
  * [10] P. Minervini, M. Bosnjak, T. Rocktäschel, and S. Riedel, “Towards Neural Theorem Proving at Scale,” in ICML Workshop on Neural Abstract Machines and Program Induction (NAMPI), 2018.
  * [11] L. Weber, P. Minervini, J. Münchmeyer, U. Leser, and T. Rocktäschel, “NLProlog: Reasoning with weak unification for question answering in natural language,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 6151–6161. [Online]. Available: https://aclanthology.org/P19-1618
  * [12] P. Minervini, M. Bosnjak, T. Rocktäschel, S. Riedel, and E. Grefenstette, “Differentiable Reasoning on Large Knowledge Bases and Natural Language,” in Proceedings of the Association for the Advancement of Artificial Intelligence Conference on AI (AAAI), 2020.
  * [13] R. Evans and E. Grefenstette, “Learning explanatory rules from noisy data,” J. Artif. Int. Res., vol. 61, no. 1, Jan. 2018, p. 1–64.
  * [14] A. W. Sandvik, “The stochastic series expansion method for quantum lattice models,” in Computer Simulation Studies in Condensed-Matter Physics XIV, D. P. Landau, S. P. Lewis, and H.-B. Schüttler, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2002, pp. 182–187.